Table of Contents

Property LoraRank

Namespace
LMKit.Finetuning
Assembly
LM-Kit.NET.dll

LoraRank

Gets or sets LoRA's rank ('r'), which is the default rank used for low-rank adaptations.
This parameter also defines the scaling factor together with LoraAlpha for the LoRA approach.

public int LoraRank { get; set; }

Property Value

int

The default value is 4.

Remarks

The rank value ('r') affects the size and capabilities of the model:

  • Low values (e.g., 4 or 8): Ideal for stylistic modifications with minimal impact on model size and training speed.
  • Medium values (e.g., 128 or 256): Suitable for substantive content adjustments, enhancing the model's ability to integrate new information.
  • High values (1024+): Although challenging to train due to increased computational demands, these values can significantly enhance the model’s detail recognition in large datasets. Such settings require considerable memory resources.

For more detailed information on the effects of different rank settings, refer to: The Effects of Rank, Iterations, and Learning Rate on Training Textual LoRAs.