Table of Contents

Property MicroBatchSize

Namespace
LMKit.Finetuning
Assembly
LM-Kit.NET.dll

MicroBatchSize

Tokens evaluated per training micro-batch. Activation memory scales linearly with this, so it is the first lever when a run does not fit device memory; larger values run faster when memory allows. 0 (the default) picks 512 or 256 to divide the window, and halves it (down to 128) when the devices cannot hold the run's optimization graph at that size, before refusing the run. The value snaps down to the nearest power of two that divides the resolved window (see ResolvedMicroBatchSize). A memory and speed lever only: attention spans the whole window and the backward pass crosses micro-batches exactly, so every split trains the same gradient. Architectures carrying a recurrent state (hybrid and recurrent models) are the exception: the gradient does not flow back through that state across a micro-batch boundary, so every sample must fit one micro-batch. On them the default grows to hold the longest sample, and an explicit value that would split a sample makes training throw a FinetuningException naming the micro-batch the data needs, unless AllowRecurrentMicroBatchSplit accepts the inexact split.

public int MicroBatchSize { get; set; }

Property Value

int
Share