Table of Contents

Class SemanticChunkQualityGate

Namespace
LMKit.Retrieval.ChunkQuality
Assembly
LM-Kit.NET.dll

A fast, deterministic quality gate that decides whether a text chunk is worth embedding into a semantic (vector) index. It rejects chunks that are technically valid text but carry no standalone semantic value: binary extraction noise, legacy-font mojibake, markup scaffolding, repeated boilerplate, and metadata-only fragments. Once embedded, such chunks become recurring high-score false positives across unrelated queries; dropping them before embedding both protects search quality and saves embedding time.

public sealed class SemanticChunkQualityGate
Inheritance
SemanticChunkQualityGate
Inherited Members

Examples

var gate = new SemanticChunkQualityGate();

DocumentChunkGateSession session = gate.BeginDocument();
foreach (var page in pages)
{
    IReadOnlyList<ChunkIndexUnit> units = session.EvaluatePage(page.ChunkTexts);
    foreach (ChunkIndexUnit unit in units)
    {
        Embed(unit.EmbeddingText);   // only chunks worth indexing reach the model
    }
}
DocumentQualitySummary summary = session.Complete();
if (summary.LooksExtractionFailed)
{
    // The source document's text layer is unusable; consider re-extracting with OCR.
}

Remarks

The gate is built exclusively from lightweight statistical and linguistic signals (character-class profiles, Unicode-script consistency, function-word coverage across the 35 languages the SDK ships stopword lists for, Latin word-shape naturalness, repetition and entropy measurements). It performs no model inference and no regular-expression matching; gating a typical chunk costs on the order of tens of microseconds.

The decision policy favors precision over aggressive filtering: a chunk is only dropped on strong, multi-signal evidence, short-but-meaningful fragments are merged rather than discarded, and text in any script-consistent language is protected even when no curated word list covers it. See ChunkQualityProfile for the calibrated presets.

Instances are immutable and thread-safe; create one gate and share it across an entire ingestion pipeline. Use Assess(string) for stateless single-chunk scoring, or BeginDocument() to gate a whole document with duplicate suppression, neighbor merging, and a document-level extraction verdict.

Constructors

SemanticChunkQualityGate()

Creates a gate with the recommended defaults (Balanced).

SemanticChunkQualityGate(ChunkQualityGateOptions)

Creates a gate with the given options. The option values are copied: mutating options afterwards does not affect this gate.

Properties

Options

The effective configuration of this gate (a snapshot; read-only in effect).

Methods

Assess(string)

Scores a single chunk with no document context. Duplicate suppression and neighbor merging require a DocumentChunkGateSession; here a Merge result is a hint that the chunk is clean but too weak to stand alone.

BeginDocument()

Starts a gating session for one document. The session accumulates per-document state (duplicate detection, totals) and applies the neighbor-merge policy inside each page. Sessions are single-threaded and cheap; create one per document.

Share