Struct ChunkQualitySignals
- Namespace
- LMKit.Retrieval.ChunkQuality
- Assembly
- LM-Kit.NET.dll
The raw, deterministic measurements the SemanticChunkQualityGate computed for one chunk. Exposed for observability: logging these values alongside a decision makes every rejection explainable and reproducible.
public readonly struct ChunkQualitySignals- Inherited Members
Remarks
All values are computed on the normalized text (see EmbeddingText), in a single pass, without any model inference. Typical ranges for natural prose: CharEntropyBits 3.8-5.2, LetterRatio 0.7-0.95, MeanWordLength 3.5-7, FunctionWordCoverage 0.15-0.45, NaturalWordRatio above 0.8.
Properties
- CaseChaosWordRatio
Fraction of multi-letter words with a chaotic casing pattern (uppercase letters in the middle of otherwise lowercase words and similar). Clean prose measures near 0; the occasional camel-case identifier is tolerated by the thresholds.
- CharCount
Length of the normalized text, in characters.
- CharEntropyBits
Shannon entropy, in bits per character, of the non-whitespace character distribution. Prose in alphabetic scripts sits around 3.8-5.2; binary noise drifts well above; repeated filler falls well below. Not meaningful for CJK scripts, whose natural entropy is intrinsically high; the gate accounts for that.
- DigitRatio
Fraction of non-whitespace characters that are decimal digits.
- DominantScript
The most common Unicode script family among the chunk's letters.
- DominantScriptShare
Share of letters belonging to the most common Unicode script of the chunk. Natural text is script-consistent (near 1.0, ignoring incidental foreign words).
- FunctionWordCoverage
Highest fraction of words (two letters or longer) matching the curated function-word list of a single language, across the 35 languages the SDK ships stopword lists for. Real sentences in a supported language score 0.15 or higher; noise scores near 0.
- FunctionWordHits
Absolute number of function-word matches behind FunctionWordCoverage. A high coverage backed by only one or two hits is an accident of a tiny sample, not language evidence; real sentences accumulate hits.
- FunctionWordLanguage
The language whose function-word list produced FunctionWordCoverage, or Undefined when coverage is 0.
- InterleavedDigitWordRatio
Fraction of multi-letter words containing a decimal digit with letters on both sides ("3HWHU0LFKDHO"). Broken font CMaps that shift characters produce these systematically; clean text essentially never does (trailing digits as in "v2" or "A4" do not count).
- LatinWordCount
Number of all-Latin words with at least three letters, the population judged by NaturalWordRatio and VowelRatio. Word-morphology evidence is only trusted when this population is large enough to be meaningful.
- LetterRatio
Fraction of non-whitespace characters that are letters (any script).
- MarkupRemovedRatio
Fraction of the original chunk removed by normalization as markup scaffolding (HTML tags, table rails, quote markers, heading markers). High values on a chunk with little remaining content indicate structure-only fragments.
- MaxLineRepetition
Highest number of times a single non-empty line repeats verbatim inside the chunk. Values above a handful indicate degenerate repetition.
- MeanWordLength
Mean length of letter-bearing words. Prose in European languages sits around 3.5-7; shattered extraction noise falls below 2.5.
- MixedScriptWordRatio
Fraction of words that internally mix two or more Unicode scripts, a strong extraction-artifact fingerprint. Clean text measures 0.
- NaturalWordRatio
Fraction of Latin-script words (three letters or longer) whose letter sequence looks like a plausible word of a Latin-script language (common-bigram and consonant-cluster profile). Mojibake and garbled OCR score far below clean text, which sits above 0.8. The value is 1 when the chunk has no Latin words to judge.
- ReplacementCharCount
Number of Unicode replacement characters (U+FFFD) in the chunk.
- SentenceClosure
Fraction of paragraph blocks ending with sentence-closing punctuation. Well-formed prose scores high; fragments, captions, and cell dumps score near 0.
- SingleLetterWordRatio
Fraction of words that are a single letter, counting the single-letter tokens that letter-spacing collapse rejoined. Prose sits near 0 (French "à", Spanish "y"); extractions that shattered words into pieces score far higher.
- SymbolRatio
Fraction of non-whitespace characters that are neither letters nor digits. Natural prose sits around 0.10-0.20.
- VowelRatio
Fraction of Latin-script words containing at least one vowel. Legacy-font mojibake is characteristically vowel-starved. The value is 1 when the chunk has no Latin words.
- WeirdCharRatio
Fraction of non-whitespace characters that essentially never occur in natural prose in any language (C1 controls, box-drawing and geometric symbols, spacing modifier soup, currency/dingbat noise). The strongest single indicator of binary extraction noise; clean text measures 0.
- WordCount
Number of whitespace-delimited words in the normalized text.