Enum ChunkQualityIssues

Namespace

LMKit.Retrieval.ChunkQuality

Assembly

LM-Kit.NET.dll

Machine-readable reasons explaining a SemanticChunkQualityGate decision. Several issues can apply to the same chunk, so the enumeration is a flags set.

[Flags]
public enum ChunkQualityIssues

Fields

None = 0

No issue detected.

BinaryNoise = 1

The chunk looks like binary or compressed data decoded as text: high density of characters that never occur in natural prose, shattered one-or-two letter tokens, and an out-of-band character distribution. Typical of a PDF whose text layer holds raw stream bytes.

MojibakeSuspected = 2

The chunk is letter-like but its words do not form a plausible language: vowel-starved tokens, rare letter combinations, and no function words of any supported language. Typical of legacy-font encodings (for example Preeti-encoded Nepali extracted as Latin characters) and severely garbled OCR.

MixedScript = 4

Tokens mix several Unicode scripts (for example Latin letters spliced with Cyrillic or box-drawing characters inside one word), a strong extraction-artifact fingerprint.

NoLinguisticSignal = 8

After normalization the chunk contains no measurable linguistic signal: no function words, no natural word shapes, and no value-bearing tokens.

MarkupScaffolding = 16

The chunk contained markup scaffolding (HTML tags, table rails, quote markers) that was removed from the embedding text. Informational when content remains; combined with NoLinguisticSignal when nothing of value was left.

RepeatedBoilerplate = 32

The chunk is a repeat of a chunk already seen in the same document (page furniture, repeated headers or footers, duplicated boilerplate). Only the first occurrence is indexed.

DegenerateRepetition = 64

The chunk consists largely of one short pattern repeated many times (separator runs, degenerate extraction loops).

HeadingOnly = 128

The chunk is a bare structural heading with no body payload. Headings are merged into the following chunk so the title context still contributes to retrieval.

TooShortStandalone = 256

The chunk is too short to answer a semantic query on its own. Clean short fragments are merge candidates, never silent drops.

NavigationFragment = 512

The chunk looks like navigation or link-list content (URLs, anchors, menu fragments) rather than document prose.

MetadataOnly = 1024

The chunk carries only identifiers or file metadata (content-id references, bare file names, tracking tokens) with no searchable meaning.

NumericNoise = 2048

The chunk is dominated by unlabeled numbers (table interiors dumped without their headers). Numbers accompanied by labels or units are not given this issue.

EncodingArtifacts = 4096

The chunk contains Unicode replacement characters (U+FFFD), evidence that the source bytes could not be decoded.

MixedContent = 8192

The chunk mixed clean text with garbage lines (typical of partially garbled extractions). The garbage lines were removed from the embedding text; when too little clean content remained, the chunk was dropped instead.

ChoppedExtraction = 16384

The extraction shattered words into single-letter scraps ("p o u r u n diag n ostic"), so the chunk is token soup rather than embeddable prose. The page's verbatim text remains covered by full-text search.

Remarks

Issue flags are diagnostic metadata: they name the evidence that contributed to the decision, so rejected content can be logged, audited, and used to refine thresholds. A non-zero issue set does not by itself imply rejection; for example MarkupScaffolding is routinely present on perfectly indexable table chunks whose markup was stripped from the embedding text.