Class DocumentChunkGateSession
- Namespace
- LMKit.Retrieval.ChunkQuality
- Assembly
- LM-Kit.NET.dll
Gates the chunks of one document, page by page, with the per-document features a stateless assessment cannot provide: suppression of chunks repeated inside the document (page furniture, boilerplate), folding of weak-but-clean chunks into an adjacent chunk of the same page, and a document-level extraction verdict.
public sealed class DocumentChunkGateSession- Inheritance
-
DocumentChunkGateSession
- Inherited Members
Remarks
Obtain a session from BeginDocument(), call EvaluatePage(IReadOnlyList<string>) once per page (in document order), embed the returned units, then call Complete() for the totals. A session is intended for one document and is not thread-safe; the parent gate is immutable and can serve many concurrent sessions.
Merged units always cover a contiguous run of kept source chunks. A dropped or flagged chunk breaks a pending merge, so a caller that addresses a merged unit by the character span from its first to its last source chunk never re-includes rejected content.
Properties
- OnChunkRejected
Optional observer invoked for every chunk the session rejects (dropped or flagged), with the chunk's zero-based index in the current EvaluatePage(IReadOnlyList<string>) call and its full assessment. Intended for logging, auditing, and rejected-content dumps; exceptions thrown by the observer propagate to the EvaluatePage(IReadOnlyList<string>) caller.
- RepetitionTracker
Optional cross-document repetition knowledge (see IChunkRepetitionTracker). When set, short chunks already seen in enough other documents are dropped as RepeatedBoilerplate, and this document's chunks are reported to the tracker. Consulted only while EnableDuplicateSuppression is on.
Methods
- Complete()
Finishes the session and returns the document totals and extraction verdict.
- EvaluatePage(IReadOnlyList<string>)
Assesses one page's chunks (in page order), applies duplicate suppression and the neighbor-merge policy, and returns the units to embed for this page.