Interface IChunkRepetitionTracker

Namespace

LMKit.Retrieval.ChunkQuality

Assembly

LM-Kit.NET.dll

Cross-document repetition knowledge for a DocumentChunkGateSession. A session detects chunks repeated INSIDE one document by itself; content repeated ACROSS documents (email footers, legal notices, page furniture, template fragments) needs state that outlives a document. Implementations own that state and its scope (typically one tracker per index or per tenant); the session reports what it sees and asks before indexing.

public interface IChunkRepetitionTracker

Remarks

Chunks are identified by the FNV-1a hash of their normalized embedding text, so implementations never store content. The session only consults the tracker for short, non-value-bearing chunks (repeated long-form paragraphs, for example a clause shared by hundreds of contracts, must stay searchable in every document), and only when EnableDuplicateSuppression is on. Implementations must be thread-safe: many sessions report concurrently.

Methods

IsRepeatedAcrossDocuments(ulong)

Whether this chunk is known to repeat across enough distinct documents to be considered boilerplate. Returning true makes the session drop the chunk as RepeatedBoilerplate.

RecordDocumentChunk(ulong)

Records that the session's current document contains this chunk. Called at most once per document for a given hash, so an implementation counting calls counts distinct documents.