Table of Contents

Method NormalizeForEmbedding

Namespace
LMKit.Retrieval.ChunkQuality
Assembly
LM-Kit.NET.dll

NormalizeForEmbedding(string)

The normalized embedding view of a chunk, exactly as the gate computes it before scoring: HTML tags and entities, markdown markers, table rails, code fences, link URLs, metadata tokens ([cid:…], [image:…]) and over-long non-word tokens are removed; letter-spaced words are rejoined; whitespace folds. Use it wherever text reaches an embedding model WITHOUT the full gate - markup embeds as markup, not meaning, and poisons the vector space.

public static string NormalizeForEmbedding(string text)

Parameters

text string

The candidate chunk text (Markdown, HTML-bearing Markdown, or plain text).

Returns

string

The markup-free text to embed; empty for null or empty input.

Share