Indexing Well

From documents to answers: what happens at ingestion, the tenant settings that decide retrieval quality, how the retrieval modes differ, and how to measure quality instead of guessing at it. Who may reach what is the Access Model; this one is about making what they reach good.

1Ingestion#

POST /lmkit/v1/search/documents indexes one document into a collection. What matters:

• Upserts are content-addressed: the outcome is Created, Replaced, or Unchanged by content hash, so re-sending a corpus is idempotent and only real changes cost work.
• Pages are extracted per format; scanned pages and images go through OCR when the tenant enables it, and OCR output is screened for recognition noise (a confidence gate, on by default) so garbage pixels never become searchable "words".
• Carry identity with the document: a display name, a source_uri pointing back at the original, and custom_metadata (a flat JSON object) for everything you will later filter or facet by. Metadata is decided at index time; retrofitting it means re-indexing.
• Expiry is first-class: ttl_seconds makes a document self-delete, for corpora with a retention policy.
• Embedding runs in the background by default; wait_for_embedding makes the call return only when the document is semantically searchable.

2Tenant settings decide quality#

All quality-relevant choices are tenant-wide, so every collection in a tenant behaves the same:

Pick the embedding model for the corpus and the queries, not for the leaderboard: dimension and model size buy quality at the cost of memory and indexing throughput, and a mid-corpus model change is a full re-embed.

3Retrieval modes, and when each wins#

• Full-text finds what the words say: exact terms, names, codes, numbers. Immune to paraphrase, blind to synonyms.
• Semantic finds what the words mean: paraphrase and synonymy, at the cost of occasionally preferring "about the same thing" over "contains the thing".
• Hybrid fuses both and is the default answer for mixed corpora and mixed queries.

On top of the mode sits the whole relevance toolbox (fusion tuning, reranking, query understanding, diversity, freshness, filters, facets, RAG-ready context), covered request by request in Querying and Relevance; and when the consumer wants an answer rather than sources, Grounded Answers runs retrieval and cited synthesis in one call.

4Scaling the index#

A tenant's vectors share pooled capacity by default, which is right for most tenants. Before bulk-indexing a LARGE tenant, reserve it dedicated capacity (the promote action on the tenant): its vector index then holds only its own data, which keeps nearest-neighbor search fast and accurate as the corpus grows. Demoting returns it to the pool without re-embedding. The full scaling story (engines, automatic promotion, background reindexing, ingestion bounds) is Storage Engines and Deployment.

5Measure, do not guess#

The evaluation endpoints make retrieval quality a number instead of an anecdote:

• Golden sets: per collection, a set of question-to-expected-source pairs; write them by hand or generate candidates from the corpus and curate.
• Evaluation runs: execute the golden set against the live index and store the scores, so a settings change (model, normalization, reranker) is judged by before-and-after numbers on YOUR corpus, newest run first.

Change one variable per run; the settings above interact. The harness and its method are Measuring Search Quality.

6Stated plainly#

• Idempotent ingestion, content-addressed: re-sending is safe and cheap.
• Quality knobs live on the tenant, and every expensive change runs in the background while search keeps serving.
• Hybrid plus reranking is the strong default; full-text alone for codes and exact language; semantic alone rarely.
• Metadata is queryable structure, not security; grants are (see the Access Model).
• Golden sets turn tuning into measurement.