Measuring Search Quality
Search tuning without measurement is folklore: someone changes a setting, tries three queries, and declares it better. The engine ships an evaluation harness so a change is judged by numbers on YOUR corpus: golden sets of question-to-source pairs, evaluation runs over the live index, and persisted results for before-and-after comparison.
1Golden sets#
A golden set is a per-collection list of entries pairing a realistic question with the document(s) that should answer it. Two ways to build one:
- Write entries by hand (
POST /lmkit/v1/search/golden-set): the highest-value entries come from real user questions, especially ones the system once failed. - Generate candidates from the corpus (
POST /lmkit/v1/search/golden-set/generate): a local model reads the collection's own content and drafts question/source pairs. Generation bootstraps coverage in minutes; CURATE the result, because a model asks the questions the text suggests, not necessarily the ones your users ask.
Twenty to fifty curated entries per collection is a working harness; grow it whenever a real query fails and gets fixed, so regressions of that fix become visible forever.
2Evaluating retrieval#
POST /lmkit/v1/search/evaluate runs the golden set against the live index under a named
retrieval configuration (mode, reranking, fusion settings) and scores whether the expected
sources come back, and how high they rank. Because the configuration is part of the request,
one golden set arbitrates every choice in
Querying and Relevance: full-text vs hybrid, reranking on or off,
fusion weights, a different embedding model after a
re-embed.
The discipline that makes results meaningful:
- Change one variable per run. The settings interact; a run that changes the embedding model AND enables reranking attributes its delta to nothing.
- Name runs after their intent, so the history reads as an experiment log.
- Keep the golden set stable while comparing. Growing the set between runs changes the exam; grow it between EXPERIMENTS, not between the A and the B.
3Evaluating answers#
POST /lmkit/v1/search/evaluate/answers runs the same golden set through the full
grounded-answer pipeline and judges the ANSWERS: grounding in the
expected sources, citation quality, and refusal behavior. Retrieval evaluation says "the right
documents come back"; answer evaluation says "what the user reads is right". When answers
disappoint, run both: the pair tells you whether retrieval or synthesis is the weak link, which
is the single most time-saving diagnostic in RAG work.
4Runs are records#
Every evaluation persists (GET /lmkit/v1/search/evaluate/runs, newest first), so quality has
a history: what was measured, under which configuration, scoring what. That history is the
honest answer to "did the upgrade regress search" and the evidence base for tuning decisions
made months apart by different people.
5Stated plainly#
- A golden set turns "feels better" into a number; generation bootstraps it, curation makes it yours, real failed queries keep it honest.
- Evaluate retrieval and answers separately: the pair localizes every quality problem to one half of the pipeline.
- One variable per run, named runs, stable golden set: the harness is only as good as the method.