Evaluation

Lists a collection's golden-set entries.#

Parameters

Responses

curl -X GET "$LMKIT_ONE_URL/lmkit/v1/search/golden-set" \
  -H "Authorization: Bearer $LMKIT_API_KEY"

Adds golden-set entries to a collection.#

A golden set is a collection's retrieval-quality reference data: (question, expected documents) pairs, optionally with a reference answer for future answer-quality metrics. The evaluation endpoint scores retrieval configurations against it with recall@k, MRR, and nDCG@k.

Request body

application/json · GoldenSetAddRequest

Responses

curl -X POST "$LMKIT_ONE_URL/lmkit/v1/search/golden-set" \
  -H "Authorization: Bearer $LMKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "cluster_id": "string",
  "tenant_id": "string",
  "collection_id": "string",
  "entries": [
    {
      "question": "string",
      "expected_document_ids": [
        "string"
      ],
      "reference_answer": "string"
    }
  ]
}'

Generates a golden set from a collection's own content.#

Bootstraps evaluation data for a corpus that has none: pages are sampled from the collection (at most one per document, ordered by a seeded hash so a run is reproducible from its reported seed), a local chat model writes one self-contained question and reference answer per page, and what passes the quality gate is stored as golden entries whose expected document is the page's document. The gate rejects questions that only make sense beside their passage ('what does this document say about...') and near-duplicates, and the per-reason discard counts are returned: a low acceptance rate is a fact about the corpus or the model, not a silent failure. It deliberately does NOT check whether retrieval currently finds the source document, since keeping only the questions the engine already answers would make every later measurement self-fulfilling. Questions already in the set take part in duplicate detection, so repeated calls grow a set. Generation runs one page at a time and can download the model, so a slow call returns a job.

Request body

application/json · GoldenSetGenerateRequest

Responses

curl -X POST "$LMKIT_ONE_URL/lmkit/v1/search/golden-set/generate" \
  -H "Authorization: Bearer $LMKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "cluster_id": "string",
  "tenant_id": "string",
  "collection_id": "string",
  "count": "string",
  "model": "string",
  "seed": "string"
}'

Deletes one golden-set entry.#

Parameters

Responses

curl -X DELETE "$LMKIT_ONE_URL/lmkit/v1/search/golden-set/$ENTRYID" \
  -H "Authorization: Bearer $LMKIT_API_KEY"

Evaluates the grounded-answer pipeline against a golden set.#

Runs every golden question through the real answer path once per configuration and has a local judge model score the result: faithfulness (the fraction of the answer's atomic statements the retrieved context supports, so the hallucination measure), context recall (the fraction of the reference answer found in that context, which separates a retrieval bottleneck from a generation one), answer relevancy, and citation precision (whether the sources the answer cited actually support it). Supply 'unanswerable_questions' to measure refusal in both directions: refusal recall alone is gamed by a pipeline that declines everything, which is why the false-refusal rate is reported beside it. A metric whose judgement could not be read excludes that question rather than scoring it zero, and says so in 'warnings'. Runs persist beside retrieval runs, so this is the headless CI hook for answer quality. Cost: several judging passes per question per configuration, bounded by 'max_questions', so a call can take minutes and returns a job when it does.

Request body

application/json · AnswerEvaluateRequest

Responses

curl -X POST "$LMKIT_ONE_URL/lmkit/v1/search/evaluate/answers" \
  -H "Authorization: Bearer $LMKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "cluster_id": "string",
  "tenant_id": "string",
  "collection_id": "string",
  "collection_ids": [
    "string"
  ],
  "judge_model": "string",
  "configs": [
    {
      "label": "string",
      "model": "string",
      "search_type": "Hybrid",
      "groundedness": "Off"
    }
  ]
}'

Evaluates retrieval configurations against a golden set.#

Runs every golden question of the selected collections through the real search read path once per configuration, scores the ranked documents against each entry's expected documents, and returns mean recall@k, MRR, and mean nDCG@k per configuration. Each configuration is persisted as one run (grouped by a shared batch id), so quality can be compared across configurations and over time via the runs endpoint. Several configurations in one call is the A/B shape: for example 'fulltext baseline' versus 'hybrid + rerank'. This endpoint is the headless hook for eval-driven development: call it from CI with plain HTTP and assert on the returned metrics. Retrieval work per call is bounded: questions times configurations searches, each at depth max(ks).

Request body

application/json · EvaluateRequest

Responses

curl -X POST "$LMKIT_ONE_URL/lmkit/v1/search/evaluate" \
  -H "Authorization: Bearer $LMKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "cluster_id": "string",
  "tenant_id": "string",
  "collection_id": "string",
  "collection_ids": [
    "string"
  ],
  "ks": [
    "string"
  ],
  "configs": [
    {
      "label": "string",
      "search_type": "FullText",
      "fusion": "Rrf",
      "rerank": false
    }
  ]
}'

Lists the tenant's persisted evaluation runs, newest first.#

Parameters

Responses

curl -X GET "$LMKIT_ONE_URL/lmkit/v1/search/evaluate/runs" \
  -H "Authorization: Bearer $LMKIT_API_KEY"