The Search Engine
This server ships a complete search engine: full-text, semantic, and hybrid retrieval with
reranking, faceting, typed metadata filtering, grounded answers, multi-turn grounded chat, and a
built-in evaluation harness, all behind one REST surface under /lmkit/v1/search. It plays the
role an external search cluster usually plays, with two differences that change the operating
story: everything runs locally (documents, indexes, and the models that embed, rerank, and
answer never leave your hardware), and the storage layer adapts to YOUR infrastructure instead
of demanding its own.
1What it is for#
One engine serves workloads that usually require separate systems:
| Workload | What the engine provides |
|---|---|
| Application search | Full-text with language-aware stemming, typed filters, facets, pagination, recency ranking |
| Semantic search | Local embedding models per tenant, nearest-neighbor retrieval, similar-document queries |
| High-quality retrieval | Hybrid fusion of both arms, cross-encoder reranking, query understanding (rewriting, expansion, HyDE) |
| RAG | Context-ready hits (chunk or page expansion), grounded answers with citations, multi-turn grounded chat sessions |
| Document corpora | The full document pipeline at ingestion: format extraction, OCR with noise screening, pagination, idempotent upserts, TTL |
| Quality engineering | Golden sets (hand-written or generated from the corpus), persisted evaluation runs |
2The shape of the system#
Data is organized along three deliberately decoupled axes: the cluster (a physical storage deployment), the tenant (the isolation and configuration boundary), and the collection (a dataset inside a tenant). Callers hold keys whose grants decide exactly which tenants they reach. Why the axes are separate, and how to map your domain onto them, is Clusters, Tenants, Collections; who may reach what is the Access Model.
Storage adapts to the infrastructure you already run: a zero-dependency embedded store that works out of the box, PostgreSQL for the full multi-tenant feature set, or SQL Server and MySQL where enterprise policy dictates the database. Engines and their trade-offs are Storage Engines and Deployment.
3Working with it, end to end#
- It already works. A fresh installation auto-provisions a local cluster, so the first tenant, collection, and document need no database setup at all.
- Provision a tenant (
POST /lmkit/v1/search/tenants) and choose its behavior: embedding model, search modes, OCR, normalization, reranker, query-understanding model. Everything quality-relevant is a tenant setting, covered in Indexing Well. - Index documents into collections: plain text, uploaded files, or base64 content in any format the document pipeline reads. Ingestion is idempotent by content hash, and embedding runs in the background so writes never wait on inference.
- Query with the mode the corpus deserves: full-text, semantic, or hybrid, plus reranking, filters, facets, and the rest of the relevance toolbox in Querying and Relevance.
- Ask instead of searching when the consumer is a person: cited grounded answers and grounded chat sessions run retrieval and synthesis on local models.
- Measure with golden sets and evaluation runs, so tuning is numbers, not anecdotes.
The operator side lives in the admin panel's Search section (clusters, tenants, telemetry) and Access section (key grants); long-running work (indexing backlogs, reindexes) surfaces in Jobs.
4The chapters#
| Sub-topic | The question it answers |
|---|---|
| Clusters, Tenants, Collections | Why the three axes are decoupled, and how to map your domain and isolation needs onto them |
| Storage Engines and Deployment | Which storage backend fits your infrastructure, and how the index scales with volume |
| Indexing Well | Ingestion, OCR, tenant settings, and the choices that decide retrieval quality |
| Querying and Relevance | The retrieval modes and the full relevance toolbox, request by request |
| Access Model | Keys, grants, discovery, and the security properties the API guarantees |
| Grounded Answers | Turning retrieval into cited answers and multi-turn grounded chat |
| Measuring Search Quality | Golden sets and evaluation runs: judging changes by numbers on your own corpus |
5Stated plainly#
- One REST surface covers application search, semantic retrieval, RAG grounding, and quality measurement; nothing about it requires an external search cluster or a cloud service.
- The storage layer adapts to your infrastructure, from a single embedded file to the databases your organization already operates.
- Every capability is governed by the same key model as the rest of the server, and every quality claim can be checked with the built-in evaluation harness.