Parsing Effort for RAG

Retrieval can only find what indexing read. Every document a tenant indexes is read by the document parser, and the tenant's parsing effort decides how: from the text the document carries, with no model, or with the parser's vision model, which reads tables, charts and headings the way a person sees them. This guide is the one place that explains the levels; the rest of ingestion is in Indexing Well, and the parser itself in Document Parsing.

1The four levels#

What the parser does inside each model level belongs to the parser and improves from one release to the next; a tenant's setting does not change when it does.

Whatever the level, a document is never slower to become searchable: it is always indexed at once at Minimal, and a model level improves its pages afterwards (section 2). Minimal is the reading every tenant had before the setting existed, word for word.

What a model level buys in a RAG answer: tables come back as tables (cells in their rows and columns, merged headers kept) instead of runs of numbers, a chart becomes the data table it plots, multi-column pages read in order, and headings are recognized from the page's typography rather than guessed, so chunks start and end where the document's sections do.

2What happens when a document is indexed#

1. At once, at every level: the document is read at Minimal and written. The call returns with the document full-text searchable (and semantically searchable once its embedding is done, as always).
2. At Low, Medium or High: the server keeps the document's source and a background worker parses it again at the tenant's level, one document at a time, through the same completion slots as every request that runs a model.
3. When the parse finishes: the document's pages are replaced by the parsed ones, its metadata and expiry kept, and it is embedded again. Search keeps answering from the Minimal pages until then.

The background parse respects the present, not the moment the document was queued:

• A document indexed again while it waits is parsed from its newest source; the older parse is dropped.
• A document deleted, or expired, is never parsed.
• A tenant set back to Minimal stops parsing; what was already parsed stays.
• A document whose parse fails is retried three times, then stays at Minimal, still indexed.

Word documents, HTML pages, e-mails and plain text are read from their own structure at Minimal and are not parsed again: the model would read a rendering of what the file already states. PDFs, scans and images are.

3Which level for which corpus#

• Born-digital PDFs, Office files, e-mail, at volume: Minimal. The text is already in the files; nothing a model reads would change the words, only the structure of tables and charts.
• Scans, photos of documents, faxes: Low at least. OCR gives Minimal the words, but not the tables or the reading order of a complex page.
• Reports, filings, papers with tables and charts that answers must quote: Medium, or High when every figure matters.
• No GPU: stay at Minimal, or accept that a model level reads in minutes per page on the processor. The backlog then grows faster than it drains on a busy tenant (section 6 shows how to watch it; section 4 how to time a page first).

Two tenants are the way to run two levels: a level is tenant-wide, so a corpus that needs High for one collection and Minimal for another is two tenants.

4Evaluate the levels in the Playground first#

The Playground reads a document exactly as a tenant would, so the level can be chosen on your own documents before a single one is indexed. Playground > Parse runs the same parser, with the same server OCR engine, as search indexing and its background parse.

How the Playground maps to a tenant

Markdown converter, Fast + OCR reads the same words as Document parser, Minimal: it is the reading Minimal is built from, without the elements.

What is indexed is what the Markdown tab shows, page by page: its words feed full-text search (table cells included, never table tags) and its chunks the embeddings. The Reading view shows why a page reads the way it does: each element, its box on the page and its place in the reading order.

The Playground reads with the server's own settings; three tenant or collection settings can make a tenant's reading differ, and only on scanned pages or photos:

• A collection language (its full-text configuration, such as french): the tenant gives OCR that language as a hint, the Playground lets the OCR engine choose. Born-digital pages read the same either way.
• Reject photo noise turned off on the tenant: the Playground screens noise as the server is configured to (on by default), so a photo's stray "words" appear in the tenant's reading only.
• OCR turned off on the tenant: use Markdown converter, Fast (first table).

Figure images changes nothing in the indexed text; leave it as you like. A model level reads the same document the same way in the Playground and in the background parse; a reading made while the device serves many requests at once can differ in rare details.

Compare two levels on one document

1. Open Playground > Parse and drop a document typical of the corpus (or start from a sample). Pick Document parser and the level the tenant has now (usually Minimal), then Parse.
2. Select Compare: the Playground parses the same document at High (at Low when the first parse was High) and shows both pages side by side, with what changed, element by element. Each side switches to any level already run on that document; run Low or Medium first to add them.
3. Read what answers will quote: tables (cells in their rows and columns), charts (a data table, or nothing), headings, the order of multi-column pages, and text on scanned pages.
4. Note the time each parse shows, in total and per page: at a model level that is what every document costs the background parse, so it says how fast a backlog drains on this server.

When a model level reads your documents no better than Minimal, stay at Minimal: it costs nothing and is what the tenant reads already. Get code shows the parse on screen as an SDK call or a REST request, for a pipeline that parses outside search.

5Set the level#

When the tenant is provisioned:

curl -s "$LMKIT_URL/lmkit/v1/search/tenants" \
  -H "Authorization: Bearer $LMKIT_API_KEY" -H "Content-Type: application/json" \
  -d '{ "display_name": "Contracts", "enable_semantic_search": true, "parsing_effort": "High" }'

On an existing tenant, with the same update every tenant setting uses (omitted fields are left as they are; local is the built-in cluster, GET /lmkit/v1/search/clusters lists yours):

curl -s -X PUT "$LMKIT_URL/lmkit/v1/search/clusters/local/tenants/$TENANT" \
  -H "Authorization: Bearer $LMKIT_API_KEY" -H "Content-Type: application/json" \
  -d '{ "parsing_effort": "Medium" }'

The tenant answers with parsing_effort and parsing_backlog. In the admin panel the level is Search > (tenant) > Settings > Parsing, and the tenant's setup card shows it beside the number of documents still waiting.

A new level applies to documents indexed from then on. To bring documents already indexed to it, send them again without their content_hash (or with a new one): a document sent with the hash it was indexed with is reported Unchanged and keeps its reading, which is what makes a corpus sync cheap the rest of the time.

OCR stays a tenant setting of its own (enable_ocr, on by default) because every level reads a scanned page with OCR first, at Minimal. A tenant that turned it off indexes scans as empty pages at Minimal; a model level still reads them in the background.

6Watching the background parse#

parsing_backlog on the tenant counts the documents still waiting, and the worker appears on the admin Scheduler page as search-parsing, with its last pass. The parser's model (lmkit-doc-parser:1) downloads on first use like any model, visibly in the admin panel; a server that cannot download it (no network, no room) pauses the parse and retries every ten minutes, the documents staying searchable at Minimal meanwhile, and the server log says why. An air-gapped server pulls it beforehand:

lmkit pull lmkit-doc-parser:1

On a farm. The source of a document waiting for its parse is kept on the node that indexed it, and that node parses it: parsing_backlog counts the documents of the node that answers, and each node drains its own. The pages it writes are shared like every indexed page, so every node serves the parsed document. See Scaling Out for what is shared and what is per node.

7Stated plainly#

• Minimal reads exactly what indexing read before parsing efforts existed: same words, same page layout, same OCR flag. Moving a tenant to a model level changes the documents it indexes next, never the speed of indexing.
• A model level never makes a document wait to be searchable; it makes its pages better later.
• Tables reach the index as HTML, so their spans survive; full-text search indexes their cells, never their tags.
• The level is per tenant, and so is the backlog. The same levels exist on POST /lmkit/v1/document-parsing and in Playground > Parse (section 4), which reads a document exactly as the tenant will.