Structured Extraction

POST /lmkit/v1/extract-structured-data reads a document (text, PDF, HTML, EML, MBOX, Office, or image, OCR included) against a JSON Schema you define and returns the filled JSON. What makes it production-grade rather than a demo is everything that comes WITH the JSON: per-field confidence, validation status, source coordinates, and an explicit needs-review flag. This guide covers the workflow: designing schemas that extract well, reading the result, and building the review loop the flags are designed for. The schema language itself, every keyword and its enforcement guarantee, has its own chapter: The Extraction Schema Reference.

1The contract#

Send the document, a JSON Schema for the shape you want (jsonSchema), and optional free-text guidance steering interpretation (what "amount" means in your domain, which date wins when two appear). The response carries:

• json: the extracted data, shaped exactly by your schema.
• elements: per-field detail keyed by JSON path: the extraction confidence, a validation status, the entity_kind recognized, the original_value as it appeared in the document, and where it came from: page_index plus bounds as a quadrilateral (four corner points, so rotated scans are located honestly rather than forced into an upright rectangle).
• human_verification_required: the document-level flag that says whether any field fell below the trust bar; the same flag exists per element.
• pages: page dimensions, so coordinates can be drawn over a render from the thumbnail endpoint.

2Designing schemas that extract well#

• Constrain everything constrainable. Enums for anything categorical, formats for dates and numbers, patterns for references, entry caps on arrays: every constraint removes a way to be wrong, and most compile into the generation grammar where violation is impossible (the schema reference states each keyword's guarantee).
• Name fields the way the documents do. A schema that speaks the corpus's own vocabulary ("invoice_number", not "ref") needs less guidance and misreads less.
• Extract, do not compute. Ask for what is ON the page (line items, unit prices) and compute derived values (totals, deltas) in code where arithmetic is free and auditable; reserve guidance for genuine interpretation rules.
• Mark truly-optional fields optional. A required field that is genuinely absent from a document forces a choice between a low-confidence guess and a validation failure; optional fields let absence be absence.

3Confidence is the product#

The design assumption is that no extractor is 100% right, so the output tells you WHERE to look instead of asking you to trust:

1. Route on the flag. human_verification_required false: straight through to the database. True: into a review queue. That single branch is the difference between a pipeline you audit by exception and one you babysit.
2. Review with coordinates. The per-field bounds point a reviewer's eye at the exact spot on the page; a review UI that renders the page and highlights the doubtful fields turns review from re-reading into confirming, and original_value shows what the document actually said next to what was extracted.
3. Tune the loop, keep the flags honest. When a field is repeatedly doubted for the same reason, fix the schema or the guidance rather than lowering your standards; the flag's value IS its honesty.

4The batch pipeline#

Extraction is the third pillar of the IDP pipeline, and the classic mailroom flow assembles from this category's own chapters:

1. Split the scanned batch into its logical documents; with split_documents: true, each segment arrives as its own file id.
2. Classify each segment against your taxonomy (invoice, contract, ID, correspondence).
3. Extract with the schema matching its class, passing the segment's file id straight in.
4. Route on human_verification_required; reviewed and clean records land in your system.

Every step is an API call; the only human in the loop is the one the flags summon. For document classes you process at real volume, fine-tuning a compact model on your own corrected extractions is the natural next investment.

5Stated plainly#

• The output is not "JSON from a model"; it is JSON plus confidence, validation, provenance coordinates, and a review flag per field: the metadata a production pipeline actually runs on.
• Schema constraints and domain guidance are the two levers; spend on them before spending on a bigger model.
• Route on the review flag from day one: exception-driven review scales, proofreading does not.