The IDP Pipeline

Intelligent document processing turns unstructured documents into structured, validated, system-ready data. The discipline rests on three pillars: splitting (finding the documents inside a batch), classification (deciding what each one is), and extraction (reading its fields into your schema). This server implements all three as first-class endpoints, plus every stage around them: capture, OCR, validation, review routing, redaction, archival, and search. Each stage is a stateless API call over a shared file identifier, so any pipeline you can draw on a whiteboard is a sequence of calls you can make today, entirely on hardware you control.

1The canonical pipeline#

The flow that defines the category, mapped call by call:

Straight-through processing is the goal and the flags are the mechanism: a document whose fields extract confidently and validate cleanly needs no human at all, and the one that does not is routed, not silently wrong.

2Why these three pillars carry the discipline#

• Splitting is the gate. Real input arrives as batches: a mailroom scan, a claims dossier, an email with four attachments. Nothing downstream can be right about a "document" that is actually three. Intelligent Splitting finds boundaries by reading content, not counting pages, and can hand back each segment as its own file, ready to chain.
• Classification is the router. The document's class decides everything after it: which extraction schema, which business queue, which retention rule. Document Classification takes your taxonomy at request time, so a new document class is a request change, not a training project.
• Extraction is the payoff. Structured Extraction is where paper becomes data, and its schema dialect compiles your constraints into generation itself, so output that violates the schema is not merely unlikely: it is unrepresentable.

Every stage between and around them (OCR, conversion, thumbnails for review UIs, validation, redaction, archival, search) ships as endpoints in the same API, documented across this category and Documents.

3The classic pipelines, assembled#

Every one of these is a composition of the calls above, running locally, and the flagship is implemented request by request in the cookbook: Invoice Automation, End to End.

• Mailroom automation: split the daily scan batch, classify each segment, extract with the class's schema, route exceptions to review. The founding IDP use case, four endpoints.
• Invoice and AP automation: extract header, line items, and totals with entity validation on IBAN, VAT, and amounts; review only what the flags raise; deliver to the ERP.
• Claims processing: split the dossier (form, photos, correspondence, reports), extract per document class including photographs of documents, and assemble the claim record.
• KYC and onboarding: classify submitted documents against the required set, extract identity fields with validators on names, dates, and identifiers, and flag what a human must see.
• Contract intake: split contracts from annexes, extract parties, dates, amounts, and clause presence, then summarize for the reviewer and index for search.
• Email-driven intake: EML and MBOX are native formats, so an inbox export processes like any other document: split the thread from its attachments, classify, extract.
• Compliance and archival: detect PII, review, then destroy it with true redaction; convert the record to PDF/A and validate the conformance claim, storing the report as evidence.
• Knowledge and retrieval: processed documents feed Search with classification and keywords as filterable metadata, so the archive answers questions instead of holding files.

If your pipeline is not on this list, it is still built the same way: the stages are independent calls with no hidden coupling, so they compose in any order, branch on any condition your code can express, and run entirely inside your network.

4Operating IDP at volume#

• Scale is a setting. Stages parallelize across the slot pool; saturation behavior (queue or shed) is your choice, and long documents ride the jobs contract rather than blocking.
• Models are per stage. Classification and splitting run well on compact, fast models; extraction earns a stronger one. Set per-capability defaults once, or pin a model per request.
• The corpus is the moat. Corrected outputs from your review queue are training data: fine-tune a compact model on your own document classes and the accuracy compounds where generic models plateau.

5Stated plainly#

• Split, classify, extract: the three pillars are first-class endpoints, not features hidden inside a suite.
• Every classic IDP pipeline (mailroom, AP, claims, KYC, contracts, email intake, compliance archival) assembles from the same stateless calls over shared file ids.
• Confidence, validation, and review flags make straight-through processing honest: the pipeline tells you which documents it is sure about, and hands you the rest.