Intelligent Splitting

A scanned batch is rarely one document. The mailroom PDF is thirty invoices, the claims dossier is a form plus photos plus correspondence, the loan file is a contract wearing its annexes. POST /lmkit/v1/document-splitting finds the boundaries by READING the pages, the way a person sorting the stack would: a new letterhead, a restarted page numbering, a change of subject mid-batch. Layout tricks alone miss what content makes obvious, which is why the detection is model-driven. Splitting is the gate of the IDP pipeline: everything downstream is wrong about a "document" that is actually three.

1The contract#

Send a PDF (base64 or a file id); the response is the batch's logical structure:

• segments: one entry per detected document, each with start_page and end_page (1-based, inclusive), page_count, and a label: a descriptive name the model assigns from the content ("Invoice", "Employment Contract"), a head start for classification and for naming review queues.
• document_count and contains_multiple_documents: the batch-level answer, usable as a cheap guard ("route single documents straight through, split the rest").
• confidence: how sure the detection is, so borderline batches can be routed to a human sorter instead of silently mis-split.

An optional guidance string states domain rules the pages alone cannot carry: what counts as one document in YOUR workflow ("statements and their remittance advice belong together", "each purchase order starts a new document").

2Physical splitting: segments as files#

With split_documents: true, the server does the surgery itself: each segment is written as its own PDF and returned with a file_id. Those ids are first-class citizens of the whole API:

• feed one straight into classification, extraction, Markdown conversion, summarization, or search indexing by passing it as input with input_format: "FileIdentifier";
• download it (GET /lmkit/v1/files/{file_id}) or delete it (DELETE /lmkit/v1/files/{file_id}); undeleted files expire with the server's configured retention period.

This is the difference between a boundary REPORT and a working pipeline stage: no client-side PDF manipulation, no page-range bookkeeping, no bytes crossing the wire twice. Split once, then chain every segment through the rest of the pipeline by id.

3Where splitting earns its keep#

• The mailroom batch: one scanner pass over the day's mail, one call, thirty labeled documents ready for triage.
• The dossier: claims files, loan applications, and HR packets arrive as one PDF with predictable-in-hindsight structure; splitting recovers it, and per-segment labels tell the router what each piece is before classification even runs.
• The append-only archive: legacy systems that merged everything into one PDF per case can be un-merged at scale, segment by segment, into a searchable, per-document archive.
• Email exports: an EML or MBOX file processed by the document pipeline brings its attachments along; splitting separates the cover message from what it carried.

4Practical notes#

• Scanned input is expected input. OCR runs inside ingestion, so a batch of pure images splits as readily as digital PDFs; noise screening keeps photographic pages from reading as text.
• Long batches are windowed. Detection reads the whole batch in overlapping windows and merges the boundary decisions, so batch length is not a schema constraint you manage.
• Model choice is a throughput lever. Boundary detection is a constrained task where compact models perform well; set the default in Models and escalate only if your batches measurably confuse it. Past the configured timeout the endpoint returns 202 Accepted with a job to poll, per the jobs contract.
• Splitting composes with page surgery. For FIXED page rules (every 2 pages, extract pages 5-8), the toolbox's deterministic split is the right tool; intelligent splitting is for boundaries only the content reveals.

5Stated plainly#

• Boundaries are detected by reading content, and each segment comes back labeled, with a confidence you can route on.
• split_documents: true turns detection into delivery: per-segment files, by id, ready for the next stage without touching a PDF library.
• Splitting first, everything else after: it is the stage that makes the rest of the IDP pipeline correct.