Class DocumentParser

Namespace: LMKit.Document.Parsing

Assembly: LM-Kit.NET.dll

Parses documents into pages of positioned, categorized and typed elements (text, headings, tables, formulas, figures and the data of the charts they plot) in reading order, and renders them as Markdown, JSON, HTML and DocLang.

public sealed class DocumentParser : IDisposable

Inheritance: object

DocumentParser

Implements: IDisposable

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Examples

using LMKit.Document.Parsing;

using var parser = new DocumentParser(ParsingEffort.Medium);

ParsedDocument document = parser.Parse("report.pdf");
File.WriteAllText("report.md", document.ToMarkdown());
File.WriteAllText("report.json", document.ToJson());
File.WriteAllText("report.html", document.ToHtml());

foreach (ParsedPage page in document.Pages)
{
    foreach (ParsedElement element in page.Elements)
    {
        Console.WriteLine($"{element.Category} at {element.Bounds}: {element.Content.Text}");
    }
}

Remarks

A parser is created for one ParsingEffort level, the trade-off between the time a document takes and the fidelity of the result; every other decision belongs to the engine.

Model. The parser runs on the one model LM-Kit tests it with, a single archive that bundles everything it reads with (GetModelCards(ParsingEffort)). Created without a model, it loads it from the model catalog, downloading it on first use into ModelStorageDirectory, and releases it when it is disposed. A caller that ships the file itself passes its path (DocumentParser(ParsingEffort, string)), and one that already holds the model passes it loaded (DocumentParser(ParsingEffort, LM)); the parser checks it and refuses any other model with an InvalidModelException.

Threading. One parser can serve several documents at once from several threads.

Results. A parse (ParsedDocument) renders as Markdown for LLM and RAG ingestion, lossless JSON that FromJson(string) reads back, semantic HTML, and DocLang. IncludeFigureImages keeps the picture of every figure, PageParsed reports progress, and the page-range overloads of Parse read part of a document.

For fast Markdown across office and email formats, or to run a vision model of the caller's choice, DocumentToMarkdown is the universal converter.

Constructors

DocumentParser(): Creates a parser at the Medium effort level, on the model it loads from the model catalog.

DocumentParser(ParsingEffort): Creates a parser at the given effort level, on the model it loads from the model catalog.

DocumentParser(ParsingEffort, LM): Creates a parser at the given effort level on the model the caller has already loaded, for instance to share it with other components.

DocumentParser(ParsingEffort, string): Creates a parser at the given effort level on the model file the caller provides, for a deployment that ships the model itself instead of downloading it.

Properties

Effort: The effort level the parser was created at.

IncludeFigureImages: Whether the parse keeps the picture of every figure and chart, cut from the page as it renders. Off by default.

MaximumImageDetail: The most pixels a reading model is given for a page, a figure or a second reading. null (the default) decides from the hardware.

OcrEngine: The engine that reads the words a page's text layer does not carry: the pages and pictures of a scan under Minimal, and the words inside figures at every level. Defaults to a built-in LMKitOcr the parser creates on first use and releases with itself.

OcrLanguages: The languages the documents are written in, given to OcrEngine when it reads a whole page under Minimal. null (the default) leaves the choice to the engine.

Methods

Dispose(): Releases the models and the resources the parser loaded itself. The models given at construction stay with their owner.

GetModelCards(ParsingEffort): The model the parser runs on at an effort level, as an entry of the model catalog (none at Minimal, which reads with no model): what a caller passes to DocumentParser(ParsingEffort, string) or, loaded, to DocumentParser(ParsingEffort, LM), or downloads ahead of time for a machine that parses offline (Download(ModelDownloadingProgressCallback)).

Parse(Attachment, string, CancellationToken): Parses the pages of document that pageRange selects.

Parse(Attachment, CancellationToken): Parses every page of document.

Parse(string, string, CancellationToken): Parses the pages of the document at path that pageRange selects.

Parse(string, CancellationToken): Parses the document at path.

ParseAsync(Attachment, string, CancellationToken): Asynchronously parses the pages of document that pageRange selects.

ParseAsync(Attachment, CancellationToken): Asynchronously parses every page of document.

ParseAsync(string, string, CancellationToken): Asynchronously parses the pages of the document at path that pageRange selects.

ParseAsync(string, CancellationToken): Asynchronously parses the document at path.

Events

PageParsed: Raised each time a page has been read, on the thread that parses, with the page and the progress of the parse: a progress bar, a log line, or an estimate of the time left.

PageProgress: Raised while a page is read, on the thread that reads it: the stage the page is at and, while the page model reads, the characters it has written, at most every 150 milliseconds. A progress bar moves with it between two PageParsed.

Table of Contents