Table of Contents

👉 Try the demo: https://github.com/LM-Kit/lm-kit-net-samples/tree/main/console_net/vision/vlm-ocr/vlm_ocr_with_coordinates

VLM OCR with Text Coordinates for C# .NET Applications


🎯 Purpose of the Demo

VLM OCR with Coordinates demonstrates how to use LM-Kit.NET with vision-language models to detect positioned regions in images and documents, then draw the detected regions onto annotated output images. Depending on the selected model, the demo performs line-level text spotting (PaddleOCR VL) or full semantic layout analysis (Infinity-Parser2), where every region is classified as title, text, table, formula, figure, caption, header, or footer and drawn with one color per category.

The sample shows how to:

  • Download and load a vision model that supports coordinate output (PaddleOCR VL, Infinity-Parser2).
  • Query VlmOcr.GetSupportedIntents and pick the richest spatial intent the model supports (VlmOcrIntent.LayoutAnalysis when available, VlmOcrIntent.OcrWithCoordinates otherwise).
  • Feed images or multi-page PDFs as Attachment objects.
  • Iterate over the returned TextElement instances with pixel-accurate bounding boxes and a semantic Category.
  • Draw category-colored bounding boxes onto the original image using the Canvas drawing API.
  • Save the annotated image as a PNG file.

Why VLM OCR with Coordinates?

  • Location-aware extraction: know where each region sits on the page, not just what it says.
  • Semantic layout understanding: with a document-parsing model, each region carries its layout category, and its content is typed accordingly (tables carry HTML, formulas carry LaTeX, other categories carry Markdown).
  • Automatic coordinate translation: LM-Kit.NET maps the model's normalized grid coordinates back to source image pixels through the preprocessing transform chain.
  • Visual verification: draw bounding boxes to validate detection quality before integrating into production pipelines.
  • Document format support: the same code path handles single images and multi-page PDFs.

👥 Target Audience

  • Document Processing: workflows that need spatial information (form field extraction, zone-based reading order, region classification).
  • Quality Assurance: visually verify OCR accuracy by overlaying bounding boxes on the source image.
  • Data Labeling: generate bounding-box annotations for training or review without manual annotation tools.
  • RPA and Back-office: locate specific fields on invoices, receipts, or forms by position.

🚀 Problem Solved

  • Know where text lives on the page: plain text OCR loses spatial information. This demo preserves region positions.
  • Visual debugging: immediately see which regions the model detected by looking at the annotated image.
  • Multi-page documents: process each page of a PDF individually, with per-page annotated output.
  • Model flexibility: the model selection menu is structured for future expansion as more engines add coordinate support.

💻 Sample Application Description

Console app that:

  • Lets you choose a vision model that supports coordinate output (PaddleOCR VL for line spotting, Infinity-Parser2 for layout analysis) or paste a custom model URI.

  • Downloads the model if needed, with live progress updates.

  • Picks the richest spatial intent the model supports via VlmOcr.GetSupportedIntents and announces the active mode.

  • Repeatedly asks you for a file path (image or PDF), then:

    • Creates a VlmOcr instance with the resolved intent.
    • Loads the file as an Attachment.
    • Runs OCR page-by-page via ocr.Run(attachment, pageIndex).
    • For each page, prints every detected region with its category tag (colored), a content preview, position, and size.
    • Draws category-colored bounding boxes on the image using Canvas and Pen, and prints a matching color legend.
    • Saves the annotated image next to the original file (e.g. document_annotated.png).

  • Displays a stats block (elapsed time, tokens, quality, speed, context usage).

  • Loops until you type q to quit.

✨ Key Features

  • 📍 Coordinate extraction: each region includes Left, Top, Width, Height in source image pixels.

  • 🏷️ Layout categories: with Infinity-Parser2, each region carries a TextElement.Category (title, text, table, formula, figure, captions, footnotes, header, footer). Figure regions are reported with their bounding box even though they carry no text.

  • 🖼️ Annotated output: bounding boxes drawn directly on the image with the Canvas API, one color per category.

  • 📄 Image + PDF support: images are loaded directly; PDF pages are rendered before annotation.

  • 📑 Multi-page aware: each page of a multi-page document gets its own annotated image (e.g. contract_page1_annotated.png).

  • 📊 Telemetry:

    • Elapsed time (seconds)
    • Generated tokens count
    • Stop reason
    • Quality score
    • Token generation rate
    • Context tokens vs context size

  • 📦 Model lifecycle:

    • Automatic download on first use.
    • Loading progress shown in the console.

  • ❌ Nice errors: friendly message when a file path is invalid or the annotated image cannot be saved.


🧰 Built-In Models (menu)

On startup, the sample shows a model selection menu:

Option Model Approx. VRAM Needed Spatial Output
0 PaddlePaddle PaddleOCR VL 1.6 0.9B ~1 GB VRAM Text-line spotting
1 INF Tech Infinity-Parser2 Flash 2B ~2 GB VRAM Full layout analysis with categories
other Custom model URI (GGUF / LMK, etc.) depends on model model-dependent

Only models that support bounding-box coordinate output are listed. The menu will grow as more engines add this capability. Any input other than a listed index is treated as a custom model URI and passed directly to the LM constructor.


🧠 How Coordinate Translation Works

Each engine encodes spatial information its own way:

  • PaddleOCR VL emits eight <|LOC_nnn|> tokens per text region (four corners, each with an X and Y coordinate on a normalized 0..999 grid).
  • Infinity-Parser2 runs its native doc2json task and returns a JSON array of layout elements, each with a bbox on a normalized 1000-step grid, a category, and typed content (HTML for tables, LaTeX for formulas, Markdown otherwise), sorted in human reading order.

LM-Kit.NET translates both through the same two steps:

  1. Normalized grid to processed image pixels. The grid values are denormalized against the content dimensions of the image that was actually fed to the model.
  2. Processed image pixels to source image pixels. The preprocessing transform (crop, scale) is reversed so the final coordinates match the user's original image.

This happens automatically inside VlmOcr when the intent is OcrWithCoordinates or LayoutAnalysis. The result is a PageElement populated with TextElement instances whose Left, Top, Width, and Height are expressed in source image pixels, and whose Category carries the model's layout classification when available. Under LayoutAnalysis, the raw machine-readable JSON payload additionally remains accessible through VlmOcrResult.NormalizedText.


🛠️ Commands and Flow

Inside the console loop:

  • On startup

    • Select a model (0) or paste a custom model URI.
    • The model is downloaded (if needed) and loaded with progress reporting.

  • Per document (image or PDF)

    • The app prompts: enter image or document path (or 'q' to quit):

    • Type a file path and press Enter.

    • The app loads the file into an Attachment.

    • The app iterates pages:

      • For images, this is typically 1 page.
      • For PDFs, this can be N pages.

    • For each page, OCR runs and prints:

      • Each detected text region with [index] "text" and Position: (x, y) Size: w x h
      • The total number of detected regions
      • The path to the annotated output image
      • A Stats section

  • Quit

    • At any prompt, typing q exits the app cleanly.

🗣️ Example Use Cases

Try the sample with:

  • A scanned invoice to locate line items, totals, and dates by position.
  • A multi-page PDF contract to find where signature blocks and clause headings appear.
  • A phone-captured photo of a receipt to verify which text regions the model detects.
  • A form or ID card to identify field positions for downstream extraction.

After each run, inspect:

  • The annotated image to verify bounding-box accuracy.
  • The console output for exact pixel coordinates of each region.
  • The quality score and token count to assess detection completeness.

💻 Minimal Integration Snippet

using LMKit.Data;
using LMKit.Document.Layout;
using LMKit.Extraction.Ocr;
using LMKit.Graphics.Drawing;
using LMKit.Graphics.Geometry;
using LMKit.Graphics.Primitives;
using LMKit.Media.Image;
using LMKit.Model;

// Load a document-parsing model with native layout analysis
using LM model = LM.LoadFromModelID("infinity-parser2-flash");

// Create OCR engine with full layout analysis
var ocr = new VlmOcr(model, VlmOcrIntent.LayoutAnalysis)
{
    MaximumCompletionTokens = 8192
};

// Run OCR
var attachment = new Attachment("document.png");
VlmOcr.VlmOcrResult result = ocr.Run(attachment);

// Iterate layout regions: each carries a category and typed content
// (tables carry HTML, formulas carry LaTeX, figures carry no text)
foreach (TextElement element in result.PageElement.TextElements)
{
    Console.WriteLine($"[{element.Category}] \"{element.Text}\"  " +
                      $"at ({element.Left:F1}, {element.Top:F1})  " +
                      $"size {element.Width:F1} x {element.Height:F1}");
}

// Draw bounding boxes on the image, one color per category
static Color32 ColorFor(LayoutElementCategory category) => category switch
{
    LayoutElementCategory.Title => new Color32(147, 51, 234),
    LayoutElementCategory.Table => new Color32(22, 163, 74),
    LayoutElementCategory.Formula => new Color32(13, 148, 136),
    LayoutElementCategory.Figure => new Color32(249, 115, 22),
    _ => new Color32(37, 99, 235),
};

using ImageBuffer image = ImageBuffer.LoadAsRGB("document.png");
var canvas = new Canvas(image) { Antialiasing = true };

foreach (TextElement element in result.PageElement.TextElements)
{
    canvas.DrawRectangle(
        Rectangle.FromSize(element.Left, element.Top, element.Width, element.Height),
        new Pen(ColorFor(element.Category), 2) { LineJoin = LineJoin.Miter });
}

image.SaveAsPng("document_annotated.png");

🛠️ Getting Started

📋 Prerequisites

  • .NET 8.0 or later

📥 Download

git clone https://github.com/LM-Kit/lm-kit-net-samples
cd lm-kit-net-samples/console_net/vision/vlm-ocr/vlm_ocr_with_coordinates

Project Link: vlm_ocr_with_coordinates (same path as above)

▶️ Run

dotnet build
dotnet run

Then:

  1. Select a vision model by typing 0, or paste a custom model URI.
  2. Wait for the model to download (first run) and load.
  3. When prompted, type the path to an image or document file (or q to quit).
  4. Inspect the detected text regions with coordinates in the console.
  5. Open the annotated image saved next to the original file.
  6. Press Enter to process another file, or q to exit.

🔍 Notes on Key Types

  • VlmOcr (LMKit.Extraction.Ocr): OCR engine built on top of a vision model.

    • Construct with new VlmOcr(model, VlmOcrIntent.LayoutAnalysis) for full layout analysis, or VlmOcrIntent.OcrWithCoordinates for positioned text spotting.
    • GetSupportedIntents(model) reports which intents the loaded model natively serves.
    • Run(Attachment, pageIndex) returns a result with PageElement containing TextElement instances.

  • TextElement (LMKit.Document.Layout): a recognized region with spatial information.

    • Text: the recognized content; its format depends on the category (HTML for tables, LaTeX for formulas, Markdown or plain text otherwise).
    • Category: the semantic layout category (LayoutElementCategory enum: title, text, table, formula, figure, captions, footnotes, header, footer), or Unknown when the source does not classify regions.
    • Left, Top: top-left corner position in source image pixels.
    • Width, Height: bounding box dimensions in source image pixels.

  • Canvas (LMKit.Graphics.Drawing): a fluent drawing surface that wraps an ImageBuffer.

    • DrawRectangle(Rectangle, Pen) renders an outline on the underlying image.
    • All drawing is immediate and modifies the image in place.

  • Pen (LMKit.Graphics.Drawing): defines stroke color, width, and line join style.

  • Attachment (LMKit.Data): wraps external data (images or documents).

    • PageCount exposes the number of pages (images are typically 1; PDFs can be many).
    • RenderPage(pageIndex, format) renders a specific page of a multi-page document to an ImageBuffer.

🔧 Extend the Demo

  • Change the box color or thickness by modifying the Pen constructor (e.g. green boxes with 4 px stroke).
  • Add a semi-transparent fill behind each text region for better visibility.
  • Write detected regions to a JSON file for downstream processing.
  • Filter regions by position or size to focus on specific areas of the document.
  • Combine with LM-Kit's Structured Extraction to extract field values from specific regions.
  • Add page selection for PDFs (--pages 1,3-5) to process only specific pages.

📚 Additional Resources

📚 Related Content

Share