Measuring What Matters

Half the guides in this library end the same way: measure on your own traffic. This page is the method. Published benchmarks rank models on someone else's distribution; your invoices, your taxonomies, and your hardware are the only benchmark that predicts your production, and building one costs an afternoon, not a project. The payoff compounds: the same labeled sample answers "which model", "did the prompt change help", "is the small model enough", and eventually "did fine-tuning pay off", for as long as you keep it.

1The labeled sample: your benchmark, built once#

• Size: thirty to a hundred REAL items per task is enough to rank options and catch regressions; statistical perfection is not the goal, decision-grade evidence is.
• Stratify by difficulty, not convenience. Include the crumpled scans, the two-language invoices, the borderline categories: the sample exists to represent what breaks, and a sample of easy cases measures nothing.
• Write ground truth once, as data. The correct category per document, the correct field values as JSON, the correct answer per query. Version the sample with your schemas and taxonomies; when the taxonomy changes, the sample changes with it, deliberately.
• Never tune on it blindly. When a change is inspired by failures in the sample, confirm on a handful of fresh items before believing the improvement.

2Accuracy protocols, per task#

• Classification: run the sample, build the confusion table (predicted versus truth), and read WHICH pairs confuse, not just the rate: confusion between two categories is usually a description problem, fixable in the request. Test allow_unknown_category honesty with a few out-of-taxonomy items.
• Extraction: compare field-by-field against the golden JSON (exact for enums, dates, and numbers; normalized for free text). Then CALIBRATE the review loop: with include_elements, bucket each field's reported confidence against whether it was actually correct. That table tells you what the review flag is worth on YOUR corpus and whether your threshold routes the right exceptions to humans.
• OCR and searchable PDFs: extraction accuracy downstream is usually the measure that matters; for the text layer itself, spot-check recognized text against a few known passages and let per-page results point at the weak pages.
• Search: retrieval has a full harness built in: golden query sets, persisted evaluation runs, and metrics, covered by Measuring Search Quality. Use it rather than rebuilding it.

3Comparing honestly#

One comparison rule covers every "A or B" question, model against model, prompt against prompt, before against after: same sample, one variable changed, and fixed randomness. Pin the model per request, pass a fixed seed, and hold reasoning settings constant so the difference you observe is the difference you changed. Where quality needs human judgment (summaries, rewrites), judge blind: present outputs unlabeled, decide, then unblind. It is the cheapest defense against believing the model you already prefer.

4Throughput and latency, read right#

• Measure steady state. The first request pays model load and pool construction; discard it (or measure cold start separately as its own number, because it is one).
• Measure at your real concurrency. Serving is slot-based, so single-request latency says little about the throughput N parallel callers get: drive the test at production concurrency against the slot pool as configured.
• Separate input from output cost. Long prompts and long completions scale differently; when comparing runs, keep the workload shape fixed or the numbers compare workloads, not configurations.
• Read from the server's own records. The Requests page records per-request timing, model, and outcome; the OpenAI-compatible surface returns the standard usage token counts; the Telemetry panel (and Prometheus or OTLP export when enabled) carries the continuous view: queue depth, active inferences, and the runtime instruments. Client-side stopwatches measure your network too; the server's records measure the server.

5The improvement loop#

Measurement is the loop's engine, in one direction: measure, change the cheapest lever, re-measure. The lever order that pays: descriptions and schema constraints first (free), then model choice per capability (a default-slot change), then fine-tuning when a task has volume and your corrected outputs have accumulated into training data, at which point the same sample becomes the before/after proof that the tune earned its keep.

6Stated plainly#

• Build one labeled, stratified, versioned sample per task; it is the only benchmark that predicts your production.
• Per task: confusion tables for classification, field-level comparison plus confidence calibration for extraction, the built-in harness for search.
• Compare with one variable, fixed seeds, and blind judgment; read performance from the server's records at real concurrency, steady state.