Shipping AI Features with CI

An AI feature is code plus a model plus prompts plus schemas, and any of the four can change behavior. This page puts the feature under the same discipline as the rest of your software: a server in the pipeline, contract tests on every commit, a quality gate on your own labeled sample when the model or the prompts move, and an upgrade procedure with a rollback. Measuring What Matters supplies the sample and the protocols; this page wires them into a pipeline.

1Two tiers, two servers#

The split is what keeps the pipeline honest: a small model on a CPU runner proves the plumbing in minutes, and only the staging server can say anything about quality, because quality depends on the model and the shape you actually serve.

2The CI server#

The image starts in seconds, warmup gates readiness, and a health check makes the runner wait for it. Bake the CI model into a derived image so the job never downloads it: the two-line Dockerfile in Running in Containers section 2, with RUN lmkit pull qwen3.5:2b. Then, as a GitHub Actions service container (any CI with service containers works the same way):

jobs:
  ai-contract-tests:
    runs-on: ubuntu-latest
    services:
      lmkit:
        image: ghcr.io/acme/lmkit-one-ci:2026.9.1     # FROM lmkitone/lm-kit-one:<version>, plus the pulled model
        env:
          DefaultChatModel: "qwen3.5:2b"
          Inference__WarmupModels__0: default          # readiness answers 200 only once the model is resident
          Inference__SlotCount: "2"
          Inference__SlotContextSize: "8192"
          Admin__InitialPassword: ${{ secrets.LMKIT_CI_ADMIN_PASSWORD }}
          Security__RequireHttpsForAdmin: "false"      # the runner reaches the container over plain HTTP
          MaxJobsPerOwner: "2"                         # small on purpose: the 429 path is a contract test
        ports: ["5189:5189"]
        options: >-
          --health-cmd "curl -sf http://localhost:5189/lmkit/v1/health/ready"
          --health-interval 10s --health-timeout 5s --health-retries 30 --health-start-period 60s
    steps:
      - uses: actions/checkout@v4
      - name: Mint a key for this run
        env:
          ADMIN_PASSWORD: ${{ secrets.LMKIT_CI_ADMIN_PASSWORD }}
        run: |
          SESSION=$(curl -sf -X POST http://localhost:5189/lmkit/v1/admin/login -H 'Content-Type: application/json' \
            -d "{\"password\":\"$ADMIN_PASSWORD\"}" | jq -r .token)
          KEY=$(curl -sf -X POST http://localhost:5189/lmkit/v1/admin/apikeys -H "X-Admin-Password: $SESSION" \
            -H 'Content-Type: application/json' -d "{\"name\":\"ci-$GITHUB_RUN_ID\",\"expiresInDays\":1}" | jq -r .token)
          echo "::add-mask::$KEY"; echo "LMKIT_API_KEY=$KEY" >> "$GITHUB_ENV"
      - run: LMKIT_URL=http://localhost:5189 ./tests/ai-contract.sh

Inside a container the posture is Network, so every call carries a key: the job mints one for itself with the admin API (the console's own session header) and lets it expire. The server's version is in GET /health and in lmkit_build_info; pin the image tag, never latest, so a server upgrade is a commit in your repository like any other dependency.

3Contract tests: assert on structure#

Everything the feature relies on is visible in the API's own responses, so a contract test is plain HTTP with assertions on fields. The four families, with what to pin per request:

• Schema-bound answers. Use response_format with json_schema: validity is guaranteed by grammar-constrained decoding, so the test asserts VALUES (the enum chosen, the field present, the number in range) and never wastes a case on "is it valid JSON" (Structured Outputs).
• Tool routing. For an agent or a tool-enabled chat, assert on tool_calls and on the tool_events summary: "the refund question consulted the policy document and never the web" is a check on structured output (Building and Testing Agents section 3; Function Calling for your own tools).
• Streaming. Parse a streamed completion end to end: the chunks, the usage chunk under stream_options.include_usage, the [DONE] sentinel, and the error EVENT a stream carries when it fails after tokens began (revoking the run's key while a long completion streams produces one on demand, since a revoked key fails closed mid-flight).
• The error paths you promised to handle. 429 too_many_inflight_jobs (submit more jobs than the CI server's MaxJobsPerOwner of 2), 413 input_too_large (a prompt above the window on a task route), 503 with Retry-After under a Reject saturation policy, and a 4xx that must not be retried (Errors, Retries, and Jobs sections 4 to 6).

Per request, always: model pinned by id (a request that names no model follows the server's default slot, which is a configuration you do not control from a test), a fixed seed, and X-Request-Id set to the test case's name (64 characters at most): the server echoes it and the Requests page filters on it, so a failing case is one click from the server's own record of the exchange.

# One contract case: classification must return one of the schema's values, never prose.
curl -sf "$LMKIT_URL/v1/chat/completions" -H "Authorization: Bearer $LMKIT_API_KEY" -H 'Content-Type: application/json' \
  -H 'X-Request-Id: contract-triage-0007' -d @- <<'JSON' | jq -e '.choices[0].message.content | fromjson | .category == "billing"'
{ "model": "qwen3.5:2b", "seed": 7, "temperature": 0,
  "messages": [{ "role": "system", "content": "Classify the support message." },
               { "role": "user", "content": "I was charged twice for September." }],
  "response_format": { "type": "json_schema", "json_schema": { "name": "triage", "schema": {
    "type": "object", "properties": { "category": { "type": "string", "enum": ["billing", "outage", "how-to", "other"] } },
    "required": ["category"] } } } }
JSON

4The quality gate#

The gate runs your golden set (Measuring What Matters section 1: thirty to a hundred real, stratified, versioned items per task) against the staging server and fails the pipeline on thresholds:

Two protocol rules, both from the measurement guide: one variable per comparison (the same sample, the same seeds, the same shape, one change), and judge prose blind. And one caveat that belongs in the harness: the shared slot pool decodes several requests together, so a seed makes a run reproducible in distribution, not byte for byte across different co-batching. Grade outcomes (the category, the field value, the score), never string equality on free text.

Where the server has a harness of its own, use it: retrieval quality has golden query sets, persisted runs and metrics built in (Measuring Search Quality).

5Versioning what changes behavior#

6Upgrading a model without a regression#

1. Pull the candidate beside the incumbent (lmkit pull <id>, or the console); both are resident, and the panel prices a per-model override if the candidate wants its own shape.
2. Run the gate against both, same golden set, same seeds, model pinned to each in turn: one variable. The candidate must clear every threshold AND the margin against the incumbent.
3. Canary from your application, not from the server: route a share of traffic by naming the candidate in model. Changing the server's default chat slot moves every client at once, so keep per-request pinning until the soak is over.
4. Promote: switch your pinned id, or the default slot if your clients rely on it (a Domain object, applied on every node within the sync period on a farm).
5. Roll back by pinning the previous id; keep the previous model's file until the soak passes, since a delete is the one step that is not instant to undo.

Run the acceptance script after the promotion when the candidate is larger or has a different shape: quality is the gate's question, capacity is acceptance's.

7Stated plainly#

• Two tiers: contract tests on every commit against the image on the runner with a small model; a quality gate on the staging server with the production model when model, prompt, schema or shape change.
• Contract tests assert on structure the API guarantees: schema values, tool_calls and tool_events, the stream's shape, the error codes you handle.
• The gate is your golden set with thresholds on accuracy, calibration, tokens and latency, one variable per comparison, graded outcomes rather than string equality.
• Pin model, seed and X-Request-Id on every test request; version prompts, schemas and models beside the sample they were graded on.
• Upgrade by running the gate on both models, canarying from your application, and rolling back by pinning the previous id.