Load Testing and Go-Live Acceptance

Inference Capacity declares the shape (slots times window) and Hardware Sizing prices it against the device. Acceptance is the third step: prove, with your prompts at your concurrency, that the shape serves the latency you promised, that saturation behaves the way the configured policy says, that the probes and the balancer route correctly, and that a node leaves without dropping a request. This page is that run, written so that it can be repeated after every change of model, shape or hardware. Every figure it asks for comes from the server's own records, not from a stopwatch on the client side.

1Preflight: nothing to measure until this passes#

Run these on the node itself before generating any load; a failure here invalidates every number below.

curl -s -H "Authorization: Bearer $LMKIT_API_KEY" "$LMKIT_URL/lmkit/v1/ready"

{ "status": "ready", "loaded_model_names": ["qwen3.5:9b"], "active_inferences": 0, "queue_depth": 0, "slots_saturation": 0.0, "node": "a1b2c3", "mode": "single_node" }

Record the cold-start number once, on its own, because it is a number of its own: the seconds between process start and the first 200 from /lmkit/v1/health/ready. It is what a rolling upgrade or a scale-out pays per node (Scaling Out section 8).

start=$(date +%s); until curl -sf -o /dev/null "$LMKIT_URL/lmkit/v1/health/ready"; do sleep 1; done; echo "ready after $(( $(date +%s) - start )) s"

2The traffic model: three numbers and a shape#

Acceptance tests the traffic you will actually serve, so write it down before the run: N, the concurrent callers at peak (not requests per second: serving is slot-based, so concurrency is the unit that matters); P, the prompt length in tokens, from your real prompts; C, the completion length you allow (max_tokens). Then the shape:

Two disciplines from Measuring What Matters apply verbatim: measure at steady state (the first request pays model load and pool construction; the hold phase is the measurement) and keep the workload shape fixed between runs, or the numbers compare workloads rather than configurations.

3The k6 run#

k6 is enough: one script, three environment variables, a ramp to N, a ten-minute hold, a ramp down. The script streams, asks for the usage chunk, and derives the two numbers a stopwatch cannot give: time to first token and decode speed.

// acceptance.js: k6 run -e LMKIT_URL=https://ai.example.com -e LMKIT_API_KEY=lmk_... -e MODEL=qwen3.5:9b -e VUS=16 acceptance.js
import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter } from 'k6/metrics';

const ttft = new Trend('lmkit_ttft_ms', true);
const tokensPerSecond = new Trend('lmkit_tokens_per_second');
const completionTokens = new Counter('lmkit_completion_tokens');

export const options = {
  scenarios: {
    steady: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: Number(__ENV.VUS) },   // ramp to N
        { duration: '10m', target: Number(__ENV.VUS) },  // the measurement
        { duration: '1m', target: 0 },
      ],
      gracefulRampDown: '2m',
    },
  },
  thresholds: {
    http_req_failed: ['rate==0'],            // nothing refused at N: the shape fits
    lmkit_ttft_ms: ['p(95)<1500'],           // your promise, in milliseconds
    http_req_duration: ['p(95)<20000'],      // the whole completion
    lmkit_tokens_per_second: ['p(50)>15'],   // per request, at N, streaming
  },
};

const headers = { 'Content-Type': 'application/json', Authorization: `Bearer ${__ENV.LMKIT_API_KEY}` };

export default function () {
  const body = JSON.stringify({
    model: __ENV.MODEL,
    stream: true,
    stream_options: { include_usage: true },
    max_tokens: 300,
    messages: [
      { role: 'system', content: 'Answer in two short paragraphs.' },
      { role: 'user', content: 'Explain why a train timetable is a graph, for a new dispatcher.' },
    ],
  });
  // X-Request-Id is honored and echoed (64 characters at most), so every row in the
  // Requests page can be traced back to a virtual user and an iteration of this run.
  const res = http.post(`${__ENV.LMKIT_URL}/v1/chat/completions`, body,
    { headers: { ...headers, 'X-Request-Id': `k6-${__VU}-${__ITER}` }, timeout: '180s' });
  check(res, { 'status 200': (r) => r.status === 200 });
  if (res.status !== 200) return;

  // A streamed completion sends nothing before its first token, so the first byte IS the
  // first token: k6 reports that as `waiting`.
  ttft.add(res.timings.waiting);

  // The last chunk carries usage (every earlier chunk says "usage":null); tokens divided
  // by the time spent after the first token is the decode speed this request saw.
  const usageLine = res.body.split('\n').filter((l) => l.includes('"usage":{')).pop();
  if (usageLine) {
    const usage = JSON.parse(usageLine.slice(6)).usage;  // strip "data: "
    completionTokens.add(usage.completion_tokens);
    const decodeMs = res.timings.duration - res.timings.waiting;
    if (decodeMs > 0) tokensPerSecond.add(usage.completion_tokens / (decodeMs / 1000));
  }
}

Replace the two messages with your own prompts: load a JSON array of real requests with open() into a SharedArray and pick one per iteration, so P and C are yours. For the document profile, the prompt is a real document's text at the length your users send; for a model with a per-model override, name that model in MODEL so the override is what gets tested. max_tokens is part of the workload: fix it.

4Where the figures are, and which ones count#

Read the run from the server's records; the k6 summary is the client's view of the same events and includes your network.

The Metrics page and the Prometheus or OTLP export carry the continuous view; enabling the export before the run is the way to keep the run's curves (Observability section 4).

5Overload, on purpose#

Run the same script at 2N once. The point is not the latency, it is the behavior, which must match the saturation policy you configured:

• Queue (the default): no request is refused, latency stretches, lmkit_inference_queue_depth rises and falls back to 0 when the ramp ends, saturation sits at 1.0 for the hold. The one failure this run exposes is a proxy whose read timeout is shorter than a queued completion; a burst of 499 or proxy-side 504 during the hold is that, and Behind a Reverse Proxy section 3 has the setting.
• Reject: every request that finds no free slot answers 503 at once with Retry-After: 1, nothing queues, and the clients own the retry. Tell k6 that 503 is an expected outcome for this run and count it:

http.setResponseCallback(http.expectedStatuses(200, 503));
const shed = new Counter('lmkit_shed_503');
// after the request:
if (res.status === 503) { shed.add(1); check(res, { 'Retry-After set': (r) => r.headers['Retry-After'] !== undefined }); return; }

Two more refusals belong to this run when the workload includes them: a task pipeline that submits faster than it drains meets 429 too_many_inflight_jobs with Retry-After: 5 at the per-key ceiling, and a prompt larger than the window is answered 413 input_too_large on the task routes, while a chat completion that runs out of window stops with finish_reason length. All three are the contract, not defects; the acceptance question is whether your clients handle them the way Errors, Retries, and Jobs section 6 says.

6On a farm: four more runs#

A fleet (Scaling Out) adds behaviors a single node cannot show. Enable the Prometheus export on every node first; the fleet reads itself through it.

1. Distribution. During the hold, sum by (instance) (rate(lmkit_http_server_requests_total[1m])) is roughly even across nodes, and each node's Requests page shows only its own rows, stamped with its name. A 421 anywhere in the trail means a node-bound follow-up (a document chat turn, a skip-thinking call) reached the wrong node: the balancer's affinity rules from Behind a Reverse Proxy section 7 are missing or wrong.
2. Drain. Mid-hold, stop one node the way your platform will (kubectl delete pod, or systemctl stop on a VM). The run's http_req_failed stays at zero: the node advertised draining for DrainAdvertiseSeconds while the balancer noticed, then finished its streams inside ShutdownDrainSeconds. A failure here is a stop allowance below advertise plus drain plus 15 seconds, or a VM fleet that never set DrainAdvertiseSeconds=5 (Fleet Recipes has the setting per platform).
3. Warm join. Start a node (or let the platform replace the one you stopped) and watch lmkit_ready_state for it: 0 with status="warming" until its lineup is resident, then 1, and only then does traffic reach it. The seconds it took is the cold-start figure from section 1, which your autoscaler's cooldown must respect.
4. The scale signal. At 2N with the Queue policy, avg(lmkit_inference_slots_saturation) crosses the threshold you configured; on Kubernetes the KEDA ScaledObject adds a pod within its polling interval and the average falls back below the threshold once that pod is ready (Running in Containers section 9). On a VM fleet the same gauge is what your own automation reads.

7The sign-off sheet#

One row per claim, each with the evidence that closes it. A go-live with an open row is a decision someone signs, not an oversight.

Keep the script, the environment values and the k6 summary with the release: the next model upgrade or shape change reruns the same file and compares against them (Backup and Upgrades section 2 places the rerun in the upgrade sequence).

8Stated plainly#

• Acceptance is preflight, a ten-minute hold at your real concurrency with your real prompts, one deliberate overload, and, on a farm, a drain and a warm join; every figure is read from the server's own records.
• The numbers that decide: time to first token and decode speed at N, zero refusals and zero queue at N, saturation below the scale threshold, device memory flat across the hold.
• Overload must behave as configured: queue with no refusals, or 503 with Retry-After at once; 429 at the per-key ceiling and 413 for oversized inputs are part of the same contract.
• Keep the script and its results beside the release, and rerun it before every model, shape or hardware change.