Deployment/Going Live

Behind a Reverse Proxy

How to put nginx, Caddy, or Traefik in front of the server without breaking it. Most of what a default proxy configuration does wrong here shows up as a product symptom: streamed answers arriving all at once, generations dying at sixty seconds, uploads refused with an error the client cannot parse. This page is for the platform engineer wiring the server into existing ingress; the network posture and HTTPS fundamentals it builds on are Going Live.

1Where the server sits#

The standard shape: the proxy terminates the public name, and the server listens on loopback behind it (default ports: HTTP 5189, HTTPS 7221, shown in the Network section). The Local only posture is exactly right for this, with one consequence you must handle:

A same-host proxy makes every caller look local. The server decides "is this caller on this machine" from the real connection peer, and a proxy on the same box IS a loopback peer. With anonymous access enabled, every request the proxy forwards, whoever sent it, would be served without a key. Before the proxy goes live, turn the anonymous toggle off (or switch the posture to Network, which refuses keyless calls outright) and mint keys for every caller: Keys and Authentication.

2Streaming must survive the proxy#

Chat responses stream as server-sent events on the OpenAI and Anthropic dialects and as newline-delimited JSON on the Ollama dialect (the surfaces are API Compatibility). The server writes and flushes every frame the moment the model produces it. A proxy with response buffering enabled collects those frames and releases them in large blocks, or all at once at the end: the client still gets a correct response, so nothing errors, but streaming is visibly dead and the evaluation reads it as a product bug.

The fix is proxy-side, per proxy:

nginx buffers upstream responses by default: set proxy_buffering off on the locations that carry API traffic. Do not rely on the X-Accel-Buffering response header to do this for you: the server does not send it on the dialect endpoints that third-party clients use.
Caddy recognizes text/event-stream and flushes immediately; set flush_interval -1 on the reverse_proxy anyway so the Ollama dialect's NDJSON streams get the same treatment.
Traefik flushes upstream responses on an interval (100 ms by default); set the service's responseForwarding.flushInterval to -1 to forward every chunk as it arrives.

Anything else in the chain that buffers (a CDN, a WAF, response compression applied by the proxy) breaks streaming the same way. Do not gzip SSE responses at the proxy.

3Timeouts and body limits#

Read timeouts. A proxy's upstream read timeout is the longest silence it tolerates, and default values (60 seconds in nginx) are far too short for inference. Two silent stretches matter: the time before the first token, while the model processes a long prompt, and a synchronous task endpoint running with the default Sync Timeout of 0, which blocks until the work finishes however long that takes. Size the timeout to the longest response you intend to serve (an hour is a reasonable ceiling), and point bulk clients at the job workflow instead: Prefer: respond-async returns 202 immediately and polling is cheap and proxy-friendly (Errors, Retries, and Jobs).

Body limits. Document endpoints accept uploads up to the Max Upload Size configured in the Admin Console (100 MB by default; changes apply per request, no restart). The proxy's own limit must be at least as large, or the proxy refuses the request before the server sees it, with the proxy's error shape instead of the server's structured 413. nginx defaults to 1 MB (client_max_body_size); Caddy and Traefik impose no request-body limit unless you configure one. Non-upload JSON endpoints keep a conservative ceiling of about 30 MB on the server side, which is deliberate: a payload that large usually means base64-embedded files, and the remedy the server's 413 names is to upload the file first and pass its id.

4TLS at the proxy or at the server#

Both are legitimate; what changes is the internal leg.

Terminate at the proxy (the usual choice when ingress already holds the certificates): proxy to the server's HTTP port. Two headers keep everything coherent: pass the original Host through, and set X-Forwarded-Proto to the client's scheme. The admin surface refuses plaintext connections from the network when Security:RequireHttpsForAdmin is on (it is by default, see Security); it honors X-Forwarded-Proto only from a proxy listed in Security:TrustedProxies, never from an arbitrary caller (a forgeable header would let anyone talk the enforcement out of existence). List the ingress address or network there, or the admin surface stays locked to HTTPS and loopback.
Re-encrypt to the server: proxy to the HTTPS port instead. The server's out-of-the-box certificate is self-signed, so the proxy must either trust it or skip verification for that upstream; loopback traffic never crosses a wire, so skipping verification on this leg costs nothing real.
The built-in ACME issuance (Going Live) expects the public name to resolve to the server and its HTTPS port to be reachable for validation. Behind ingress that is usually not true, so let the proxy own the public certificate and leave ACME off.

Admin SSO behind a proxy has one wrinkle: the OIDC callback URL is built from the scheme and host of the server's own leg of the connection. Pass the public Host header through so the callback carries the public name, and either re-encrypt to the HTTPS port (the callback is then https://...) or register the http-scheme callback at the provider. The SSO setup itself is Single Sign-On.

5Forwarded headers: what the server actually reads#

The server's use of forwarded headers is deliberately narrow, and you should know exactly where the line sits:

X-Forwarded-Proto is honored only from Security:TrustedProxies. When the connection peer is a listed proxy, the header restores the client leg's scheme before anything reads it, and the admin HTTPS enforcement sees the real transport. From any other caller the header is ignored: honoring it raw would let an on-path attacker relay a stripped-to-HTTP admin session while claiming HTTPS, which is the exact interception the enforcement exists to stop.
X-Forwarded-For follows the same rule. From a listed proxy it restores the real client address, so sign-in throttling, request records, and the audit trail attribute traffic to the client rather than the ingress. With no proxies listed, the anonymous-access check and the admin IP allowlist use the real connection peer, never a header a client could forge; behind an unlisted proxy that peer is the proxy, request records attribute traffic to its address, and an admin IP allowlist cannot distinguish the clients it fronts. Either list the ingress in Security:TrustedProxies or keep client-level network rules at the proxy, where the real peer is visible.

6Health endpoints for load balancers#

Two probes, different jobs; the full contract is in Observability:

GET /health: liveness. Returns 200 whenever the process is alive, requires no authentication, and is the endpoint to wire into load-balancer checks and container livenessProbe configuration (Running in Containers).
GET /lmkit/v1/ready: readiness. Returns 200 when the inference backend is initialized and 503 until then. Because it exposes internal state it requires a bearer token, so a readiness check must send an Authorization header.

For plain "is it up" routing, /health is enough. Gate traffic on /lmkit/v1/ready when the fleet should not receive requests during startup, before models are resident; it also answers 503 draining the moment a node begins a graceful stop, which is what makes scale-down lossless behind a balancer (Scaling Out).

7nginx, Caddy, Traefik, annotated#

One block per proxy, holding only the directives this guide argued for. nginx first, since its defaults break the most:

server {
    listen 443 ssl;
    server_name ai.example.com;
    # ssl_certificate / ssl_certificate_key here

    client_max_body_size 100m;            # at least the server's Max Upload Size

    location ~ ^/(admin|lmkit/v1/admin) { return 404; }   # admin stays private

    location / {
        proxy_pass http://127.0.0.1:5189;
        proxy_http_version 1.1;
        proxy_set_header Host $host;                  # public name reaches the server
        proxy_set_header X-Forwarded-Proto $scheme;   # honored once this proxy is in Security:TrustedProxies
        proxy_buffering off;                          # tokens leave as they arrive
        proxy_read_timeout 1h;                        # longest generation or sync task
    }
}

Caddy provisions the public certificate itself and sets the forwarded headers automatically, so the block is short:

ai.example.com {
    @admin path /admin* /lmkit/v1/admin/*
    respond @admin 404                    # admin stays private

    reverse_proxy 127.0.0.1:5189 {
        flush_interval -1                 # stream every chunk immediately
    }
}

Traefik, file-provider form; Traefik also sets the forwarded headers itself:

http:
  routers:
    lmkit:
      rule: "Host(`ai.example.com`)"
      entryPoints: [websecure]
      tls: {}
      service: lmkit
  services:
    lmkit:
      loadBalancer:
        servers: [{ url: "http://127.0.0.1:5189" }]
        responseForwarding:
          flushInterval: "-1"             # stream every chunk immediately

Two Traefik notes rather than directives, because both are version-dependent: check your version's entrypoint respondingTimeouts defaults against your longest generation, and if you enable the buffering middleware for body-size limits, set maxRequestBodyBytes at or above the server's upload limit. For the admin paths, route them on a higher-priority router restricted by an ipAllowList middleware, or do not route them on the public entrypoint at all.

8Keeping the admin surface private#

The admin console lives at /admin and its API under /lmkit/v1/admin/; the blocks above return 404 for both on the public host. This is defense in depth, not the only lock: the console still authenticates (password, two-factor, SSO), and each human-facing web destination (API reference, guides, playground, training, MCP guide) has its own served-or-not toggle and access level in the Interfaces section. The proxy rule makes the operator plane unreachable from outside, which is the posture most ingress policies expect. Reach the console from inside the network, or through a separate internal virtual host that only routes there.

9Stated plainly#

Turn proxy buffering off for API traffic, or streaming reads as broken: this is the single most common integration failure.
Raise the upstream read timeout to the longest response you serve, and push bulk work to Prefer: respond-async.
The proxy's body limit must be at least the server's Max Upload Size, or clients get the proxy's error instead of the server's.
The server honors X-Forwarded-Proto and X-Forwarded-For only from proxies listed in Security:TrustedProxies; a TLS-terminating proxy that is not listed locks the admin surface to HTTPS and loopback, and client attribution then stays at the proxy.
A same-host proxy makes every caller loopback: disable anonymous access before it fronts anything.
GET /health for liveness (anonymous), GET /lmkit/v1/ready for readiness (bearer token, 503 until the backend can serve).