Deployment/Going Live

Behind a Reverse Proxy

How to put nginx, Caddy, or Traefik in front of the server without breaking it. Most of what a default proxy configuration does wrong here shows up as a product symptom: streamed answers arriving all at once, generations dying at sixty seconds, uploads refused with an error the client cannot parse. This page is for the platform engineer wiring the server into existing ingress; the network posture and HTTPS fundamentals it builds on are Going Live.

1Where the server sits#

The standard shape: the proxy terminates the public name, and the server listens on loopback behind it (default ports: HTTP 5189, HTTPS 7221, shown in the Network section). The Local only posture is exactly right for this, with one consequence you must handle:

A same-host proxy makes every caller look local. The server decides "is this caller on this machine" from the real connection peer, and a proxy on the same box IS a loopback peer. With anonymous access enabled, every request the proxy forwards, whoever sent it, would be served without a key. Before the proxy goes live, turn the anonymous toggle off (or switch the posture to Network, which refuses keyless calls outright) and mint keys for every caller: Keys and Authentication.

2Streaming must survive the proxy#

Chat responses stream as server-sent events on the OpenAI and Anthropic dialects and as newline-delimited JSON on the Ollama dialect (the surfaces are API Compatibility). The server writes and flushes every frame the moment the model produces it. A proxy with response buffering enabled collects those frames and releases them in large blocks, or all at once at the end: the client still gets a correct response, so nothing errors, but streaming is visibly dead and the evaluation reads it as a product bug.

The fix is proxy-side, per proxy:

nginx buffers upstream responses by default: set proxy_buffering off on the locations that carry API traffic. Do not rely on the X-Accel-Buffering response header to do this for you: the server does not send it on the dialect endpoints that third-party clients use.
Caddy recognizes text/event-stream and flushes immediately; set flush_interval -1 on the reverse_proxy anyway so the Ollama dialect's NDJSON streams get the same treatment.
Traefik flushes upstream responses on an interval (100 ms by default); set the service's responseForwarding.flushInterval to -1 to forward every chunk as it arrives.

Anything else in the chain that buffers (a CDN, a WAF, response compression applied by the proxy) breaks streaming the same way. Do not gzip SSE responses at the proxy.

3Timeouts and body limits#

Read timeouts. A proxy's upstream read timeout is the longest silence it tolerates, and default values (60 seconds in nginx) are far too short for inference. Two silent stretches matter: the time before the first token, while the model processes a long prompt, and a synchronous task endpoint running with the default Sync Timeout of 0, which blocks until the work finishes however long that takes. Size the timeout to the longest response you intend to serve (an hour is a reasonable ceiling), and point bulk clients at the job workflow instead: Prefer: respond-async returns 202 immediately and polling is cheap and proxy-friendly (Errors, Retries, and Jobs).

Body limits. Document endpoints accept uploads up to the Max Upload Size configured in the Admin Console (100 MB by default; changes apply per request, no restart). The proxy's own limit must be at least as large, or the proxy refuses the request before the server sees it, with the proxy's error shape instead of the server's structured 413. nginx defaults to 1 MB (client_max_body_size); Caddy and Traefik impose no request-body limit unless you configure one. Non-upload JSON endpoints keep a conservative ceiling of about 30 MB on the server side, which is deliberate: a payload that large usually means base64-embedded files, and the remedy the server's 413 names is to upload the file first and pass its id.

4TLS at the proxy or at the server#

Both are legitimate; what changes is the internal leg.

Terminate at the proxy (the usual choice when ingress already holds the certificates): proxy to the server's HTTP port. Two headers keep everything coherent: pass the original Host through, and set X-Forwarded-Proto to the client's scheme. The admin surface refuses plaintext connections from the network when Security:RequireHttpsForAdmin is on (it is by default, see Security); it honors X-Forwarded-Proto only from a proxy listed in Security:TrustedProxies, never from an arbitrary caller (a forgeable header would let anyone talk the enforcement out of existence). List the ingress address or network there, or the admin surface stays locked to HTTPS and loopback.
Re-encrypt to the server: proxy to the HTTPS port instead. The server's out-of-the-box certificate is self-signed, so the proxy must either trust it or skip verification for that upstream; loopback traffic never crosses a wire, so skipping verification on this leg costs nothing real.
The built-in ACME issuance (Going Live) expects the public name to resolve to the server and its HTTPS port to be reachable for validation. Behind ingress that is usually not true, so let the proxy own the public certificate and leave ACME off.

Admin SSO behind a proxy has one wrinkle: the OIDC callback URL is built from the scheme and host of the server's own leg of the connection. Pass the public Host header through so the callback carries the public name, and either re-encrypt to the HTTPS port (the callback is then https://...) or register the http-scheme callback at the provider. The SSO setup itself is Single Sign-On.

5Forwarded headers: what the server actually reads#

The server's use of forwarded headers is deliberately narrow, and you should know exactly where the line sits:

X-Forwarded-Proto is honored only from Security:TrustedProxies. When the connection peer is a listed proxy, the header restores the client leg's scheme before anything reads it, and the admin HTTPS enforcement sees the real transport. From any other caller the header is ignored: honoring it raw would let an on-path attacker relay a stripped-to-HTTP admin session while claiming HTTPS, which is the exact interception the enforcement exists to stop.
X-Forwarded-For follows the same rule. From a listed proxy it restores the real client address, so sign-in throttling, request records, and the audit trail attribute traffic to the client rather than the ingress. With no proxies listed, the anonymous-access check and the admin IP allowlist use the real connection peer, never a header a client could forge; behind an unlisted proxy that peer is the proxy, request records attribute traffic to its address, and an admin IP allowlist cannot distinguish the clients it fronts. Either list the ingress in Security:TrustedProxies or keep client-level network rules at the proxy, where the real peer is visible.
A front whose addresses cannot be listed (Azure App Service, Azure Container Apps, a managed balancer with a changing address pool) takes Security:TrustedProxies = *, which trusts every forwarder. The server accepts the sentinel on the two Azure platforms it detects, where an instance is reachable only through the platform front, and refuses to start with it anywhere else unless Security:ProxyFrontOnly=true states that the nodes accept connections from the front alone (private subnets, a security group, a host firewall): with the sentinel on a node a client can reach directly, X-Forwarded-For would let any client impersonate any address to the sign-in throttle and the admin IP allowlist.
Security:PublicBaseUrl names the public address when TLS ends at the front (for example https://ai.example.com): the SSO callback registered with the identity provider and the Secure flag on the server's cookies then follow the front rather than the node's own listener. The server does not read X-Forwarded-Host.

6Health endpoints for load balancers#

Three probes, different jobs; the full contract is in Observability:

GET /health: liveness. Returns 200 whenever the process is alive, warming or draining included, requires no authentication, and is the endpoint for container livenessProbe configuration and restart decisions (Running in Containers). It never decides where traffic goes.
GET /lmkit/v1/health/ready: readiness, anonymous and status-only. Returns 200 when the node can serve and 503 with the reason (not_ready, warming, draining) otherwise, on GET and HEAD. This is the balancer health check: it needs no header, so the IIS ARR health test, Azure Load Balancer, an AWS ALB target group and the App Service health check gate on it, as do HAProxy (option httpchk GET /lmkit/v1/health/ready) and nginx Plus.
GET /lmkit/v1/ready: the same decision with the detailed body (counts, model names, saturation, node), behind a bearer token because it exposes internal state; for probers and scalers that want the detail and can send an Authorization header.

Gate traffic on readiness, never on liveness: a node answers 503 warming until its configured lineup is resident and 503 draining the moment a graceful stop begins, which is what makes warm joins and lossless scale-down work behind a balancer (Scaling Out section 8). nginx open source has no active health check (it retires an upstream after max_fails failed requests) and Windows NLB probes nothing above the network layer; behind either, a node takes traffic before its models are resident and keeps taking it while it drains, so put an HTTP-aware balancer in front where warm joins matter.

7Affinity for the node-bound surfaces#

A few surfaces live in the memory of the node that created them (chat-with-document sessions including the Playground document chat, grounded chat at /lmkit/v1/search/chat, MCP sessions at /mcp, every async job id, skip-thinking and cancel, the PII detect-then-redact pair), and the admin console and Playground read one node's operator plane, so a fleet pins at the proxy (Scaling Out section 6 is the tier table). The server emits a contract the proxy and the client can work with, and this section says, per balancer, which part of it the balancer can actually use:

X-LMKit-Node on every response carries the answering node's six-character tag (the full node name only on admin-authenticated responses); GET /health on a node's own address reads its tag.
lmk-node=<tag>, an HttpOnly, SameSite=Lax cookie valid one day (Secure over HTTPS), set by a fleet member (managed configuration or a shared identity store) on the responses that mint a node-bound id and on the /admin and /playground page loads. Its value is the same tag.
421 Misdirected Request with { "error": "not_on_this_node", "node": "<tag>", "hint": ... } when a request names a session or job that belongs to another node; the job 202 also carries Location: /lmkit/v1/jobs/<job_id>.

A 421 is therefore the symptom of a lost pin, and X-LMKit-Node is the verification tool: when the tag on the minting response, the tag inside the id and the value of lmk-node agree and a later request still answers 421, the balancer is not honouring the pin. Three of the balancers below read the server's cookie as a routing key (HAProxy with per-server cookie values, nginx Plus sticky route, AWS ALB application-cookie stickiness); the others pin browsers with a cookie THEY mint, and lmk-node then tells the operator which node a browser is on. Bearer API clients keep no cookies on any of them, so they hash on Authorization.

The rule in one sentence: the request that mints a job or starts a stream and the request that follows it must hash identically, which only a whole-surface hash guarantees. A job id is minted by whichever endpoint was asked for async work (Prefer: respond-async on any document, audio or classification endpoint), and the request-id form of skip-thinking targets the exchange streaming on /lmkit/v1/chat; neither can be pinned by a list of paths, so the API rule is the complement of the browser set, never a path list. Two sets recur below:

The browser set, /admin, /lmkit/v1/admin and /playground, driven by a browser that keeps cookies. The console signs its calls with a session header (X-Admin-Password) and the web areas carry the area-ticket cookie; neither is a bearer, so these paths pin through a cookie (the server's lmk-node where the balancer can read it, the balancer's own otherwise), or the source address. The Playground's own chat and skip-thinking ride /lmkit/v1/admin/playground/chat/*, inside this set.
The API set, everything else, hashed by the Authorization header: one bearer client reaches one node for as long as the pool is stable, so the node that minted its job id or is streaming its exchange is the node its poll or skip-thinking reaches. This is also the KV prefix-cache optimization: a client that resends long system prompts keeps its prefixes warm on one node.

Every header-hash policy falls back to round-robin (or a random pick) on a request without the header: nginx, HAProxy, Caddy and ingress-nginx all do. That is why bearer-less browser calls sit in the browser set, and why an API client that omits its key loses its pin along with its authorization. The section 8 blocks return 404 for the admin paths on the public host, so the browser-set rules belong on the internal virtual host that serves the console.

nginx (open source) has hash and ip_hash; it cannot map a cookie VALUE to a server (the cookie-reading sticky directive is nginx Plus only), so browsers pin by source address here and lmk-node is a verification aid only. Hashing $cookie_lmk_node is not a pin: the tag hashes to an arbitrary node, not to the node that minted it.

upstream lmkit_by_key {
    hash $http_authorization consistent;   # bearer clients: same key, same node
    server 10.0.0.11:5189;
    server 10.0.0.12:5189;
}
upstream lmkit_by_client {
    hash $remote_addr consistent;          # browsers: same client address, same node
    server 10.0.0.11:5189;
    server 10.0.0.12:5189;
}

# Browser set, internal host only: pinned by client address.
location ~ ^/(admin|playground|lmkit/v1/admin)(/|$) { proxy_pass http://lmkit_by_client; }
# Everything else is the API set: pinned by key. Same proxy_* directives as the section 8 block.
location / { proxy_pass http://lmkit_by_key; }

nginx Plus adds sticky cookie (a cookie nginx mints) and sticky route, which reads a value the application sets: sticky route $cookie_lmk_node; with route=<tag> on each server line, the tag being what X-LMKit-Node shows on that node's /health, pins on the server's own cookie.

HAProxy can pin on the server's cookie or mint its own, hashes headers natively, streams responses without buffering by default, and gates membership on the anonymous readiness route. The block below lets HAProxy mint its own cookie (cookie SRV insert indirect nocache; the server never sees it):

backend lmkit_browser
    balance roundrobin
    cookie SRV insert indirect nocache      # HAProxy mints the cookie; the server never sees it
    option httpchk GET /lmkit/v1/health/ready
    timeout server 1h
    server node1 10.0.0.11:5189 check cookie n1
    server node2 10.0.0.12:5189 check cookie n2

backend lmkit_api
    balance hdr(Authorization)              # bearer clients: same key, same node
    hash-type consistent
    option httpchk GET /lmkit/v1/health/ready
    timeout server 1h
    server node1 10.0.0.11:5189 check
    server node2 10.0.0.12:5189 check

frontend lmkit_in
    bind :443 ssl crt /etc/haproxy/ai.example.com.pem
    acl browser  path_beg /admin /playground /lmkit/v1/admin
    use_backend lmkit_browser if browser
    default_backend lmkit_api                # everything else: the API set, hashed by key

To route on the server's cookie instead, so a browser that already carries lmk-node lands on the node it names, give each server the node's tag as its cookie value and let the server's Set-Cookie stand:

backend lmkit_browser
    balance roundrobin
    cookie lmk-node insert indirect nocache preserve   # preserve: the server's own lmk-node wins
    server node1 10.0.0.11:5189 check cookie k7m2xa    # the tag X-LMKit-Node shows on node1's /health
    server node2 10.0.0.12:5189 check cookie p3vq7w    # and on node2's

AWS ALB pins with target-group stickiness, and its application-based mode reads the server's cookie directly: stickiness.enabled=true, stickiness.type=app_cookie, stickiness.app_cookie.cookie_name=lmk-node, stickiness.app_cookie.duration_seconds=86400. ALB maps the value it first sees to the target that set it, so the browser set pins on lmk-node without further configuration. The duration-based mode (lb_cookie, the AWSALB cookie) works too and ignores lmk-node. ALB has no header hashing, so a bearer client that keeps no cookie is unpinned behind it; its target-group health check, which sends no header, probes /lmkit/v1/health/ready.

ingress-nginx pins per Ingress resource with a cookie IT mints (lmk-node is informational there), so the browser set gets its own Ingress object with cookie affinity and the main Ingress (path /) carries the header hash; the two annotations are exclusive on one Ingress, and the longer path prefix wins between them:

metadata:
  annotations:                                        # browser set: /admin, /playground, /lmkit/v1/admin
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "lmkit-route"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "86400"
---
metadata:
  annotations:                                        # the main Ingress, path /: the API set
    nginx.ingress.kubernetes.io/upstream-hash-by: "$http_authorization"

Traefik offers a sticky cookie on the service (a cookie Traefik mints; lmk-node is informational) and nothing else in the open-source build: there is no header hash, and a bearer client keeps no cookie, so Traefik OSS cannot pin a bearer client's job polls or skip-thinking at all. Behind it, route bearer clients to one node by hostname (one router per node), or let the client follow X-LMKit-Node itself when it can reach nodes by address. The cookie still serves the browser set:

http:
  services:
    lmkit:
      loadBalancer:
        sticky:
          cookie: { name: lmkit_route, httpOnly: true, secure: true }
        servers: [{ url: "http://10.0.0.11:5189" }, { url: "http://10.0.0.12:5189" }]
        responseForwarding: { flushInterval: "-1" }
  routers:
    lmkit:
      rule: "Host(`ai.example.com`)"
      service: lmkit

Caddy has both policies built in; its cookie policy mints its own cookie, and the default reverse_proxy takes the API set:

ai.example.com {
    @browser path /admin* /playground* /lmkit/v1/admin/*
    reverse_proxy @browser 10.0.0.11:5189 10.0.0.12:5189 {
        lb_policy cookie lmkit_route      # Caddy mints and reads the cookie itself
        flush_interval -1
    }
    reverse_proxy 10.0.0.11:5189 10.0.0.12:5189 {
        lb_policy header Authorization    # everything else: same key, same node
        flush_interval -1
    }
}

IIS ARR pins with Client affinity on the server farm (the ARRAffinity cookie ARR sets itself; lmk-node is informational), which serves the browser set and any API client that keeps a cookie jar; ARR has no header hash, and its health test URL (no header) is /lmkit/v1/health/ready, so farm membership follows readiness. Azure App Service pins the same way with its own ARRAffinity cookie when ARR affinity is on, and its platform health check takes the same path. Azure Application Gateway pins with cookie-based affinity (its ApplicationGatewayAffinity cookie) and probes a custom path per backend setting, and Azure Front Door pins with session affinity per origin group and probes a path with GET or HEAD; neither sends a header, and both gate on /lmkit/v1/health/ready. Windows NLB and Azure Load Balancer pin by source IP at most, which is enough for a fixed set of API clients and nothing more; the Load Balancer's HTTP probe takes the readiness path, NLB has no application probe.

8nginx, Caddy, Traefik, annotated#

One block per proxy, holding only the directives this guide argued for. nginx first, since its defaults break the most:

server {
    listen 443 ssl;
    server_name ai.example.com;
    # ssl_certificate / ssl_certificate_key here

    client_max_body_size 100m;            # at least the server's Max Upload Size

    location ~ ^/(admin|lmkit/v1/admin) { return 404; }   # admin stays private

    location / {
        proxy_pass http://127.0.0.1:5189;
        proxy_http_version 1.1;
        proxy_set_header Host $host;                  # public name reaches the server
        proxy_set_header X-Forwarded-Proto $scheme;   # honored once this proxy is in Security:TrustedProxies
        proxy_buffering off;                          # tokens leave as they arrive
        proxy_read_timeout 1h;                        # longest generation or sync task
    }
}

Caddy provisions the public certificate itself and sets the forwarded headers automatically, so the block is short:

ai.example.com {
    @admin path /admin* /lmkit/v1/admin/*
    respond @admin 404                    # admin stays private

    reverse_proxy 127.0.0.1:5189 {
        flush_interval -1                 # stream every chunk immediately
    }
}

Traefik, file-provider form; Traefik also sets the forwarded headers itself:

http:
  routers:
    lmkit:
      rule: "Host(`ai.example.com`)"
      entryPoints: [websecure]
      tls: {}
      service: lmkit
  services:
    lmkit:
      loadBalancer:
        servers: [{ url: "http://127.0.0.1:5189" }]
        responseForwarding:
          flushInterval: "-1"             # stream every chunk immediately

Two Traefik notes rather than directives, because both are version-dependent: check your version's entrypoint respondingTimeouts defaults against your longest generation, and if you enable the buffering middleware for body-size limits, set maxRequestBodyBytes at or above the server's upload limit. For the admin paths, route them on a higher-priority router restricted by an ipAllowList middleware, or do not route them on the public entrypoint at all.

9Keeping the admin surface private#

The admin console lives at /admin and its API under /lmkit/v1/admin/; the blocks above return 404 for both on the public host. This is defense in depth, not the only lock: the console still authenticates (password, two-factor, SSO), and each human-facing web destination (API reference, guides, playground, training, MCP guide) has its own served-or-not toggle and access level in the Interfaces section. The proxy rule makes the operator plane unreachable from outside, which is the posture most ingress policies expect. Reach the console from inside the network, or through a separate internal virtual host that only routes there.

10Stated plainly#

Turn proxy buffering off for API traffic, or streaming reads as broken: this is the single most common integration failure.
On a fleet, hash every request outside /admin, /lmkit/v1/admin and /playground by Authorization and pin those three by cookie or source address: the request that mints a job or starts a stream and the one that follows it must reach the same node. The server sets lmk-node=<tag> on fleet members and names its node in X-LMKit-Node on every response; HAProxy, nginx Plus (sticky route) and ALB route on that cookie, the others mint their own, and a 421 is the sign a pin was lost. A balancer without a header hash (Traefik OSS, ARR, ALB) cannot pin bearer clients.
Raise the upstream read timeout to the longest response you serve, and push bulk work to Prefer: respond-async.
The proxy's body limit must be at least the server's Max Upload Size, or clients get the proxy's error instead of the server's.
The server honors X-Forwarded-Proto and X-Forwarded-For only from proxies listed in Security:TrustedProxies; a TLS-terminating proxy that is not listed locks the admin surface to HTTPS and loopback, and client attribution then stays at the proxy.
A same-host proxy makes every caller loopback: disable anonymous access before it fronts anything.
GET /health for liveness (anonymous), GET /lmkit/v1/health/ready for readiness (anonymous, status only, 503 while the node warms or drains), GET /lmkit/v1/ready for the detailed body behind a bearer; every balancer health check takes the anonymous readiness route, and only Windows NLB, which probes nothing, loses the gate.