Fleet Recipes

Scaling Out is the contract: identical nodes behind a balancer, one shared identity database, shared volumes for uploads and models, and a lifecycle that joins warm and leaves draining. This page is the per-platform recipe for that one contract. Every section answers the same three questions, because they are the only ones that change from one platform to the next: which probe path the platform can reach, how long it lets a node stop, and where the shared paths mount. Everything else is the environment block from Scaling Out section 10, unchanged.

Two routes serve every prober. GET /lmkit/v1/health/ready is anonymous and answers the readiness word alone (200 ready, 503 with warming, draining or not_ready); it is the path for a platform health check that cannot send a header. GET /lmkit/v1/ready carries the detailed body behind a bearer key for probers that can. GET /health is liveness only and answers 200 while the process is alive, warming or draining included, so it is never a routing probe.

A stop needs DrainAdvertiseSeconds + ShutdownDrainSeconds + 15 seconds of teardown. The server derives the advertise default from the platform it detects (5 seconds on Kubernetes, App Service, Container Apps and ECS, 0 elsewhere) and logs one warning at startup when the platform's stop allowance is shorter than that sum, naming the setting to raise. Where a platform caps the allowance, lower ShutdownDrainSeconds to fit; the default of 100 seconds assumes a supervisor that can wait two minutes.

1Azure App Service (Web App for Containers, Linux)#

A CPU-only farm: App Service instances carry no GPU, so this is the right home for embeddings, document processing and small chat models, and the wrong one for a large-model lineup (section 2 is the GPU-capable Azure target). The platform fronts every request, terminates TLS and scales instances of one container image.

/home is the one path the platform shares between every instance of the plan (with WEBSITES_ENABLE_APP_SERVICE_STORAGE), and it is exactly where a state directory must never be: two instances would write one keyring, one identity database and one set of embedded stores at once. The server refuses to start with its state directory under /home on App Service, in every mode, and names LMKIT_STATE_DIR as the remedy; set it to a directory the image creates, for example /var/lmkit/state. The rest of the block:

• WEBSITES_PORT=5189 tells the platform which container port to route to (the server also honours a PORT the platform injects). The platform detects nothing about HttpPort itself.
• Security__TrustedProxies__0=* is accepted here, because the instance is reachable only through the platform front and the front's addresses cannot be listed; the server logs that condition once at startup. Set Security__PublicBaseUrl=https://<app>.azurewebsites.net (or your custom domain) so the SSO callback and the Secure flag on cookies follow the front, and Https__Enabled=false because TLS ends at the front.
• The node name is the platform's WEBSITE_INSTANCE_ID, read automatically.
• Browser affinity is the platform's own ARR affinity cookie (the session affinity setting of the app); bearer API clients need none (Scaling Out section 6).
• The scale signal is per node on /metrics, but App Service autoscale cannot scrape it: export through OTLP to your monitoring backend and drive an autoscale rule from the custom metric, or scale on the platform's CPU and request metrics as an approximation.

2Azure Container Apps#

The GPU-capable Azure target: serverless GPUs (NVIDIA A100 and T4) on Consumption workload profiles, KEDA scaling built in, one container per replica with the GPU. Scale to zero is possible on the platform and still not this product's shape: keep the minimum replica count at one, because a cold node loads models for minutes before it can serve.

• The ingress target port is the server's HttpPort (5189); the platform sets CONTAINER_APP_PORT to it and CONTAINER_APP_REPLICA_NAME to the replica's name, which the server takes as its node name.
• Security__TrustedProxies__0=* is accepted (platform front); set Security__PublicBaseUrl to the app's public address.
• Scale rules: a custom KEDA scaler of type prometheus reading avg(lmkit_inference_slots_saturation) from a Prometheus that scrapes the replicas, or the HTTP concurrency rule as the fallback when no Prometheus exists.

3Kubernetes#

The complete manifests (single pod and fleet, the KEDA ScaledObject in both trigger styles) live in Running in Containers sections 7 and 9; this section only states the three answers.

The advertise window defaults to 5 seconds because the platform is detected; nothing else is platform-specific.

4Plain VMs and Windows service farms#

Two or more machines running the Linux package under systemd or the Windows service, a balancer in front, PostgreSQL for the identity database, an NFS or SMB share for models and uploads: the fleet of Scaling Out section 10, whose environment block applies verbatim.

• Set DrainAdvertiseSeconds=5 explicitly: the default is 0 under systemd and the Windows service, where nothing routes by readiness unless you put a balancer in front, which you did.
• Name the balancer in Security__TrustedProxies__0 by address or the narrowest CIDR. The * sentinel is refused on these platforms unless Security__ProxyFrontOnly=true states that the machines are unreachable except through the front.
• Deliver the environment machine-wide: Environment= or EnvironmentFile= in the unit, setx /M or Group Policy on Windows, so the service and the lmkit CLI see the same keys (The Command Line).
• Pinning for the node-bound surfaces: HAProxy cookie lmk-node prefix or its own inserted cookie, IIS ARR client affinity (a cookie the proxy inserts), Windows NLB single affinity by client address, Azure Load Balancer session persistence by client IP, F5 cookie persistence (Behind a Reverse Proxy section 7 has the configuration lines).

5Docker Compose replicas and Docker Swarm#

A single host or a Swarm can run several nodes from one Compose file, with two changes to the single-node file of Running in Containers section 6.

• Drop the fixed ports: publication: two replicas cannot bind one host port. Route through the front proxy on the Compose network instead, and pin there (Swarm's mesh has no stickiness).
• Give each replica its own name with a templated hostname (hostname: "lmkit-{{.Task.Slot}}" on Swarm) or LMKIT_NODE_NAME, so training journals and the node tag stay stable across recreation.
• Set DrainAdvertiseSeconds=5: the default is 0 for a bare container.

6Amazon ECS#

GPU tasks run on the EC2 launch type with a GPU capacity provider; Fargate is CPU-only. An Application Load Balancer fronts the service.

• The server detects the platform through the task metadata variable the agent injects and takes the container hostname as its node name.
• Pinning: target-group stickiness with an application-based cookie named lmk-node (the server's own marker), or the balancer's duration-based cookie.
• Trust the balancer by listing its subnets in Security__TrustedProxies, or use * with Security__ProxyFrontOnly=true when the tasks sit in private subnets reachable only through the balancer.
• Scale: a target-tracking policy on a custom CloudWatch metric fed from the exported saturation gauge (through an OTLP collector or Amazon Managed Prometheus), or request-count tracking on the target group as the approximation.

7Stated plainly#

• One contract, six recipes: the probe path, the stop allowance and the shared paths are the only things that change between platforms; the environment block does not.
• Header-less probers use /lmkit/v1/health/ready; /health is liveness and never a routing decision.
• App Service and ECS cap a stop at 120 seconds: lower ShutdownDrainSeconds to 90 there.
• A state directory is never shared: App Service's /home is refused, a second live server on one state directory is refused, and every recipe puts state on the node's own disk.