Reference Architectures

Seven named deployment patterns, each a shape you can sketch into a design document: what it is for, its honest boundaries, the sizing pointer, and the guides that carry the depth. Written for the architect drawing the deployment before the pilot, and it closes with the section that decides sign-off: the ladder for when one box is no longer enough.

1The workstation#

One user, one machine, nothing exposed. The tray application supervises the server in desktop mode, the listeners stay on loopback, and local calls need no key, so there is no credential ceremony at all (Going Live explains the posture). This is the evaluation shape, the developer laptop, and the private assistant at home (A Private ChatGPT at Home); it is also a complete production deployment for a single professional whose data must not leave the desk.

• Size it. Pick the model to the machine (The Right Model for Your Machine), then verify the GPU is actually carrying it (GPUs and Backends).
• Composes. Going Live section 4 (desktop mode), Choosing and Operating Models.

2The team server#

One box, bought once, shared by a team. The server runs as a Windows service or under systemd, the network posture flips to Network in Ports (from that moment every API call authenticates, no exceptions), the admin console gets a password, and HTTPS comes from the self-managed certificate, ACME, or your own ingress (Behind a Reverse Proxy). Mint one key per consuming system in Access and the request trail attributes load for free.

• Size it. Two numbers: concurrent callers at peak (slots) and the longest prompt-plus-answer you must honor (window); the panel prices the shape against the device before you commit (Inference Capacity).
• Composes. Going Live, Keys and Authentication, Observability; Windows fleets add Windows Server Deployments.

3The department server#

The team server, grown into an organization: several teams, hard data boundaries, central identity, central monitoring. The owner boundary isolates each credential's files, stores, memories, and jobs automatically; grants decide what each key may do; Search adds tenants and clusters for the harder cases (One Server, Many Teams assembles the whole tenancy picture). Sign-in goes through your identity provider (Single Sign-On), server-executed tools run behind an explicit egress policy (Tools and Egress), and telemetry exports to your monitoring stack over OpenTelemetry or Prometheus (Observability).

• Size it. Compute is shared by design: size slots for the sum of the teams and choose the saturation policy deliberately (Inference Capacity); give outlier models per-model overrides in Inference instead of resizing the server around them.
• Composes. One Server, Many Teams, Single Sign-On, The Security Model, Backup and Upgrades.

4The document factory#

A server whose job is throughput, not conversation: scanned batches in, validated structured data out. Every stage of the pipeline (capture, split, classify, extract, redact, archive) is a stateless call over a shared file id (The IDP Pipeline), and bulk callers send Prefer: respond-async so long work becomes a polled job from the first request (Errors, Retries, and Jobs). Jobs and their outcomes are watched from Jobs, queue depth from the dashboard.

• Size it. The opposite trade from chat: fewer slots, windows sized to your real documents rather than the model's trained maximum (Inference Capacity), and disk for the upload volume plus retention you configure. Provisioning the box itself is Hardware Sizing.
• Composes. The IDP Pipeline, Document Processing, Structured Extraction, Intelligent Splitting.

5The search platform#

A server whose center of gravity is the index. Search works out of the box on the auto-provisioned embedded cluster; the shape grows into PostgreSQL when the review wants row-level security below the application or the volume wants partitioned storage, and onto SQL Server or MySQL plus Qdrant where database policy dictates (Storage Engines and Deployment). Clusters, tenants, and collections keep deployment, isolation, and dataset scoping as three independent decisions (Clusters, Tenants, Collections), and search reach is always an explicit per-key grant (Access Model).

• Size it. Embedding runs on its own inference pool, so bulk indexing never steals chat slots; size the embedding window and the chat pool separately (Inference Capacity, Embeddings and Reranking). External engines ride the database sizing and backup practice your operations team already has.
• Composes. The Search Engine, plus the three guides above.

6The air-gapped enclave#

Zero egress, by construction rather than by firewall heroics. Models are pulled on a connected build machine with the CLI and shipped as files, the golden image carries the state directory, certificates, and pinned path variables, and the five potential egress channels are closed in configuration an auditor can read back (Air-Gapped Deployment is the complete exercise). The guides and API reference are served by the server itself, so the manuals live inside the perimeter.

• Size it. Same arithmetic as any other shape, decided before the image ships: get the slot shape and model set right on the build side, because iteration is slower once files travel by hand.
• Composes. Air-Gapped Deployment, Where Data Lives, The Command Line, Backup and Upgrades.

7The fleet#

The team or department server, multiplied: identical nodes behind an ordinary load balancer, sharing an admin domain (operators, API keys, stored responses in one dedicated database), an upload volume, and a model volume, with configuration owned by the deployment manifest so no node can drift. Any node serves any API request; nodes join warm (readiness waits for the model lineup), leave draining (in-flight streams finish before the node exits), and the fleet grows or shrinks on the saturation gauge, with KEDA driving it on Kubernetes and the same signal driving anything else. Scaling Out is the complete contract, including what honestly stays per node.

• Size it. Size ONE node with Inference Capacity, then multiply: every node is a complete engine, so capacity is per-node throughput times node count. One request never spans machines; a model too large for one node needs a bigger node.
• Composes. Scaling Out, Running in Containers (the manifests), Behind a Reverse Proxy, Storage Engines and Deployment (fleet search runs on external engines).

8When one box is not enough#

The ladder, in the order to climb it:

1. Size the one box up first. Slots, windows, KV quantization, and per-model overrides stretch a machine further than intuition suggests, and several GPUs pool memory for models no single card holds (Hardware Sizing, Inference Capacity). Most deployments that feel out of capacity are out of tuning.
2. Scale out by replication. The fleet shape above: identical nodes behind a load balancer, the shared admin domain and volumes making any node serve any request, warm joins and lossless drains making growth and shrinkage invisible to callers (Scaling Out). This is the built-in path for more of the same traffic.
3. Dedicate servers or fleets per workload or per team. Independent installations behind ordinary routing (a DNS name per server, or host- and path-based rules at your proxy): the chat fleet, the document factory, the search platform, each sized to its own traffic shape. This is the same boundary One Server, Many Teams names for regulatory separation, applied to capacity; choose it when the workloads' shapes or audit domains differ, not merely their volume.

And the availability answer, stated plainly: a fleet IS the failover story for the data plane: any node serves any request, so losing one loses only its in-flight work and its own operator history, while keys, sessions, stored responses, uploads, and search data live in the shared stores your database and storage practice already protects. For single-box deployments the answer remains the rehearsed restore from Backup and Upgrades: the state directory and model directory onto a standby machine, started when needed, measured in minutes.

9Stated plainly#

• Seven shapes cover the ground from one desk to a load-balanced fleet; every one composes guides that already exist, so the depth is one link away.
• Compute is shared inside a node and never spans nodes: one request runs on one machine, and a fleet is more machines, not bigger requests.
• Scale by tuning the box, then by replicating it behind a load balancer, then by dedicating servers or fleets per workload; regulatory separation still means separate installations.
• A fleet is the live availability answer for the data plane; the single box keeps the rehearsed-restore answer, measured in minutes.