Durable Agent Runs
A chat request answers while the client waits. A run is the same agent on the same input, played as a durable job: every step is checkpointed in the server's shared store, a call that needs a person's approval pauses the run instead of failing it, and a run whose node restarts or disappears is picked up by another node from its last checkpoint. Runs are how an agent does work that outlives one HTTP request: a refund that waits for a manager, a cleanup that must not run twice, a task started from a webhook and read later.
A run plays exactly what a chat request with the same agent gets: its instructions, its tools under the calling key's grants, its knowledge and its memory. Nothing about the agent changes; only how its work is carried.
1Start a run#
curl -X POST https://localhost:7221/lmkit/v1/agents/support/runs \
-H "Authorization: Bearer $LMKIT_KEY" \
-H "Content-Type: application/json" \
-d '{ "input": "Refund order 4417, it arrived broken.", "user": "customer-8812" }'
The call answers when the run ends or stops. With "background": true (or the header
Prefer: respond-async) it answers 202 at once with the run's id, and the run is queued: the
node that took the call plays it at once when it has room, otherwise any node of the farm with
room does (see Queue and capacity).
| Field | Meaning |
|---|---|
input |
What the agent is asked to do. Required. |
user |
The end user the run acts for: agent memory is kept per end user, as on chat. |
memory |
A memory store to use instead of the agent's own. |
agent_revision |
The published revision to run. Omitted, the live revision is pinned at start. |
metadata |
Your own key/value pairs, stored with the run. |
background |
Answer 202 at once instead of waiting. |
A run keeps the revision it started on. Publishing a new revision of the agent never changes a run under way.
2Read, follow and list runs#
| Call | Answers |
|---|---|
GET /lmkit/v1/runs/{id} |
The run: status, output, what it waits for, error when it failed. |
GET /lmkit/v1/runs/{id}/events |
Server-sent events: one status event each time the run changes, until it ends or stops. A node reads every run its streams follow in one query per second, whatever the number of streams. |
GET /lmkit/v1/runs?status=awaiting_approval&limit=50 |
The caller's runs, newest first. |
GET /lmkit/v1/runs/{id} also returns the run's timeline, oldest first: starts and
resumptions, steps, every tool call with its outcome, duration and idempotency key, approvals
asked and decided (and by whom), questions, the hand-off and the end; and the tokens every play
of the run spent (prompt_tokens, completion_tokens). The timeline is part of the run's
checkpoint, so it reads the same from every node. Operators read every owner's runs, filtered by
agent and status, under Console › Observe › Runs; with telemetry export on, each play is an
agent_run.play span with the SDK's agent.run and agent.tool spans nested under it.
The same page sums up the last 24 hours above the list: runs, the share that completed, the
typical and slow duration (median and 95th percentile of the time finished runs spent working,
waits for a person left out) and how many runs wait for a person now. Its Agents view gives
one line per agent (runs, completed share, durations, most frequent failure), its Tools view
one line per tool (calls, errors and timeouts, durations), and Needs attention names an
agent failing more than the alert rule allows within the hour, with what failed most, and
approvals waiting longer than allowed. GET /lmkit/v1/admin/runs/overview returns the same
figures. Failed runs and long waits raise alerts too (see
Observability and Operations).
Runs belong to the owner of the API key that started them. Another owner's run answers 404,
exactly like jobs.
status |
The run |
|---|---|
running |
Plays on a node (or is about to be taken over by one). |
awaiting_approval |
Stopped before a call that needs approval: see pending_approvals. |
awaiting_input |
Stopped on a question to the user: see question. |
awaiting_reconciliation |
Stopped after a restart with a write whose outcome is unknown: see writes_in_doubt. |
completed, failed, cancelled, handed_off, max_iterations_reached |
Ended. |
3Approvals: which calls ask, and who decides#
A call needs approval when a rule holds its tool. Rules come from three places, and only ever add up:
| Where | How |
|---|---|
| Console › Build › Tools | Ask first on a tool, or Ask from risk level for every tool at or above a declared risk. |
| The agent definition | require_approval: tool names or patterns (filesystem_*), category:<name>, risk:<level>. |
| The tool itself | A tool that declares every call needs approval (deleting files, running processes, sending mail) always asks. |
Rules hold the tools this server runs, never tools a client declares in its request (those calls go back to the client). Inside a run, a held call pauses the run; in a plain chat request, nobody can wait for a decision, so the call is refused and the agent is told it needs a run.
A run stopped for approval lists each call it waits for with the exact arguments the agent produced and their hash:
"pending_approvals": [
{ "call_id": "c1", "tool": "issue_refund", "arguments": { "order": "4417" },
"arguments_hash": "9f2c...", "risk": "high", "side_effect": "NetworkWrite" }
]
Who decides. Operators decide every owner's calls from Console › Inbox › Approvals. An
application can host its own approval screen with an API key holding the approver grant
(Access › API keys, row menu): GET /lmkit/v1/approvals lists its owner's pending calls, and
curl -X POST https://localhost:7221/lmkit/v1/approvals/apr_.../approve \
-H "Authorization: Bearer $APPROVER_KEY" -H "Content-Type: application/json" \
-d '{ "arguments_hash": "9f2c..." }'
approves one (/deny with a reason denies it). The key that started a run never approves its
own calls unless Self-approval is on under Tools › Approvals; the same rule holds for
approvals sent through POST /lmkit/v1/runs/{id}/input.
- A decision is bound to the call and its arguments: a decision carrying another hash is
refused (
409on the queue,400on run input), so nobody approves a call they did not see. - A denied call never runs, and the agent is told the reason.
edited_argumentsapproves the call with arguments you corrected.- The run resumes once every call of the stopped round is decided.
- Approval timeout (Tools › Approvals, 24 hours by default): past it, the undecided calls are denied and the run continues without them. 0 waits without a deadline.
- Every decision is written to the security trail, with who made it.
Tasks. Each pending call is a task row of its own, opened by the same write that stops the
run and closed by the write that settles it (decided, cancelled, timed out, ended), so the inbox
never scans runs. GET /lmkit/v1/admin/tasks searches them (status open, closed or any;
workflow, owner), newest first, a page at a time: pass each page's next_cursor as cursor
and every task is read exactly once, however many there are. An operator takes a task with
POST /lmkit/v1/admin/approvals/{id}/claim (30 minutes, renewed by claiming again): while it is
held nobody else decides it (an Owner excepted) and a second claim is a 409 naming the holder.
The holder gives it back with /release or hands it on with /delegate ({"to": "carol"}).
A workflow's access can require several approvers per call (required_approvals, 2 for four
eyes): each approval is recorded until that many distinct operators approved, the operator who
started the run never counts, and a deny decides at once. escalate_after_minutes escalates an
approval waiting longer, once, through the notification channels; the row records it, so no
restart or other node sends it again.
A run waiting on a question takes { "text": "..." } on POST /lmkit/v1/runs/{id}/input; a run
waiting on reconciliation takes { "reconciled": { "c1": true } } (the write happened) or
false (it did not).
4Hand-offs, questions and escalations#
In the agent's People settings (Console › Agents), two switches let a run stop for a person:
- Hand over to a person: the agent gets a tool to hand the conversation over, with its
reason, a short summary and the facts it collected. The run ends
handed_offand opens an escalation. - Ask the user when something is missing: the agent asks one question instead of guessing;
the run waits
awaiting_inputuntilPOST /lmkit/v1/runs/{id}/inputbrings{ "text": "..." }.
An escalation is delivered where the agent's Deliver hand-offs to points: an https://
endpoint that receives it as JSON (your helpdesk), or a connector tool named connector/tool
called with it. Empty records it only. What the destination receives:
{ "type": "agent.handoff", "run_id": "run_...", "agent": "support",
"reason": "The customer asks for a refund above the limit",
"summary": "Order 4417 arrived broken; the customer wants 900 EUR back.",
"facts": { "order": "4417" }, "evidence_ids": [], "input": "...", "answer": "...",
"created_at": "2026-11-15T22:34:29Z" }
Operators work escalations from Console › Inbox › Escalations (take, resolve with a note);
applications read them with GET /lmkit/v1/escalations?status=open and move them with
POST /lmkit/v1/escalations/{run_id}/take and /resolve. A delivery that failed is shown on
the escalation, never retried silently. An escalation stays until it is resolved.
Notifications. The addresses under Tools › Mail and notifications receive an email when a call waits for approval and when an agent hands a conversation over; the admin panel raises an alert for both.
5Cancel#
POST /lmkit/v1/runs/{id}/cancel works from any node. A waiting run ends at once; a running one
is stopped by the node playing it within seconds.
Resume a failed workflow from its failing step. A failed workflow run reads failure: the
step, the error, its class and how many attempts the step made. POST /lmkit/v1/runs/{id}/retry
(by the run's owner) or Retry from the failed step in Console › Runs resumes it: the steps it
completed replay their recorded answer and are never run again, so a write already made is not
made twice; the failing step is attempted afresh. The body may correct the run:
{ "input": "Refund order 4417", "variables": { "region": "eu" }, "reconciled": { "update_crm": false }, "reason": "CRM was down" }
reconciled settles a write whose outcome is unknown (true: it happened, false: it did not). A
write left unsettled is sent again only under the same idempotency key to a tool that honors keys,
and the run otherwise waits in awaiting_reconciliation for a person to say. The retry answers
202 with the run queued, is recorded in the activity log with who and why, and only a failed
workflow run can be retried (409 otherwise). GET /lmkit/v1/admin/runs/incidents groups the
failed workflow runs of the last day by workflow, failing step and error class.
6What a restart or a lost node does#
A node holds a run it plays by a lease it renews every few seconds. When the lease lapses (the node stopped, crashed or lost its network), another node takes the run over and resumes it from its last checkpoint; a single node that restarts resumes its own runs. A run waiting for an approval or an answer holds no lease and costs nothing while it waits.
Each tool call is checkpointed before it is sent and after it answers, so a resumed run never
repeats a call that already answered. A write that was in flight when its node stopped is sent
again only to a tool that declares itself idempotent, with the same idempotency key; any other
such write stops the run in awaiting_reconciliation for a person to say whether it happened.
7What every resume checks again#
Nothing granted when a run started is assumed when it resumes. Each resume, on any node, reads:
| If | The run |
|---|---|
| The API key was revoked, deleted or expired | Fails authorization_revoked; no pending call runs. |
| The key's tool grants narrowed | A pending call outside them is denied (never run) and the agent is told. |
| The pinned agent revision was deleted | Fails agent_revision_deleted; it never switches to another revision. |
| The agent was deleted | Fails agent_not_found. |
| The model is no longer served | Fails model_unavailable. |
| The owner's last API key was deleted | Is cancelled and purged with its checkpoints. |
8Limits and retention#
- A run's checkpoint is capped at 16 MiB; a run past it fails
checkpoint_too_large. - Ended runs are kept
Runs:RetentionDays(30 by default), then deleted by every node's recovery pass with what refers to them (their approval tasks, their events not handed out). A workflow's access can keep its runs another number of days (retention_days), or put them under legal hold (legal_hold): none is deleted, by retention or by anyone, until it is lifted. A handed-off run is an escalation: it is kept until resolved, then swept after the retention. - Checkpoints (the run's input, every tool's arguments and answers) are encrypted at rest under
the key that seals the server's other secrets (
Runs:EncryptCheckpoints, on by default); runs written either way read either way. DELETE /lmkit/v1/runs/{id}deletes one of the caller's ended runs now (a run in progress is a 409: cancel it first; a held run is a 409), andGET /lmkit/v1/runs/exportanswers every run of the caller, one JSON run per line.DELETE /lmkit/v1/admin/runs/{id}does the same for any owner.
On a farm, runs live in the shared identity store: every node lists, reads, cancels and resumes the same runs, and recovery moves a run to a live node by itself (see Scaling out).
9Queue and capacity#
A node plays at most Runs:MaxConcurrent runs at once (0, the default, sizes it from the
processor count: twice the processors, at least 8). Runs a caller waits for count toward it but are
never refused; background runs, runs started by triggers, watched folders and schedules, runs taken
over after their node stopped, and workflow retries due later wait in the queue, which lives in the
shared store, so any node with room plays the next one and two nodes never play the same run.
| Setting | Effect |
|---|---|
Runs:MaxConcurrent |
Runs one node plays at once; 0 sizes it from the processors. |
Runs:MaxConcurrentPerOwner |
Runs one owner holds at once across the farm; 0 for no cap. |
The next run goes to the owner holding the fewest runs across the farm, so one owner's backlog
never holds another's runs back; a run taken over after its node stopped goes before new work, and
among an owner's runs a node prefers one whose model it already holds. A node that is shutting down
takes no new run. A queued run reads "queued": true; played_on names the node that played it
last. GET /lmkit/v1/admin/attention answers runs_queued and queue_lag_seconds (how long the
oldest has waited), and the metrics lmkit.agent.runs.queued, lmkit.agent.runs.queue_lag and
lmkit.agent.runs.playing follow them over time. A run whose node stopped holding it more than five
minutes before any node's recovery saw it is stranded: it is taken over like any other, and the
operators hear of it once (an alert, the Runs page's attention list, the lmkit.agent.runs.stranded
gauge). The Runs page's per-agent table adds the tokens each agent or workflow spent over the day.
Bounds keep a burst from taking the farm over. Runs:MaxQueuedPerOwner caps the runs one owner has
queued or under way at once: a start past it (the API, a trigger, a schedule, a watched folder, an
MCP call) is refused with 429 and Retry-After instead of growing the queue. A workflow's access can
cap its own runs played at once (max_concurrent_runs): its other runs wait in the queue while the
other workflows' runs go ahead. Runs:MaxParallelBranches (16 by default) bounds the branches of one
parallel step that play at once; the others start as running ones finish. Scaling down goes through
the drain: a node leaving takes no new run, and its runs under way move to the remaining nodes.
On a farm sharing one PostgreSQL, MySQL or SQL Server database, size each node's connection pool
(Maximum Pool Size in its connection string) so the nodes together stay under the database's
connection limit.