LM-Kit OneDocs2026.11.1lm-kit.comEULA

Durable Agent Runs

A chat request answers while the client waits. A run is the same agent on the same input, played as a durable job: every step is checkpointed in the server's shared store, a call that needs a person's approval pauses the run instead of failing it, and a run whose node restarts or disappears is picked up by another node from its last checkpoint. Runs are how an agent does work that outlives one HTTP request: a refund that waits for a manager, a cleanup that must not run twice, a task started from a webhook and read later.

A run plays exactly what a chat request with the same agent gets: its instructions, its tools under the calling key's grants, its knowledge and its memory. Nothing about the agent changes; only how its work is carried.


1Start a run#

curl -X POST https://localhost:7221/lmkit/v1/agents/support/runs \
  -H "Authorization: Bearer $LMKIT_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "input": "Refund order 4417, it arrived broken.", "user": "customer-8812" }'

The call answers when the run ends or stops. With "background": true (or the header Prefer: respond-async) it answers 202 at once with the run's id, and the run is queued: the node that took the call plays it at once when it has room, otherwise any node of the farm with room does (see Queue and capacity).

Field Meaning
input What the agent is asked to do. Required.
user The end user the run acts for: agent memory is kept per end user, as on chat.
memory A memory store to use instead of the agent's own.
agent_revision The published revision to run. Omitted, the live revision is pinned at start.
metadata Your own key/value pairs, stored with the run.
background Answer 202 at once instead of waiting.

A run keeps the revision it started on. Publishing a new revision of the agent never changes a run under way.

2Read, follow and list runs#

Call Answers
GET /lmkit/v1/runs/{id} The run: status, output, what it waits for, error when it failed.
GET /lmkit/v1/runs/{id}/events Server-sent events: one status event each time the run changes, until it ends or stops. A node reads every run its streams follow in one query per second, whatever the number of streams.
GET /lmkit/v1/runs?status=awaiting_approval&limit=50 The caller's runs, newest first.

GET /lmkit/v1/runs/{id} also returns the run's timeline, oldest first: starts and resumptions, steps, every tool call with its outcome, duration and idempotency key, approvals asked and decided (and by whom), questions, the hand-off and the end; and the tokens every play of the run spent (prompt_tokens, completion_tokens). The timeline is part of the run's checkpoint, so it reads the same from every node. Operators read every owner's runs, filtered by agent and status, under Console › Observe › Runs; with telemetry export on, each play is an agent_run.play span with the SDK's agent.run and agent.tool spans nested under it.

The same page sums up the last 24 hours above the list: runs, the share that completed, the typical and slow duration (median and 95th percentile of the time finished runs spent working, waits for a person left out) and how many runs wait for a person now. Its Agents view gives one line per agent (runs, completed share, durations, most frequent failure), its Tools view one line per tool (calls, errors and timeouts, durations), and Needs attention names an agent failing more than the alert rule allows within the hour, with what failed most, and approvals waiting longer than allowed. GET /lmkit/v1/admin/runs/overview returns the same figures. Failed runs and long waits raise alerts too (see Observability and Operations).

Runs belong to the owner of the API key that started them. Another owner's run answers 404, exactly like jobs.

status The run
running Plays on a node (or is about to be taken over by one).
awaiting_approval Stopped before a call that needs approval: see pending_approvals.
awaiting_input Stopped on a question to the user: see question.
awaiting_reconciliation Stopped after a restart with a write whose outcome is unknown: see writes_in_doubt.
completed, failed, cancelled, handed_off, max_iterations_reached Ended.

3Approvals: which calls ask, and who decides#

A call needs approval when a rule holds its tool. Rules come from three places, and only ever add up:

Where How
Console › Build › Tools Ask first on a tool, or Ask from risk level for every tool at or above a declared risk.
The agent definition require_approval: tool names or patterns (filesystem_*), category:<name>, risk:<level>.
The tool itself A tool that declares every call needs approval (deleting files, running processes, sending mail) always asks.

Rules hold the tools this server runs, never tools a client declares in its request (those calls go back to the client). Inside a run, a held call pauses the run; in a plain chat request, nobody can wait for a decision, so the call is refused and the agent is told it needs a run.

A run stopped for approval lists each call it waits for with the exact arguments the agent produced and their hash:

"pending_approvals": [
  { "call_id": "c1", "tool": "issue_refund", "arguments": { "order": "4417" },
    "arguments_hash": "9f2c...", "risk": "high", "side_effect": "NetworkWrite" }
]

Who decides. Operators decide every owner's calls from Console › Inbox › Approvals. An application can host its own approval screen with an API key holding the approver grant (Access › API keys, row menu): GET /lmkit/v1/approvals lists its owner's pending calls, and

curl -X POST https://localhost:7221/lmkit/v1/approvals/apr_.../approve \
  -H "Authorization: Bearer $APPROVER_KEY" -H "Content-Type: application/json" \
  -d '{ "arguments_hash": "9f2c..." }'

approves one (/deny with a reason denies it). The key that started a run never approves its own calls unless Self-approval is on under Tools › Approvals; the same rule holds for approvals sent through POST /lmkit/v1/runs/{id}/input.

  • A decision is bound to the call and its arguments: a decision carrying another hash is refused (409 on the queue, 400 on run input), so nobody approves a call they did not see.
  • A denied call never runs, and the agent is told the reason.
  • edited_arguments approves the call with arguments you corrected.
  • The run resumes once every call of the stopped round is decided.
  • Approval timeout (Tools › Approvals, 24 hours by default): past it, the undecided calls are denied and the run continues without them. 0 waits without a deadline.
  • Every decision is written to the security trail, with who made it.

Tasks. Each pending call is a task row of its own, opened by the same write that stops the run and closed by the write that settles it (decided, cancelled, timed out, ended), so the inbox never scans runs. GET /lmkit/v1/admin/tasks searches them (status open, closed or any; workflow, owner), newest first, a page at a time: pass each page's next_cursor as cursor and every task is read exactly once, however many there are. An operator takes a task with POST /lmkit/v1/admin/approvals/{id}/claim (30 minutes, renewed by claiming again): while it is held nobody else decides it (an Owner excepted) and a second claim is a 409 naming the holder. The holder gives it back with /release or hands it on with /delegate ({"to": "carol"}). A workflow's access can require several approvers per call (required_approvals, 2 for four eyes): each approval is recorded until that many distinct operators approved, the operator who started the run never counts, and a deny decides at once. escalate_after_minutes escalates an approval waiting longer, once, through the notification channels; the row records it, so no restart or other node sends it again.

A run waiting on a question takes { "text": "..." } on POST /lmkit/v1/runs/{id}/input; a run waiting on reconciliation takes { "reconciled": { "c1": true } } (the write happened) or false (it did not).

4Hand-offs, questions and escalations#

In the agent's People settings (Console › Agents), two switches let a run stop for a person:

  • Hand over to a person: the agent gets a tool to hand the conversation over, with its reason, a short summary and the facts it collected. The run ends handed_off and opens an escalation.
  • Ask the user when something is missing: the agent asks one question instead of guessing; the run waits awaiting_input until POST /lmkit/v1/runs/{id}/input brings { "text": "..." }.

An escalation is delivered where the agent's Deliver hand-offs to points: an https:// endpoint that receives it as JSON (your helpdesk), or a connector tool named connector/tool called with it. Empty records it only. What the destination receives:

{ "type": "agent.handoff", "run_id": "run_...", "agent": "support",
  "reason": "The customer asks for a refund above the limit",
  "summary": "Order 4417 arrived broken; the customer wants 900 EUR back.",
  "facts": { "order": "4417" }, "evidence_ids": [], "input": "...", "answer": "...",
  "created_at": "2026-11-15T22:34:29Z" }

Operators work escalations from Console › Inbox › Escalations (take, resolve with a note); applications read them with GET /lmkit/v1/escalations?status=open and move them with POST /lmkit/v1/escalations/{run_id}/take and /resolve. A delivery that failed is shown on the escalation, never retried silently. An escalation stays until it is resolved.

Notifications. The addresses under Tools › Mail and notifications receive an email when a call waits for approval and when an agent hands a conversation over; the admin panel raises an alert for both.

5Cancel#

POST /lmkit/v1/runs/{id}/cancel works from any node. A waiting run ends at once; a running one is stopped by the node playing it within seconds.

Resume a failed workflow from its failing step. A failed workflow run reads failure: the step, the error, its class and how many attempts the step made. POST /lmkit/v1/runs/{id}/retry (by the run's owner) or Retry from the failed step in Console › Runs resumes it: the steps it completed replay their recorded answer and are never run again, so a write already made is not made twice; the failing step is attempted afresh. The body may correct the run:

{ "input": "Refund order 4417", "variables": { "region": "eu" }, "reconciled": { "update_crm": false }, "reason": "CRM was down" }

reconciled settles a write whose outcome is unknown (true: it happened, false: it did not). A write left unsettled is sent again only under the same idempotency key to a tool that honors keys, and the run otherwise waits in awaiting_reconciliation for a person to say. The retry answers 202 with the run queued, is recorded in the activity log with who and why, and only a failed workflow run can be retried (409 otherwise). GET /lmkit/v1/admin/runs/incidents groups the failed workflow runs of the last day by workflow, failing step and error class.

6What a restart or a lost node does#

A node holds a run it plays by a lease it renews every few seconds. When the lease lapses (the node stopped, crashed or lost its network), another node takes the run over and resumes it from its last checkpoint; a single node that restarts resumes its own runs. A run waiting for an approval or an answer holds no lease and costs nothing while it waits.

Each tool call is checkpointed before it is sent and after it answers, so a resumed run never repeats a call that already answered. A write that was in flight when its node stopped is sent again only to a tool that declares itself idempotent, with the same idempotency key; any other such write stops the run in awaiting_reconciliation for a person to say whether it happened.

7What every resume checks again#

Nothing granted when a run started is assumed when it resumes. Each resume, on any node, reads:

If The run
The API key was revoked, deleted or expired Fails authorization_revoked; no pending call runs.
The key's tool grants narrowed A pending call outside them is denied (never run) and the agent is told.
The pinned agent revision was deleted Fails agent_revision_deleted; it never switches to another revision.
The agent was deleted Fails agent_not_found.
The model is no longer served Fails model_unavailable.
The owner's last API key was deleted Is cancelled and purged with its checkpoints.

8Limits and retention#

  • A run's checkpoint is capped at 16 MiB; a run past it fails checkpoint_too_large.
  • Ended runs are kept Runs:RetentionDays (30 by default), then deleted by every node's recovery pass with what refers to them (their approval tasks, their events not handed out). A workflow's access can keep its runs another number of days (retention_days), or put them under legal hold (legal_hold): none is deleted, by retention or by anyone, until it is lifted. A handed-off run is an escalation: it is kept until resolved, then swept after the retention.
  • Checkpoints (the run's input, every tool's arguments and answers) are encrypted at rest under the key that seals the server's other secrets (Runs:EncryptCheckpoints, on by default); runs written either way read either way.
  • DELETE /lmkit/v1/runs/{id} deletes one of the caller's ended runs now (a run in progress is a 409: cancel it first; a held run is a 409), and GET /lmkit/v1/runs/export answers every run of the caller, one JSON run per line. DELETE /lmkit/v1/admin/runs/{id} does the same for any owner.

On a farm, runs live in the shared identity store: every node lists, reads, cancels and resumes the same runs, and recovery moves a run to a live node by itself (see Scaling out).

9Queue and capacity#

A node plays at most Runs:MaxConcurrent runs at once (0, the default, sizes it from the processor count: twice the processors, at least 8). Runs a caller waits for count toward it but are never refused; background runs, runs started by triggers, watched folders and schedules, runs taken over after their node stopped, and workflow retries due later wait in the queue, which lives in the shared store, so any node with room plays the next one and two nodes never play the same run.

Setting Effect
Runs:MaxConcurrent Runs one node plays at once; 0 sizes it from the processors.
Runs:MaxConcurrentPerOwner Runs one owner holds at once across the farm; 0 for no cap.

The next run goes to the owner holding the fewest runs across the farm, so one owner's backlog never holds another's runs back; a run taken over after its node stopped goes before new work, and among an owner's runs a node prefers one whose model it already holds. A node that is shutting down takes no new run. A queued run reads "queued": true; played_on names the node that played it last. GET /lmkit/v1/admin/attention answers runs_queued and queue_lag_seconds (how long the oldest has waited), and the metrics lmkit.agent.runs.queued, lmkit.agent.runs.queue_lag and lmkit.agent.runs.playing follow them over time. A run whose node stopped holding it more than five minutes before any node's recovery saw it is stranded: it is taken over like any other, and the operators hear of it once (an alert, the Runs page's attention list, the lmkit.agent.runs.stranded gauge). The Runs page's per-agent table adds the tokens each agent or workflow spent over the day.

Bounds keep a burst from taking the farm over. Runs:MaxQueuedPerOwner caps the runs one owner has queued or under way at once: a start past it (the API, a trigger, a schedule, a watched folder, an MCP call) is refused with 429 and Retry-After instead of growing the queue. A workflow's access can cap its own runs played at once (max_concurrent_runs): its other runs wait in the queue while the other workflows' runs go ahead. Runs:MaxParallelBranches (16 by default) bounds the branches of one parallel step that play at once; the others start as running ones finish. Scaling down goes through the drain: a node leaving takes no new run, and its runs under way move to the remaining nodes.

On a farm sharing one PostgreSQL, MySQL or SQL Server database, size each node's connection pool (Maximum Pool Size in its connection string) so the nodes together stay under the database's connection limit.