Agent Workflows
An agent answers one request with its own judgment. A workflow fixes the order: classify, then
route; draft, then check, then ask a person before anything is sent. It is a JSON document of steps,
stored and versioned like an agent, and every run of it is a durable run: it
checkpoints after each step, stops for approvals and questions, survives restarts and lost nodes,
and is read, continued and cancelled through the same runs/{id} endpoints.
1The definition#
{
"schema_version": 1,
"name": "support-triage",
"description": "Classify a request, answer it, and ask before a refund.",
"steps": [
{ "id": "classify", "type": "agent", "agent": "classifier", "input": "{{input}}" },
{ "id": "route", "type": "condition",
"branches": [ { "when": "contains(steps.classify.output, 'refund')",
"steps": [ { "id": "check", "type": "approval", "message": "Refund requested: continue?" },
{ "id": "refund", "type": "agent", "agent": "refunds" } ] } ],
"else": [ { "id": "answer", "type": "agent", "agent": "support" } ] }
]
}
| Step type | What it does |
|---|---|
agent |
Runs a published agent on input (a template). Pin a revision with revision. |
tool |
Calls one tool with arguments; its answer is the step's output. |
condition |
Runs the first branch whose when holds, else else. |
parallel |
Runs its branches (each a list of steps) side by side; aggregate is concat or first. |
loop |
Repeats its steps until until holds, at most max_iterations times. |
foreach |
Runs its steps once for every item of the JSON array items evaluates to; they read it as item. |
workflow |
Runs another published workflow (workflow, optionally at revision) on input; its answer is the step's output. |
wait |
Waits delay_seconds, or until wait_until (an ISO 8601 moment, a template). |
wait_for_event |
Stops until another system delivers event with the correlation key; the payload is the step's output. |
approval |
Stops the workflow until a person approves; a denial ends it. With document, it is a review: the person reads that document before deciding. With approve_when, the cases the condition settles pass without a person. |
handoff |
Ends the workflow handed over to a person, with reason and summary. |
Templates and conditions read input, steps.<id>.output, steps.<id>.error and vars.<name>
(the run's variables). A field of a JSON input or answer reads as input.<field> or
steps.<id>.output.<field> (dotted, array items by index); a missing field reads as null, and
coalesce(a, b) takes b when a is empty. output (a template) is the workflow's answer; without it, the last step's output is.
A tool step fails when its tool's answer reports a failure ({"error": ...}, or "ok": false
with the status from an HTTP tool), so a refused call never passes on as a result.
Undoing what a failed run did. An agent or tool step can declare compensate: the agent or
tool steps that undo it (release a reservation, void a payment). When a later step fails the
workflow for good, the compensations of the steps that completed run newest first, each once, in
the loop pass their step ran in; then the run ends failed. Each compensation's progress is
checkpointed, so a run interrupted while undoing resumes the compensations left and never runs the
workflow again; their tool calls carry their own idempotency keys. A compensation that fails stops
the undoing and leaves the run failed with compensation_failed, the rest listed as pending
(compensations on the run), for an operator; a run that undid steps is never retried from its
failing step. A compensation cannot stop for a person or wait.
{ "id": "reserve", "type": "tool", "tool": "reserve_room", "arguments": { "date": "{{input.date}}" },
"compensate": [ { "id": "release", "type": "tool", "tool": "release_room",
"arguments": { "reservation": "{{steps.reserve.output.id}}" } } ] }
Waiting. A wait step waits up to 366 days. A wait longer than 30 seconds holds no worker and
no lease: the run is parked in the queue (resume_after) and the first node with room resumes it
when the moment comes, across restarts and upgrades. A wait_for_event step stops the run in
awaiting_event until another system delivers the event: POST /lmkit/v1/events/{event} with
{"correlation": "A-17", "payload": {...}} under an API key of the run's owner, or a signed trigger
whose target is the event (its input maps the correlation key, its body is the payload). Each waiting
run takes an event once; a repeated delivery finds nothing waiting, and another owner's delivery never
reaches it. timeout_seconds is the deadline: past it the step fails as a timeout, so its
on_error steps are the fallback. The run reads waiting_for (event, correlation, deadline).
{ "id": "paid", "type": "wait_for_event", "event": "payment-confirmed", "correlation": "{{input.order_id}}",
"timeout_seconds": 604800,
"on_error": { "steps": [ { "id": "chase", "type": "agent", "agent": "collections", "input": "{{input.order_id}}" } ] } }
Reusing a workflow. A workflow step runs another published workflow as a child, under the
same API key and owner, so it can do nothing the run itself cannot. The child keeps the revision it
started on (the step's revision, else the live one at its first run) for the rest of the run, even
if it is published again while the parent waits; its stops for a person are the parent's, answered
the same way, and a failed child fails the step, so the step's on_error applies (retries belong to
the child's own steps). The run reads children while a child is under way, each with its own run
id (<parent>/<step>). Publishing refuses a child that is not published (missing_workflow),
workflows that run each other in a circle or nest deeper than 8 (workflow_cycle), and a child that
can stop for a person inside a parallel branch or items running at once.
Many items. A foreach step runs its steps for every item of an array (items: an expression
such as input.documents or steps.list.output.rows); the steps read the item as item (fields as
item.<field>) and its position as loop.iteration. max_concurrency (1 to 32) runs that many at
once, batch_size hands the steps several items at a time (item is then their array), filter
leaves items out, and collect (a template) says what each item gives. The step answers the array
of results in the items' order (order: completion for the order they finished in), or with
keep_results: false only the counts, which suits thousands of items. on_item_error decides what a
failed item does: fail (stop starting items and fail), skip, or collect (an {"error": ...} in
its place). Each item's steps keep their outputs to that item, and a finished item's record is
dropped, so the run's checkpoint does not grow with the items; a run resumed after a restart skips
the items done and never writes a done item again. A step accepts at most max_items (10000 by
default, up to 100000). Items running at once cannot stop for a person, as parallel branches cannot;
for one run per file, a watched folder remains the simpler shape.
{ "id": "each", "type": "foreach", "items": "input.invoices", "max_concurrency": 4, "on_item_error": "skip",
"steps": [ { "id": "extract", "type": "agent", "agent": "invoice-reader", "input": "{{item.file}}" } ],
"collect": "{{steps.extract.output}}" }
Typed inputs and answers. input_schema is the JSON Schema a run's input meets: the input is
then JSON (sent as an object or as text), and a run whose input breaks the schema is refused with
400 before it exists, so nothing is written on it. output_schema on the workflow checks its
answer (a run whose answer breaks it fails output_invalid), and on an agent or tool step checks
that step's answer: an agent is asked for that shape, and an answer that breaks it fails the
attempt as an error, so the step's retry and on_error apply. A templated tool argument is cast
to the type the tool's own input schema declares (number, integer, boolean, object, array); a value
that cannot be cast fails the step before the call. At publish, a tool step's fixed arguments are
checked against the tool's schema (argument_mismatch: a required argument missing, one the tool
does not take, a wrong type or value). Workflows without schemas run as before.
{ "name": "ticket-intake",
"input_schema": { "type": "object", "required": ["subject", "priority"],
"properties": { "subject": { "type": "string" }, "priority": { "type": "integer", "minimum": 1, "maximum": 5 } } },
"steps": [ { "id": "create", "type": "tool", "tool": "helpdesk_create",
"arguments": { "title": "{{input.subject}}", "priority": "{{input.priority}}" } } ] }
When a step fails. An agent or tool step may set timeout_seconds (the most one attempt
takes), retry and on_error:
{ "id": "update_crm", "type": "tool", "tool": "http_post", "arguments": { "url": "https://crm.example/api/tickets" },
"timeout_seconds": 30,
"retry": { "max_attempts": 4, "delay_seconds": 2, "backoff": "exponential", "max_delay_seconds": 60 },
"on_error": { "then": "continue",
"steps": [ { "id": "tell_ops", "type": "tool", "tool": "notify", "arguments": { "text": "CRM update failed: {{steps.update_crm.error}}" } } ] } }
retry tries again what retry_on lists: timeout and transient (HTTP 408, 429, 5xx, or a call
that did not complete) by default, error (any other failure, such as HTTP 400) only when listed.
The wait doubles after each attempt (backoff: fixed keeps it), with a little jitter, up to
max_delay_seconds (at most 3600). A write is tried again under the same idempotency key; a write
to a tool that honors no key is tried again only when the other system said it did not process the
call (429 or 503), and a timed-out write waits for a person's answer as after a restart. The
attempt count and the time of the next attempt are kept in the run, so a restart neither resets nor
shortens them. A wait longer than 30 seconds holds no worker: the run is parked in the queue until
the attempt is due (resume_after on the run), and the first node with room resumes it then. Once the step has failed for good, on_error.steps run (they read the failure as
steps.<id>.error; an approval or a hand-off there escalates to a person), then the workflow goes
on (then: continue, the default; the step's output is the error steps' output) or fails
(then: fail). Without on_error, the failure ends the run. The builder edits the timeout, the
attempts, the first wait and whether the workflow goes on; error steps and retry_on are edited in
the JSON view.
Which models. Only agent steps use a model: the one their agent pins, else the server's
default chat model. An agent that calls tools needs a model with the tools capability (marked in
the catalog), and an agent with an output schema gets its answer constrained to
that schema on any chat model. tool, condition, parallel, loop, approval and handoff
steps use no model: the step names the tool and its arguments are templates.
2Draft, publish, roll back#
Open Admin > Automate > Workflows. New workflow and Edit open the builder: the
workflow drawn top to bottom, a + between steps to add any of the seven step types, condition
and parallel branches side by side, a loop's steps inside its box. Selecting a step opens its
fields, with Insert buttons for input and the answers of earlier steps; selecting nothing
opens the workflow's name, description and output. The JSON view shows the same definition as
the SDK writes it, and edits there show on the canvas: there is no second model, so a workflow built
on the canvas and the same workflow written by hand are stored as the same text.
New workflow asks for a name and a starting point (an answer a person approves, classify and route, extract then review, document intake, draft until it passes, or blank). A starter fills its agent and tool slots from what this server has; a slot nothing fills stays empty on its step.
The builder checks the workflow as you edit, with the same validator as the API, and shows each
problem on its step and under its field. The check beside Publish lists them all, each with a
link to its field. A draft is kept unfinished: an agent or tool not chosen yet, an empty branch or an
empty message only holds back publishing and test runs, while a workflow written wrong (an unknown
step type, an unreadable expression, a reference to a step that does not exist) cannot be saved.
Save draft keeps it serving nobody; Publish shows what changes against the live revision,
the checks, and the schedules, triggers and watch folders that start runs on it, then records the
next immutable revision. Its Impact reads GET /lmkit/v1/admin/workflows/{name}/publish-preview:
the dependencies that change, the tools that write which the draft calls and the live revision never
did (directly or through an agent), whether the input or output schema changes, the published
workflows that call this one (each keeps the revision it pins until it is published again) and the
runs under way. A draft that gives the workflow a new way to write is published only once confirmed:
/publish answers 409 with confirm_effects: true and the impact until the body sets
"confirm_effects": true. History lists every revision, and Serve again moves new runs back to one;
runs already under way keep the revision they started on.
Test runs the saved draft, published or not, on a sample input: each step shows done, failed or waiting and its time on the canvas, selecting a step reads its answer, an approval the run waits for is decided in place, and the run's timeline reads beside it. The run is an ordinary run (listed under Runs, stopping in the Approvals inbox) that carries the definition it started on, so editing the draft while it waits for a person never changes what it resumes.
The same lifecycle is under /lmkit/v1/admin/workflows/{name}/draft, /publish, /rollback and
/lifecycle, with If-Match on every write, exactly as for agents; POST .../test with
"draft": true starts a test run of the saved draft. A test run is a dry run unless the body
sets "live": true: tools that only read run for real, every tool that writes, sends or cannot be
undone (agent steps included) answers with a simulation, each call still stops for approval where
the live workflow would, no outbound event is sent, and the run reads dry_run: true. A live test
performs real writes, and its start is recorded in the activity log. POST /lmkit/v1/admin/workflows/validate
answers problems, one per fault with its step, field, kind (invalid, incomplete,
missing_agent, missing_tool, approval_in_parallel, unreachable, argument_mismatch, missing_workflow, workflow_cycle, unknown_field, tool_not_granted, unpinned_revision) and what it blocks (save, publish, or nothing for advice).
A read of a field the input's or a step's answer schema does not declare, where it allows no other,
is always empty (unknown_field). A tool the API key of a trigger, schedule or watched folder that
starts the workflow is not granted would fail every run it starts (tool_not_granted), and saving
such a starter is refused. An agent step without a revision is advice (unpinned_revision): it
blocks nothing, but a run resumed after the agent is published again plays the new revision.
A step may read only a step that can have run before it: a read of a later step, of a step in
another branch of the same condition or in a sibling parallel branch, or of the step itself, and a
step placed after a hand-off, are unreachable and block publishing and test runs. Steps after a
condition may read any of its branches (a skipped branch reads empty, test it with empty()), and
a loop's steps may read each other, since a later pass sees the previous pass's outputs.
A parallel branch cannot stop for a person, so a tool step inside one whose tool asks for approval
under the server's tools policy (approval_in_parallel) blocks publishing and test runs; should the
policy change after publishing, that step fails before its call is sent.
What a revision runs against. Publishing records the revision's manifest: the revision of every
agent and child workflow its steps name (the one a step pins, else the live one at that moment) and
a fingerprint of every custom tool it calls (method, address, parameters, header names; never a
credential nor the description). Its runs play those, so publishing an agent or a child workflow
again never changes a workflow already published or a run under way; publish the workflow again to
take the new ones. A tool changed since fails its step as tool_changed, naming what to do, and
deleting an agent, a child workflow or a custom tool a published workflow runs on is refused,
naming the workflows. A draft carries no manifest: it resolves its dependencies when published.
Moving a workflow between servers. GET /lmkit/v1/admin/workflows/{name}/export (optionally
?revision=) answers a bundle: the revision, its manifest, the agents and child workflows at the
revisions the manifest records, and its custom tools without their secret values. On the target,
POST /lmkit/v1/admin/workflows/import with {"bundle": ..., "apply": false} compares each
dependency (same, missing, differs, needs_secret); "apply": true publishes the workflow,
refused (409, with the plan) while a dependency is not the same, unless "install_dependencies": true
publishes the bundle's agents and child workflows and creates its secret-free tools. A tool whose
secrets the target lacks always refuses until it is created there. The Workflows page carries Export
on each published workflow and Import, which shows the plan before publishing.
Who may touch a workflow. Every Admin operates every workflow until the workflow's access says
otherwise. PUT /lmkit/v1/admin/workflows/{name}/access (Owner tier) names its operators, the
operators (usernames, or SSO emails) who alone may save its draft, publish, roll back, delete and
test it and retry its runs, and its approvers, who alone decide its runs' approvals; an empty list
leaves the action to every Admin, and the Owner tier always may. The access is kept beside the
workflow, never in its revisions: publishing, rolling back or deleting the workflow leaves it as it
is, so a workflow recreated under the same name stays restricted. GET .../access reads it with
whether the caller may_operate and may_approve. Refusals answer 403 with the reason. The same
access sets required_approvals (four eyes at 2) and escalate_after_minutes for its runs'
approvals (see the runs guide, Approvals).
Where its tools may reach, and with which credentials. The access's allowed_hosts narrows
the server's egress policy for the workflow's runs: a tool call to a host it does not name (or one
the server itself refuses) is refused before any request; child workflows inherit the narrowing.
A custom tool's secret headers can carry test values (test_secrets on the custom tool): test
runs send them instead of the production ones, and a tool with a secret header but no test value
refuses in a test before any request, so a test never uses a production credential. Credentials
are read when a tool is built, so a rotation reaches the next call of a run already under way.
Known secret values (custom tool and connector credentials, test values, service keys) are masked
as [redacted] in run checkpoints, run events and the activity log, whatever a tool echoes back.
A test does no more than production. A test run plays under the tool grants of the API key the
workflow runs under in production: the access's test_key, or else the one key its triggers,
schedules and watched folders share (the server's tools policy only when none names a key). A tool
that key is not granted refuses the test with a 400, as the production run would fail, and a revoked
or expired key stops the next test. The key lends its grants alone: the run takes neither its owner
nor its quota, and reads grants_key in its metadata.
3Review only the exceptions#
An approval step becomes a document review when it names the document to check. Give it an
approve_when condition and only the uncertain cases wait for a person: a batch of 500 scans runs
unattended and parks the few that need a look.
{ "id": "check", "type": "approval", "message": "Does {{vars.file_name}} read correctly?",
"document": "{{vars.file_path}}", "approve_when": "steps.ocr.output == 'confident'" }
The Approvals inbox shows a review with its document open in the viewer beside Approve and Deny; the decision and who made it are recorded in the run's timeline. A review nobody decides within the approval timeout is denied, never approved by default. The document is a stored file id or a path inside the directories open to the server (the file ingest roots); anything else is not shown.
4Run it#
curl -X POST https://your-server/lmkit/v1/workflows/support-triage/runs \
-H "Authorization: Bearer $LMKIT_KEY" -H "Content-Type: application/json" \
-d '{"input":"I was charged twice, please refund one.","variables":{"tier":"gold"},"background":true}'
The answer is a run (kind: "workflow", workflow_revision, and steps: each step's latest
output). GET /lmkit/v1/workflows lists what callers may start. Each agent step plays exactly as a
chat request with that agent would, under the calling key's tool grants, knowledge and memory; each
tool step resolves under the same grants.
| The run stops | What continues it |
|---|---|
An approval step, or a call that needs approval (in a tool step or inside an agent step) |
A decision through POST runs/{id}/input or the Approvals inbox |
| An agent step asks the user a question | POST runs/{id}/input with text |
A handoff step, or an agent hands off |
Nothing: the run is an escalation in the Escalations inbox |
Which calls ask first is the tools policy and each agent's own rules (see
Durable Agent Runs); a tool that writes always asks. A step inside a parallel
step cannot stop the workflow: a call there that needs approval is refused instead.
5Follow it#
A workflow run is listed on Admin > Operate > Runs with every other run; its timeline shows each
step with its outcome, the calls each agent made, approvals asked and decided, and what it spent.
Every play of the run is one agent_run.play span, so traces reach your OTLP backend as for agents.