LM-Kit OneDocs2026.11.1lm-kit.comEULA

Building and Testing Agents

An agent that only lives in a demo is a prompt; an agent in production is defined once, consumed by name from every client, and tested against evidence. This chapter is that lifecycle on this server (the cookbook runs it concretely in A Support Assistant with Memory): build the bundle, wire its tools, skills, and memory, consume it over the API, and verify what it actually does.


1Build: the definition#

Agents are defined in the console's Agents section (persisted configuration, so they survive restarts and ship in a settings file). The complete field set:

Field Role
name The identity clients adopt; unique.
description One picker line: what this agent is for.
system The agent's standing system prompt; a request's own system text appends after it.
skill The skill pinned by default.
tools Server-executed tools it brings: built-in names and connector-qualified names.
memory, memory_scope, memory_store Whether it remembers across conversations, whose memory that is, always inside one API key owner (caller, the default: each end user named by the request's user, else each owner; agent: one memory for every end user of the owner), and under which policy store.
model A catalog id pin; empty follows the request or the server default.
reasoning Thinking effort for reasoning models (none, low, medium, high); empty follows the model's default.
greeting A fixed first assistant message fresh conversations open with; pure presentation, no inference.
max_tool_calls The turn's tool budget, bounding the loop.
knowledge Search collections the agent answers from (sources: tenant/collection, a unique collection name, or a collection id), mode (always: every question is searched first; on_demand: the agent searches through a tool), citation_policy (required or optional), when_no_evidence (abstain or answer), max_results.

One rule makes the whole design predictable: the bundle supplies defaults, and anything the request states explicitly wins. Design agents accordingly: put the durable identity in the bundle, leave per-call variation to callers, and never rely on the bundle to FORBID anything (that is what the tools policy and skill allowlists are for).

Agents grounded in your collections. An agent with knowledge answers from the Search collections it names, and every request reaches only the collections ITS OWN API key is granted (Access Model): an agent naming a collection a caller may not read simply finds nothing there for that caller, never a title or a passage. Passages are numbered by the server, the model cites them as [1], and the answer comes back with a grounding object (on /lmkit/v1/chat and as an extension on non-streamed chat completions): the verdict (grounded, uncited, invalid_citation, no_evidence, not_retrieved), the passages presented with their document, page and collection, and the numbers cited. A citation is checked against the passages really presented; that proves where a claim points, not that the passage supports it.

The same agent in your own application. A definition is LM-Kit.NET's AgentDefinition, resolved by the same rule there: load an exported definition with AgentDefinition.FromJson and build it with AgentBuilder.WithDefinition, binding its tool and skill names to your own, and it behaves as it does here. Definitions carry a schema_version; a server refuses one written by a newer release instead of misreading it.

The workspace: build, test, evaluate, publish. A new agent starts blank, from a template, or from a blueprint set up step by step with its starter evaluation. Every agent opens in its workspace in Console › Agents, its definition on the left and a test bench on the right: the playground, embedded and locked on this agent, running exactly what you are editing. Every edit is saved to the agent's draft as you type, and a draft serves nobody: callers keep the published revision. The bench runs the draft with every tool that writes simulated (read-only tools run for real) and memory recalled but never taught, so testing never sends, writes or changes anything; each turn shows its tool calls, its evidence and its timings. Publish turns the draft into the next immutable revision, with a note; requests already running finish on the revision they started with. History lists every revision and makes any of them live again (a rollback), never rewriting one. The playground itself only tests: it runs any published agent, and an operator edits an agent in its workspace, never in the playground.

Every change that reaches a live agent, whatever made it (the workspace, the configuration API, an import), is an immutable numbered revision recorded before it goes live, with who made it and when; an agent that predates revisions is recorded as it was before its first change, so that change can be rolled back too. Two operators never overwrite each other: a draft saved on a stale copy is refused with what to reload, and a draft started from a revision that is no longer live must be started again rather than revert someone else's change. The admin API carries the same operations: GET .../agents/overview, GET .../agents/{name}/lifecycle, PUT|DELETE .../agents/{name}/draft (with If-Match), POST .../agents/{name}/publish, POST .../agents/{name}/rollback, DELETE .../agents/{name}.

On a farm. Agent definitions are shared objects of the identity database: a definition saved through any node reaches every node within the sync period, and GET /lmkit/v1/agents lists the same set everywhere. The skills they pin and the memory facts they teach are domain documents too, so the probe suite of section 3 gives the same answer through the balancer as through one named node (Scaling Out). A deployment that declares its manifest the domain's source (Deployment:DomainSource=Manifest) owns the catalog there: the Agents editor is read-only and a save is refused, so ship the catalog in the manifest (Export lifts it into a JSON file).

2Consume: one field, every client#

  • Discovery: GET /lmkit/v1/agents lists the definitions with their descriptions, skills, tool names, memory intent, and greeting, so pickers build themselves.
  • Native chat: POST /lmkit/v1/chat with "agent": "support", streaming or not.
  • OpenAI dialects: the same agent field rides /v1/chat/completions and /v1/responses as an extension, so any existing OpenAI-SDK application adopts a server agent by adding one field, no other change. The same holds for skill, skill_inputs, server_tools, memory, and memory_store.
  • A pinned revision: agent_revision (a number) runs that published revision, so a later publication does not change an integration's behavior until it moves its pin; absent runs the live revision. Every response names what it ran (agent: name, revision, and whether it was a draft), and a request keeps the revision it was admitted with even when a new one is published while it runs.
  • The playground: every agent appears as a persona; picking it arms the greeting, the skill, the tools, and a per-conversation memory id when the agent wants one.

A published agent also has doors of its own, listed on the Use tab of its workspace:

Door For How
Its chat route An application POST /lmkit/v1/agents/{name}/chat with {"input": "...", "new_session": true}: the server keeps the conversation, so later requests send session_id and their next message only. A session belongs to the key owner that started it and ends after 24 hours without a message
Its model alias Any OpenAI-compatible client "model": "agent:{name}" in /v1/chat/completions or /v1/responses; GET /v1/models lists every published agent this way, so a client that can only pick a model picks the agent
Its MCP tool Assistants over MCP agent_{name} delegates a whole task and returns the answer with a session_id to continue; a published workflow is workflow_{name}, and a run that waits for a person returns its run id. Listed once turned on in MCP, under Agents and workflows
Its page People /playground/agents/{name}: one conversation with the agent, behind the Playground's access setting

Every door serves the published revision under the caller's own key: its tool grants, risk ceiling and quotas apply.

The agent's own budget. Set budget on the Use tab caps what every caller of the agent spends together, through any door: requests per minute and tokens per 24 hours, on top of each caller's own quotas (PUT /lmkit/v1/admin/quotas/agents/{name}). Past it, every door answers 429 with the time to retry, the same way a key's quota does; an answer under way is never cut off.

An unknown agent name is a named error on every path, never a silent fallback: a deploy that forgot to define the agent fails loudly on the first call.

The invocation contract

The contract is the same on the three dialects; only the envelope differs.

/lmkit/v1/chat /v1/chat/completions /v1/responses
Select the agent agent agent agent
Pin a revision agent_revision agent_revision agent_revision
End user (memory, audit) user user user
Structured answer response_format response_format text.format
What ran result.agent agent agent (also on response.completed)
Citations result.grounding grounding (not streamed) grounding (also on response.completed)

agent in a response is { "name", "revision", "draft" }. A structured answer is generated after the agent's tools ran, under the format (Structured Outputs). grounding is present when the agent has knowledge sources. Refusals happen before any model work: on the OpenAI dialects they are a 400 whose code names the problem (agent_not_found, agent_revision_not_found, invalid_agent_revision, agent_draft_forbidden, skill_not_found, server_tools_denied) and whose param names the field; native chat answers the same sentence as its error.

curl https://lmkit.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LMKIT_API_KEY" -H "Content-Type: application/json" \
  -d '{"agent":"support","agent_revision":4,"user":"customer-1842",
       "messages":[{"role":"user","content":"Can I return a used item?"}]}'

With an OpenAI SDK, the LM-Kit fields ride the SDK's extra-body option (extra_body in Python, AdditionalProperties or a raw JSON body in .NET); nothing else changes.

The same agent in process. Read a published revision as JSON (GET /lmkit/v1/admin/agents/{name}/revisions/{revision}, its definition), ship it with your application, and build it with LM-Kit.NET:

var definition = AgentDefinition.FromJson(File.ReadAllText("support.r4.agent.json"));
var agent = Agent.CreateBuilder(model)
    .WithDefinition(definition, resolveTool: name => myTools.Get(name))
    .Build();

It resolves exactly as the server resolves it, so the in-process agent and the published one answer alike.

3Test: evidence, not vibes#

Agents are software; test them like it. The workspace gives you a bench, a suite and a gate; the API gives you determinism levers and after-the-fact evidence:

  • Interactive first: the workspace's test bench runs the draft as you edit it: converse, watch tool activity as it happens, read the evidence, change the definition, and restart on it. From the bench turns that conversation into a case.
  • Evaluate: an agent's suite holds its cases, each a few user turns and mechanical checks (what the answer must and must not say, which tools must and must not run, at most how many calls, whether it must cite its evidence or abstain, which properties a structured answer carries). A citation proves only that the answer points at a passage it was shown, so claim support is labelled: Supported facts are facts you checked against the sources, which the answer must state and a passage it cites must say. Grounding verdict requires how the answer ends: retrieval_failed for a case whose source is down (the agent says the sources cannot be reached, never answers from general knowledge), no_evidence for a key that may not read the collection holding the answer, paired with what must never appear (a title, a snippet). A run plays every case against the draft or the live revision, under the API key you name (its tools and Search collections, exactly as its callers get them), with every tool that writes simulated; it reports each failed check in words and which cases it won and lost against the live revision. The suite is LM-Kit.NET's AgentEvaluationSuite: Export it and run the same file in process with AgentEvaluator.RunAsync.
  • The gate: with Require a passing run to publish, a draft publishes only when a run of exactly that version clears the gate (a least pass rate, and no case that passes on the live revision failing). An operator can publish over a failing gate by stating why; the reason is recorded with the revision and in the activity log.
  • Reproducibility: chat accepts a seed, so a scripted probe suite can pin sampling and compare runs; request_id names each exchange so your harness can correlate its calls with the server's records.
  • The loop's own record: every response summarizes tool activity as tool_events (streamed live as tool_use events), so an assertion like "the refund question must consult the policy document, never the web" is a check on structured output, not a guess.
  • Memory checks: after a probe conversation, read the facts the agent stored (admin memory inspection) and assert what SHOULD have been remembered was, and nothing else; delete the probe id afterward.
  • Server-side evidence: the requests and logs pages hold every exchange with timing, models, and outcomes, which is where "it behaved differently yesterday" gets answered.

A practical regression suite is therefore plain HTTP: a fixed set of conversations (seeded), assertions over answers and tool_events, memory-state checks where memory matters, run against a staging server on every skill or definition change. No special framework is required, because everything the agent does is visible in the API's own responses.

4Harden: the governance seams#

Production agents run inside the operator's decisions, each covered in its own guide: tool policy and egress decide what the loop may touch, skill allowlists and sources decide what procedure may load, memory stores decide what persists and the admin surface makes it correctable, and API keys decide who may converse at all. The agent definition itself deliberately holds no secrets and no permissions: it is a convenience bundle, and the security model stays where security lives.

5Iterate: the improvement loop#

Treat the definition, its skills, and its taxonomies as one reviewable unit: version them, change one lever at a time, and re-run the probe suite. When the agent's model is the bottleneck at volume, the models guide covers per-capability defaults and measurement, and fine-tuning turns your corrected transcripts into a compact model that speaks your domain natively. The corpus your agents generate under real traffic, with tool events and outcomes attached, is exactly the evidence that loop needs.

6Supported models#

Knowledge-bound agents ask more of a model than plain chat. In the same turn, it must decide whether the passages answer the question, cite them, follow the blueprint's structured reply and call tools only when the case needs one. Two reference models cover the range. Both were tested end to end through every agent feature: knowledge with citations, structured answers after tools, the three API dialects, evaluation runs and the publish gate.

Reference Model Fit
Mid-size (recommended) qwen3.5:9b Every agent feature, served and in process. The default choice for customer-facing agents.
Small qwen3.5:4b Every feature through the server. Run in process with reasoning off, it sometimes routes a plain question to a hand-off instead of answering from the passages. Keep reasoning on, or use it for internal agents behind an evaluation gate.

Both were tested with embeddinggemma-300m as the embedding model of the Search tenant, with the server's default slot pool (four slots). Other models in the catalog with tool calling run every feature too. Before you publish an agent on one of them, run its evaluation suite on that model: the suite measures what this table states, on your own cases.

7Upgrading agents made before revisions#

Agents defined on an earlier release keep serving unchanged after the upgrade. Five things are worth knowing before you change one:

What changed What it means for an existing agent
Revisions An existing agent serves as it is. Its first change records the version it had as revision 1 and publishes the change as revision 2, so you can roll back to what it was before the upgrade.
The definition format Agent definitions are LM-Kit.NET's AgentDefinition, and every JSON name is snake_case (max_tool_calls, memory_scope). Stored agents are read and rewritten automatically. An agent file exported by an earlier release, in camelCase, is refused on import with the name it did not recognize: rename its fields, or export it again from the upgraded server.
Memory Memory is now kept per caller: per end user (user) under each API key owner, or shared by one owner's end users with memory_scope: agent. Memories written before keep their plain id and are no longer read through any API key. Copy to owner in the Memory section files a copy under the owner that should keep it (legacy memories).
Knowledge Agents can bind Search collections (knowledge). Nothing is bound for you: an existing agent answers as before until you add a collection, and then searches it under each caller's own grants.
Evaluation gate A publish can require a passing evaluation run of the exact draft. The gate is off until you turn it on in the agent's Evaluate tab, so existing publishing flows are unchanged.

Clients pinning nothing keep calling the agent by name and get the live revision. A client that must not move when you publish names the revision it was tested against with agent_revision.

8Stated plainly#

  • Build the durable identity into the bundle; leave variation to requests; enforce nothing through the bundle, everything through policy.
  • Consumption is one field on any dialect, with API discovery and named refusals, so clients stay thin and deploys fail loudly.
  • Testing is ordinary HTTP against determinism levers and structured evidence (seed, request_id, tool_events, memory inspection, request logs): agents are observable software here, not black boxes.