Table of Contents

Monitor Agent Execution with Tracing and Custom Metrics

When an agent calls tools, plans steps, and generates responses, understanding what happened inside the execution is critical for debugging, optimization, and production monitoring. LM-Kit.NET provides a built-in observability stack with span-based tracing, structured metrics, and pluggable exporters. This tutorial shows how to enable console tracing for development, capture spans in memory for analysis, combine multiple tracers, and export traces to JSON.


Why Agent Tracing Matters

Two real-world problems that agent tracing solves:

  1. Debugging multi-step agent failures. When an agent produces an incorrect answer after 6 tool calls and 3 planning iterations, you need a full trace to identify which step went wrong. Span-based tracing shows the exact sequence, timing, and results of each operation.
  2. Monitoring production agent performance. In production, you need to track token usage, tool latency, and error rates across thousands of agent executions. The metrics system provides counters, histograms, and gauges for dashboards and alerting.

Prerequisites

Requirement Minimum
.NET SDK 8.0+
VRAM 4+ GB (for a tool-calling model)

Step 1: Create the Project

dotnet new console -n AgentTracingDemo
cd AgentTracingDemo
dotnet add package LM-Kit.NET

Step 2: Understand the Tracing Architecture

┌────────────────────────────────────────────────┐
│               Agent Execution                  │
│                                                │
│  ┌──────────┐   ┌──────────┐  ┌─────────────┐  │
│  │  Agent   │   │  Tool    │  │  Inference  │  │
│  │  Span    │   │  Span    │  │  Span       │  │
│  └────┬─────┘   └────┬─────┘  └──────┬──────┘  │
│       │              │               │         │
│       └──────────────┴───────────────┘         │
│                      │                         │
└──────────────────────┼─────────────────────────┘
                       │
          ┌────────────┴────────────┐
          │       IAgentTracer      │
          └────────────┬────────────┘
                       │
       ┌───────────────┼───────────────┐
       ▼               ▼               ▼
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ConsoleTracer│ │InMemoryTracer│ │JsonTrace     │
│             │ │              │ │Exporter      │
└─────────────┘ └──────────────┘ └──────────────┘
Component Purpose
AgentTracing Static class to enable/configure global tracing
IAgentTracer Interface for custom tracer implementations
AgentSpan Represents a single traced operation (agent, tool, planning, inference)
ConsoleTracer Prints trace events to the console with color and timing
InMemoryTracer Stores spans in memory for programmatic analysis
CompositeTracer Fans out to multiple tracers simultaneously
JsonTraceExporter Exports spans to structured JSON files
AgentMetrics Global counters, histograms, and gauges

Step 3: Console Tracing for Development

The fastest way to see what an agent is doing. Enable it with one line:

using System.Text;
using LMKit.Agents;
using LMKit.Agents.Observability;
using LMKit.Agents.Tools.BuiltIn;
using LMKit.Model;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

Console.InputEncoding = Encoding.UTF8;
Console.OutputEncoding = Encoding.UTF8;

// ──────────────────────────────────────
// 1. Enable console tracing
// ──────────────────────────────────────
AgentTracing.EnableConsoleTracing(tracer =>
{
    tracer.MinimumLevel = LogLevel.Debug;
    tracer.UseColors = true;
    tracer.ShowTimestamps = true;
    tracer.ShowSpanIds = false;
    tracer.MaxDisplayLength = 200;
});

Console.WriteLine("Console tracing enabled.\n");

// ──────────────────────────────────────
// 2. Load a model and build an agent
// ──────────────────────────────────────
Console.WriteLine("Loading model...");
using LM model = LM.LoadFromModelID("qwen3:4b",
    loadingProgress: p => { Console.Write($"\r  Loading: {p * 100:F0}%   "); return true; });
Console.WriteLine("\n");

var agent = Agent.CreateBuilder(model)
    .WithPersona("Research Assistant")
    .WithInstruction("Help users find information. Use tools when needed.")
    .WithPlanning(PlanningStrategy.ReAct)
    .WithTools(tools =>
    {
        tools.Register(BuiltInTools.WebSearch);
        tools.Register(BuiltInTools.Calculator);
        tools.Register(BuiltInTools.DateTime);
    })
    .WithMaxIterations(10)
    .Build();

// ──────────────────────────────────────
// 3. Run the agent (trace appears automatically in console)
// ──────────────────────────────────────
Console.WriteLine("Running agent...\n");

var result = await agent.RunAsync("What is the square root of the current year?");

Console.ForegroundColor = ConsoleColor.Cyan;
Console.WriteLine($"\nFinal answer: {result.Content}");
Console.ResetColor();

// Disable tracing when done
AgentTracing.Disable();

The console output shows each span as it opens and closes, with timing, tool arguments, and results.


Step 4: In-Memory Tracing for Analysis

For programmatic analysis, use InMemoryTracer to capture spans and query them:

using LMKit.Agents;
using LMKit.Agents.Observability;
using LMKit.Agents.Tools.BuiltIn;
using LMKit.Model;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

// ──────────────────────────────────────
// 1. Set up in-memory tracing
// ──────────────────────────────────────
var memoryTracer = new InMemoryTracer { MaxSpans = 500 };
AgentTracing.SetTracer(memoryTracer);

// ──────────────────────────────────────
// 2. Load model and build agent
// ──────────────────────────────────────
using LM model = LM.LoadFromModelID("qwen3:4b",
    loadingProgress: p => { Console.Write($"\r  Loading: {p * 100:F0}%   "); return true; });
Console.WriteLine("\n");

var agent = Agent.CreateBuilder(model)
    .WithPersona("Analyst")
    .WithInstruction("Answer questions accurately.")
    .WithPlanning(PlanningStrategy.ReAct)
    .WithTools(tools =>
    {
        tools.Register(BuiltInTools.Calculator);
        tools.Register(BuiltInTools.DateTime);
    })
    .WithMaxIterations(5)
    .Build();

// ──────────────────────────────────────
// 3. Execute multiple requests
// ──────────────────────────────────────
string[] queries = {
    "What day of the week is it today?",
    "Calculate 17 * 23 + 89",
    "What is 15% of 2400?"
};

foreach (string query in queries)
{
    Console.WriteLine($"Query: {query}");
    var result = await agent.RunAsync(query);
    Console.WriteLine($"Answer: {result.Content}\n");
}

// ──────────────────────────────────────
// 4. Analyze captured traces
// ──────────────────────────────────────
Console.WriteLine("=== Trace Analysis ===\n");

// Get summary statistics
TraceSummary summary = memoryTracer.GetSummary();
Console.WriteLine($"Total spans:    {summary.TotalSpans}");
Console.WriteLine($"Unique traces:  {summary.UniqueTraces}");
Console.WriteLine($"Total duration: {summary.TotalDurationMs:F0} ms");
Console.WriteLine($"Errors:         {summary.ErrorCount}");

Console.WriteLine("\nSpans by kind:");
foreach (var kvp in summary.SpansByKind)
    Console.WriteLine($"  {kvp.Key}: {kvp.Value}");

Console.WriteLine("\nSpans by status:");
foreach (var kvp in summary.SpansByStatus)
    Console.WriteLine($"  {kvp.Key}: {kvp.Value}");

// Query specific span types
Console.WriteLine("\n--- Tool Spans ---");
var toolSpans = memoryTracer.GetSpansByKind(SpanKind.Tool);
foreach (AgentSpan span in toolSpans)
{
    Console.WriteLine($"  Tool: {span.OperationName}");
    Console.WriteLine($"    Duration: {span.Duration?.TotalMilliseconds:F0} ms");
    Console.WriteLine($"    Status:   {span.Status}");
}

// Filter by custom predicate
var slowSpans = memoryTracer.GetSpans(s => s.Duration > TimeSpan.FromSeconds(2));
Console.WriteLine($"\nSlow spans (>2s): {slowSpans.Count}");

// Get error logs
var errors = memoryTracer.GetLogs(LogLevel.Error);
if (errors.Count > 0)
{
    Console.WriteLine("\nErrors:");
    foreach (LogEntry error in errors)
        Console.WriteLine($"  [{error.Timestamp:HH:mm:ss}] {error.Message}");
}

// Clean up
memoryTracer.Clear();
AgentTracing.Reset();

Step 5: Composite Tracing (Multiple Destinations)

Use CompositeTracer to send traces to multiple destinations simultaneously:

using LMKit.Agents.Observability;

// Console for development + in-memory for analysis
var consoleTracer = new ConsoleTracer
{
    MinimumLevel = LogLevel.Information,
    UseColors = true
};

var memoryTracer = new InMemoryTracer();

var composite = new CompositeTracer(consoleTracer, memoryTracer);

// Or use fluent Add syntax
var composite2 = new CompositeTracer()
    .Add(new ConsoleTracer())
    .Add(new InMemoryTracer());

AgentTracing.SetTracer(composite);

// All subsequent agent executions trace to both destinations

Step 6: Export Traces to JSON

For persistence and external analysis, export spans to JSON:

using LMKit.Agents.Observability;

// ──────────────────────────────────────
// Option A: Export from InMemoryTracer to JSON file
// ──────────────────────────────────────
var memoryTracer = new InMemoryTracer();
AgentTracing.SetTracer(memoryTracer);

// ... run agent ...

using var exporter = new JsonTraceExporter("traces.json");
foreach (AgentSpan span in memoryTracer.GetSpans())
{
    exporter.Export(span);
}
exporter.Flush();

Console.WriteLine($"Exported {memoryTracer.SpanCount} spans to traces.json");

// ──────────────────────────────────────
// Option B: Get JSON as a string
// ──────────────────────────────────────
using var stringExporter = new JsonTraceExporter(new StringWriter());
foreach (AgentSpan span in memoryTracer.GetSpans())
{
    stringExporter.Export(span);
}
string json = stringExporter.GetJson();
Console.WriteLine(json);

// ──────────────────────────────────────
// Option C: Auto-flush mode (writes each span immediately)
// ──────────────────────────────────────
using var autoExporter = new JsonTraceExporter("live-traces.json")
{
    AutoFlush = true
};
// Each Export() call writes immediately to disk

Step 7: Using Global Metrics

AgentMetrics provides global counters and histograms for production monitoring:

using LMKit.Agents.Observability;

// Access the global metrics singleton
AgentMetrics metrics = AgentMetrics.Global;

// After running multiple agent executions, inspect metrics
Dictionary<string, long> counters = metrics.GetCounters();
Dictionary<string, HistogramStats> histograms = metrics.GetHistograms();
Dictionary<string, double> gauges = metrics.GetGauges();

Console.WriteLine("=== Agent Metrics ===\n");

Console.WriteLine("Counters:");
foreach (var kvp in counters)
    Console.WriteLine($"  {kvp.Key}: {kvp.Value}");

Console.WriteLine("\nHistograms:");
foreach (var kvp in histograms)
{
    HistogramStats stats = kvp.Value;
    Console.WriteLine($"  {kvp.Key}:");
    Console.WriteLine($"    Count: {stats.Count}, Avg: {stats.Average:F1} ms");
    Console.WriteLine($"    Min: {stats.Min:F1}, Max: {stats.Max:F1}, P99: {stats.P99:F1}");
}

Console.WriteLine("\nGauges:");
foreach (var kvp in gauges)
    Console.WriteLine($"  {kvp.Key}: {kvp.Value:F2}");

// Reset when starting a new monitoring window
metrics.Reset();

Working with AgentSpan Attributes

Spans carry structured attributes that describe the operation:

using LMKit.Agents.Observability;

var memoryTracer = new InMemoryTracer();
AgentTracing.SetTracer(memoryTracer);

// ... run agent ...

foreach (AgentSpan span in memoryTracer.GetSpans())
{
    Console.WriteLine($"Span: {span.OperationName} ({span.Kind})");
    Console.WriteLine($"  ID:       {span.SpanId}");
    Console.WriteLine($"  Trace:    {span.TraceId}");
    Console.WriteLine($"  Parent:   {span.ParentSpanId ?? "none"}");
    Console.WriteLine($"  Start:    {span.StartTime:HH:mm:ss.fff}");
    Console.WriteLine($"  Duration: {span.Duration?.TotalMilliseconds:F0} ms");
    Console.WriteLine($"  Status:   {span.Status} {span.StatusMessage ?? ""}");

    // Custom attributes set during execution
    foreach (var attr in span.Attributes)
        Console.WriteLine($"  [{attr.Key}] = {attr.Value}");

    // Events recorded during the span
    foreach (SpanEvent evt in span.Events)
        Console.WriteLine($"  Event: {evt.Name} at {evt.Timestamp:HH:mm:ss.fff}");

    Console.WriteLine();
}

Integrating with External Telemetry

For production monitoring with external systems (e.g., OpenTelemetry, Jaeger, Datadog), use the CompositeTracer + InMemoryTracer pattern to intercept completed spans and forward them:

using LMKit.Agents.Observability;

// Use InMemoryTracer to capture spans, then periodically export them
var memoryTracer = new InMemoryTracer { MaxSpans = 2000 };
var consoleTracer = new ConsoleTracer { MinimumLevel = LogLevel.Warning };

AgentTracing.SetTracer(new CompositeTracer(consoleTracer, memoryTracer));

// After agent executions, forward spans to your telemetry backend
async Task FlushToTelemetryAsync()
{
    var spans = memoryTracer.GetSpans();

    foreach (AgentSpan span in spans)
    {
        // Map to your telemetry system's format
        // Example: OpenTelemetry, Datadog, or custom API
        Console.WriteLine($"Export: {span.OperationName} ({span.Kind}) " +
            $"duration={span.Duration?.TotalMilliseconds:F0}ms " +
            $"status={span.Status}");
    }

    // Also export metrics
    var metrics = AgentMetrics.Global;
    foreach (var kvp in metrics.GetHistograms())
    {
        Console.WriteLine($"Histogram: {kvp.Key} avg={kvp.Value.Average:F1}ms p99={kvp.Value.P99:F1}ms");
    }

    memoryTracer.Clear();
}

For file-based export, use JsonTraceExporter with a composite:

// Log warnings to console + capture everything for JSON export
var memoryTracer = new InMemoryTracer();

AgentTracing.SetTracer(new CompositeTracer()
    .Add(new ConsoleTracer { MinimumLevel = LogLevel.Warning })
    .Add(memoryTracer));

// After agent executions, write to disk
using var exporter = new JsonTraceExporter("agent-traces.json");
foreach (var span in memoryTracer.GetSpans())
    exporter.Export(span);
exporter.Flush();

Common Issues

Problem Cause Fix
No trace output Tracing not enabled Call AgentTracing.EnableConsoleTracing() or SetTracer() before running the agent
Missing tool spans IsEnabled set to false Check AgentTracing.IsEnabled = true
InMemoryTracer losing old spans MaxSpans limit reached Increase MaxSpans or export spans periodically
JSON file is empty Forgot to flush Call exporter.Flush() or set AutoFlush = true

Next Steps