Table of Contents

How Do I Monitor and Debug AI Agent Execution?


TL;DR

LM-Kit.NET provides three layers of observability: Events on conversations and agents (BeforeToolInvocation, AfterToolInvocation, AfterTextCompletion), Middleware Filters (prompt filters, completion filters, tool invocation filters) for structured interception, and OpenTelemetry integration for production tracing and metrics. Together, these let you log every tool call, measure latency, validate output quality, and debug agent behavior.


Layer 1: Events

Subscribe to events on MultiTurnConversation or Agent for direct visibility:

var chat = new MultiTurnConversation(model);

// Log every tool call
chat.BeforeToolInvocation += (sender, args) =>
{
    Console.WriteLine($"[TOOL] Calling: {args.ToolCall.Name}");
    Console.WriteLine($"  Args: {args.ToolCall.Arguments}");
};

chat.AfterToolInvocation += (sender, args) =>
{
    Console.WriteLine($"[TOOL] Result: {args.Result}");
    Console.WriteLine($"  Duration: {args.Duration.TotalMilliseconds:F0}ms");
};

// Stream generated tokens in real time
chat.AfterTextCompletion += (sender, args) =>
{
    Console.Write(args.Text);
};

// Human-in-the-loop approval
chat.ToolApprovalRequired += (sender, args) =>
{
    Console.WriteLine($"[APPROVAL] {args.ToolCall.Name} requires approval");
    args.Approved = true;  // or prompt the user
};

Layer 2: Middleware Filters

Filters follow an onion (middleware) pattern. Code before await next() runs pre-processing; code after runs post-processing.

Prompt Filters

Intercept and modify prompts before inference:

using LMKit.TextGeneration.Filters;

chat.Filters.AddPromptFilter(async (context, next) =>
{
    // Pre-processing: log or modify the prompt
    Console.WriteLine($"[PROMPT] {context.Prompt}");

    await next(context);  // Continue to model inference

    // Post-processing: prompt was sent
});

Use cases: Prompt injection detection, input sanitization, template expansion, logging.

Completion Filters

Validate or transform completions after inference:

chat.Filters.AddCompletionFilter(async (context, next) =>
{
    await next(context);  // Wait for model to generate

    // Post-processing: inspect or modify the result
    var result = context.Result;
    Console.WriteLine($"[COMPLETION] Tokens: {result.TokenCount}, Stop: {result.StopReason}");

    // Quality gate: reject low-quality output
    if (result.TokenCount < 10)
        Console.WriteLine("[WARN] Very short response");
});

Use cases: Quality validation, output sanitization, telemetry, response caching, guardrails.

Tool Invocation Filters

Intercept tool calls with full control over execution:

chat.Filters.AddToolInvocationFilter(async (context, next) =>
{
    var start = Stopwatch.GetTimestamp();

    Console.WriteLine($"[TOOL] {context.ToolName}({context.Arguments})");

    await next(context);  // Execute the tool

    var elapsed = Stopwatch.GetElapsedTime(start);
    Console.WriteLine($"[TOOL] {context.ToolName} completed in {elapsed.TotalMilliseconds:F0}ms");

    // Optional: override result, cancel execution, or terminate tool loop
    // context.Cancel = true;      // Skip execution
    // context.Result = "cached";  // Override result
    // context.Terminate = true;   // Stop further tool calls
});

Use cases: Logging, rate limiting, result caching, error recovery, early loop termination.


Layer 3: OpenTelemetry Integration

LM-Kit.NET integrates with OpenTelemetry for production-grade distributed tracing:

using LMKit.Global;

// Enable telemetry
Runtime.EnableTelemetry = true;

// Activities and metrics are exported to your configured OpenTelemetry collector
// Traces include: model loading, inference, tool calls, token counts

This feeds into standard observability stacks (Jaeger, Zipkin, Prometheus, Grafana) for dashboards, alerting, and performance analysis.


Common Debugging Patterns

Log All Agent Activity

// Combine events and filters for full visibility
chat.BeforeToolInvocation += (s, e) => Log($"TOOL_START: {e.ToolCall.Name}");
chat.AfterToolInvocation += (s, e) => Log($"TOOL_END: {e.ToolCall.Name} ({e.Duration.TotalMilliseconds}ms)");

chat.Filters.AddCompletionFilter(async (ctx, next) =>
{
    await next(ctx);
    Log($"COMPLETION: {ctx.Result.TokenCount} tokens, stop={ctx.Result.StopReason}");
});

Measure Latency Per Component

chat.Filters.AddPromptFilter(async (ctx, next) =>
{
    var sw = Stopwatch.StartNew();
    await next(ctx);
    sw.Stop();
    Metrics.RecordInferenceLatency(sw.ElapsedMilliseconds);
});

📚 Related Content

Share