Agent Observability and Debugging

Log model calls and tool calls, inspect decisions, trace failures, and measure token and cost usage.

When an agent is wrong, you need a trace: what it thought, what it called, what came back, how many tokens that cost. Print-debugging one print(message) will not survive a second user.

A JSONL trace

import json
import time
from pathlib import Path

LOG = Path("agent.jsonl")


def trace(event: str, **fields) -> None:
    row = {"ts": time.time(), "event": event, **fields}
    with LOG.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")


def think(messages, tools=None):
    response = client.chat.completions.create(
        model=MODEL, messages=messages, tools=tools or None,
    )
    usage = response.usage
    trace(
        "model",
        model=MODEL,
        prompt_tokens=getattr(usage, "prompt_tokens", None),
        completion_tokens=getattr(usage, "completion_tokens", None),
    )
    message = response.choices[0].message
    names = [c.function.name for c in (message.tool_calls or [])]
    trace("decision", tools=names, has_text=bool(message.content))
    return message

Log tool name, argument keys (not values if they might be secrets), ok, and duration. Log step index. Do not log raw API keys or file bodies.

Cost

Keep running totals of prompt and completion tokens. Multiply by the published price for your host, or treat tokens as the budget unit. Stop the loop if prompt_tokens this session exceeds a cap — same fuse as max_steps.

Check it

Run one successful scout. agent.jsonl should have a model line, a decision with a tool name, and a tool result line.
Fail a tool on purpose. The trace should show ok: false and another model line after.
Sum tokens from the file and print a one-line session cost.

Next: turn traces into a score you can rerun.