Observability for AI Agents: Tracing, Metrics, Logs and Evals

AI agent observability makes the inner workings of an autonomous agent visible: through tracing (spans across reasoning and tool calls), metrics (latency, tokens, cost, success rate), structured logs and continuous evals. It answers why an agent made a particular decision, and it is the prerequisite for being able to debug, secure and audit multi-step agents in production at all.

Unlike a classic web service, an agent does not make a single request-response decision but instead goes through several rounds of reasoning per task with intermediate tool calls. It is precisely this multi-step nature that makes conventional monitoring blind: an HTTP 200 says nothing about whether the right tool was called with the right arguments or whether the model took a wrong turn halfway through.

The three key points up front:

• Observability for agents consists of four pillars: tracing, metrics, logs and evals. They interlock: the trace provides the path, metrics provide the aggregation, logs provide the context, evals provide the quality verdict.
• Without tracing, agents are un-debuggable. Only a hierarchical trace across all reasoning and tool-call spans shows at which step a multi-step chain broke.
• Data residency is a mandatory criterion. Since traces contain prompts and responses, and therefore potentially customer data, the observability backend is itself a compliance object for DACH B2B (Langfuse self-hosted in the EU, Datadog EU, Honeycomb EU).

Why agents are un-debuggable without tracing

An agent is non-deterministic. The same request can lead to different tool calls, different numbers of reasoning rounds and different results. If a run fails or produces a substantively incorrect answer, there is no reliable way to isolate the cause without tracing. Possible sources of error within a single request:

• The LLM reasoned incorrectly and chose an unsuitable tool.
• The tool was called with faulty arguments.
• The retrieval step (RAG) pulled the wrong documents.
• A guardrail or PII filter intervened and altered the context.
• A model version changed server-side (managed APIs update models on the provider's schedule).

A flat output log does not distinguish between these cases. Tracing makes each of these steps visible as its own span, ordered temporally and causally. This shifts debugging from "guessing based on the final result" to "finding the specific span that broke". The same applies architecturally: service meshes provide observability at the network level, and agent stacks made up of many microservices (orchestrator, tool server, memory store, vector DB, retrieval service, guardrail service) can only be managed with an end-to-end observability layer.

The four pillars of agent observability

Tracing: spans across reasoning and tool calls

A trace represents a complete agent request as a tree. The root span encompasses the entire request; nested below it hang spans for each LLM call, each tool call, each retrieval step and each guardrail check. Every span carries inputs and outputs as well as attributes such as model name, prompt and completion tokens, latency and, derived from these, cost. This hierarchical structure is the decisive difference from the classic, flat request log.

Metrics: latency, tokens, cost, success rate

Metrics aggregate across many traces. Four signals form the core:

• Latency, end-to-end and per tool-call round. Between the Azure regions Germany West Central (Frankfurt) and East US, Microsoft measures a median round trip of 94 ms (Azure latency statistics, as of July 2026). An agent in Frankfurt calling an API in US East pays this time on every call. With several tool rounds, it adds up.
• Tokens (prompt and completion) per span as the basis for cost attribution.
• Cost per request, per team and per use case.
• Success rate: the share of correctly completed tasks.

Logs: structured context

Structured logs supplement spans with detailed context: raw tool responses, retry attempts, guardrail triggers, truncated contexts. The key is correlation: every log entry should be attributable to a specific span via the trace ID, otherwise a blind spot reappears.

Evals: quality as part of observability

Evals are the signal that pure tracing does not provide: the systematic assessment of output quality. What is typically assessed is success rate, tool-call correctness (was the right tool called with the right arguments?) and response quality, via heuristics, a reference dataset or "LLM-as-a-judge". Decisive for regulated contexts: evals and prompt versions are tagged against pinned model versions in the gateway or the observability platform. This ties every quality verdict to a specific model and prompt version. If an agent is classified as a high-risk system under the AI Act, this traceability makes the required evidence easier to provide, for example on accuracy and robustness under Art. 15 and on the validation and testing procedures that the technical documentation under Art. 11 and Annex IV must contain.

Signal-to-tool matrix

The following table maps each observability signal to its measured quantity and a typical tool (as of 2026).

The LLM observability stack: tools at a glance

For the observability layer, these tools are the main candidates (as of 2026):

• LangSmith: the tracing and eval platform from the LangChain/LangGraph ecosystem, as a managed service. Strong when the agent is already built on this framework. As with any managed service, check up front where the traces are stored.
• Langfuse: open source and, according to its self-hosting docs, deployable on your own infrastructure, so self-hostable in the EU (your own data center or an EU region). That fits regulated architectures with customer-controlled observability as well as lean cloud setups that keep traces in the EU for GDPR compliance.
• Arize Phoenix: observability and evaluation under the Elastic License 2.0, free to self-host and therefore an alternative to Langfuse.
• OpenTelemetry for LLMs / OpenInference: the vendor-neutral trace standard. Instrumented via the GenAI semantic conventions, the backend remains interchangeable, an important argument against lock-in. Hugging Face's TGI, for example, already exported OpenTelemetry traces and Prometheus metrics before the project went into maintenance mode.
• Datadog EU / Honeycomb EU: established APM backends with an EU region. Datadog runs its EU1 site in Germany, and Honeycomb stores data in the US or EU region you sign up in. Both are the managed route for keeping data in the EU.

An important architectural point: the AI gateway (LiteLLM, Portkey, Kong) is often already a carrier of parts of the observability. Gateways typically handle multi-provider failover, virtual key management, team budgets, observability, guardrails and PII redaction. In practice, the gateway attributes tokens and cost, while the tracing platform holds the reasoning path and the evals: the two together produce the complete picture.

Example trace: an agent run, broken down

The following pseudo-example shows how a single support-agent run looks as a span tree. The numbers are illustrative.

```
TRACE id=ag-7f3c "Customer request: cancel invoice" total: 4,210 ms | 3,320 tokens | 0.041 USD
├─ SPAN llm.reason model=gpt-4.x 620 ms | in 540 / out 80 tok "Plan: look up customer+invoice"
├─ SPAN tool.crm_lookup mTLS, in-VPC 180 ms | status=200 args={customer_id:8842}
├─ SPAN retrieval.vector qdrant-eu 95 ms | 4 hits query="B2B cancellation policy"

├─ SPAN llm.reason model=gpt-4.x 710 ms | in 1,980 / out 130 tok "Cancellation permitted, call tool"
├─ SPAN tool.invoice_void in-VPC 240 ms | status=200 args={invoice:RE-2026-0317}
├─ SPAN guardrail.pii redaction 40 ms | 0 hits
└─ SPAN llm.compose model=gpt-4.x 2,325 ms| in 380 / out 210 tok response text to customer
EVAL tool_call_correct=PASS | answer_quality=0.92 | tagged: model=gpt-4.x, prompt=v7
```

What this trace achieves that an output log cannot: had the cancellation failed, the tree immediately shows whether tool.invoice_void returned an error status, whether retrieval.vector pulled the wrong policy or whether the second llm.reason span decided incorrectly. The largest latency item (llm.compose, 2,325 ms) is immediately recognizable as an optimization candidate. And the eval line ties the quality verdict to the model and prompt version: the basis for rollback and audit.

Relationship to monitoring in the security pillar

Tracing is both a debugging signal and a security signal. Agents have an unusually large "blast radius": a compromised agent can call many tools. The end-to-end trace provides exactly the audit log that makes this attack surface traceable: which agent called which tool, when, and with which identity. If an agent is a high-risk system under the AI Act, this trace is also the technical basis for the automatic recording of events that Art. 12 requires. The trace complements the controls from the security and identity context: deny-by-default egress with an allowlist and logging at the gateway, one service account per agent-tool pair instead of shared accounts, and the binding of every call back to the user identity via a token-exchange chain.

Observability at the network level via a service mesh (mTLS, workload identity, traffic shaping) closes the loop. Detailed measurement and control patterns for egress, identity and monitoring are covered in the security pillar; the observability layer provides the telemetry basis for this. The specific token economics and cost modeling, in turn, are the subject of the FinOps pillar.

For agencies and B2B decision-makers

For marketing agencies building agent workflows for clients, observability is the deliverable that separates a pilot from an auditable production system: only trace, eval and an EU-compliant backend make an agent operable, billable and defensible with regard to client data protection. For DACH B2B decision-makers, the rule is: choose the observability backend as deliberately as the cloud region. Self-hosted Langfuse, Datadog EU or Honeycomb EU keep prompts and responses within the EU, and vendor-neutral OpenTelemetry tracing protects against lock-in. Blck Alpaca from Vienna designs and operates agent infrastructure with this observability layer from the outset, including tracing, an eval harness and a GDPR-compliant backend. Get in touch if you want to move an agent out of pilot status into auditable production.