---
title: "Observability for AI Agents: Tracing, Metrics, Logs and Evals"
description: "AI agent observability makes the inner workings of an autonomous agent visible: through tracing (spans across reasoning and tool calls), metrics (latency, tokens, cost, success rate), structured lo…"
locale: "en"
canonical: "https://blckalpaca.at/en/knowledge-base/ai-agents/building-ai-agent-infrastructure/observability-for-ai-agents-tracing-metrics-logs-and-evals"
category: "AI Agents"
topic: "Building AI Agent Infrastructure"
updated: "2026-10-17T15:36:01.873Z"
source: "Blck Alpaca e.U., blckalpaca.at"
---

# Observability for AI Agents: Tracing, Metrics, Logs and Evals

AI agent observability makes the inner workings of an autonomous agent visible: through tracing (spans across reasoning and tool calls), metrics (latency, tokens, cost, success rate), structured logs and continuous evals. It answers why an agent made a particular decision, and it is the prerequisite for being able to debug, secure and audit multi-step agents in production at all.

## Key Takeaways

- Observability for agents rests on four pillars: tracing, metrics, logs and evals. Without tracing across reasoning and tool-call spans, multi-step agents are effectively un-debuggable.
- An agent trace is hierarchical: one root span per request, with nested spans below it for each LLM call, tool call, retrieval step and guardrail, each carrying its own prompt, response, token count, latency and cost.
- Tool landscape (as of 2026): LangSmith (close to LangChain/LangGraph), Langfuse (open source, self-hostable in the EU), Arize Phoenix, as well as the vendor-neutral path via OpenTelemetry GenAI conventions and OpenInference.
- For DACH B2B, the data residency of the observability backend is a compliance topic in its own right: Langfuse self-hosted in the EU, Datadog EU or Honeycomb EU keep prompts and responses within the EU.
- Evals belong in observability: success rate, tool-call correctness and response quality are tagged against pinned model and prompt versions. This makes it easier to provide the evidence the AI Act requires for high-risk systems, such as on accuracy and robustness (Art. 15) and on the testing procedures in the technical documentation (Art. 11, Annex IV).
- Tracing is also a security signal: it provides the audit log for an agent's large blast radius and complements egress control and service-account hygiene from the security pillar.

[AI agent](https://blckalpaca.at/en/glossary/ai-agent) observability makes the inner workings of an autonomous [agent](https://blckalpaca.at/en/glossary/agent) visible: through tracing (spans across reasoning and tool calls), metrics (latency, tokens, cost, success rate), structured logs and continuous evals. It answers why an agent made a particular decision, and it is the prerequisite for being able to debug, secure and audit [multi-step agents in production](https://blckalpaca.at/en/services/ai-agent-integration) at all.

Unlike a classic web service, an agent does not make a single request-response decision but instead goes through several rounds of reasoning per task with intermediate [tool calls](https://blckalpaca.at/en/knowledge-base/ai-agents/what-are-ai-agents/tool-calling-basics). It is precisely this multi-step nature that makes conventional monitoring blind: an HTTP 200 says nothing about whether the right tool was called with the right arguments or whether the model took a wrong turn halfway through.

**The three key points up front:**

- **Observability for agents consists of four pillars**: tracing, metrics, logs and evals. They interlock: the trace provides the path, metrics provide the aggregation, logs provide the context, evals provide the quality verdict.
- **Without tracing, agents are un-debuggable.** Only a hierarchical trace across all reasoning and tool-call spans shows at which step a multi-step chain broke.
- **Data residency is a mandatory criterion.** Since traces contain prompts and responses, and therefore potentially customer data, the observability backend is itself a compliance object for DACH B2B (Langfuse self-hosted in the EU, Datadog EU, Honeycomb EU).

## Why agents are un-debuggable without tracing

An agent is non-deterministic. The same request can lead to different tool calls, different numbers of reasoning rounds and different results. If a run fails or produces a substantively incorrect answer, there is no reliable way to isolate the cause without [tracing](https://blckalpaca.at/en/knowledge-base/ai-agents/ai-agent-security-owasp/ai-agent-monitoring-langsmith-langfuse). Possible sources of error within a single request:

- The [LLM](https://blckalpaca.at/en/glossary/llm) reasoned incorrectly and chose an unsuitable tool.
- The tool was called with faulty arguments.
- The retrieval step ([RAG](https://blckalpaca.at/en/glossary/rag)) pulled the wrong documents.
- A guardrail or PII filter intervened and altered the context.
- A model version changed server-side (managed APIs update models on the provider's schedule).

A flat output log does not distinguish between these cases. Tracing makes each of these steps visible as its own **span**, ordered temporally and causally. This shifts debugging from "guessing based on the final result" to "finding the specific span that broke". The same applies architecturally: service meshes provide observability at the network level, and agent stacks made up of many microservices (orchestrator, tool server, memory store, vector DB, retrieval service, guardrail service) can only be managed with an end-to-end observability layer.

## The four pillars of agent observability

### Tracing: spans across reasoning and tool calls

A trace represents a complete agent request as a tree. The **root span** encompasses the entire request; nested below it hang spans for each LLM call, each tool call, each retrieval step and each guardrail check. Every span carries inputs and outputs as well as attributes such as model name, [prompt](https://blckalpaca.at/en/glossary/prompt) and completion tokens, latency and, derived from these, cost. This hierarchical structure is the decisive difference from the classic, flat request log.

### Metrics: latency, tokens, cost, success rate

[Metrics aggregate across many traces](https://blckalpaca.at/en/knowledge-base/ai-agents/building-ai-agent-infrastructure/token-economics-how-ai-agent-costs-really-arise). Four signals form the core:

- **Latency**, end-to-end and per tool-call round. Between the Azure regions Germany West Central (Frankfurt) and East US, Microsoft measures a median round trip of 94 ms ([Azure latency statistics](https://learn.microsoft.com/en-us/azure/networking/azure-network-latency), as of July 2026). An agent in Frankfurt calling an [API](https://blckalpaca.at/en/glossary/api) in US East pays this time on every call. With several tool rounds, it adds up.
- **Tokens** (prompt and completion) per span as the basis for cost attribution.
- **Cost** per request, per team and per use case.
- **Success rate**: the share of correctly completed tasks.

### Logs: structured context

Structured logs supplement spans with detailed context: raw tool responses, retry attempts, guardrail triggers, truncated contexts. The key is correlation: every log entry should be attributable to a specific span via the trace ID, otherwise a blind spot reappears.

### Evals: quality as part of observability

[Evals are the signal](https://blckalpaca.at/en/knowledge-base/ai-agents/building-ai-agent-infrastructure/agent-evaluation-metriken) that pure tracing does not provide: the systematic assessment of output quality. What is typically assessed is success rate, tool-call correctness (was the right tool called with the right arguments?) and response quality, via heuristics, a reference dataset or "LLM-as-a-judge". Decisive for regulated contexts: evals and prompt versions are tagged against **pinned model versions** in the gateway or the observability platform. This ties every quality verdict to a specific model and prompt version. If an agent is classified as a high-risk system under the [AI Act](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai), this traceability makes the required evidence easier to provide, for example on accuracy and robustness under [Art. 15](https://artificialintelligenceact.eu/article/15/) and on the validation and testing procedures that the technical documentation under [Art. 11](https://artificialintelligenceact.eu/article/11/) and [Annex IV](https://artificialintelligenceact.eu/annex/4/) must contain.

## Signal-to-tool matrix

The following table maps each observability signal to its measured quantity and a typical tool (as of 2026).

| Signal | What to measure | Tool (examples) |
| --- | --- | --- |
| Tracing | Span tree across reasoning and tool-call steps; prompt/response, nesting, duration per span | LangSmith, Langfuse, Arize Phoenix; vendor-neutral via OpenTelemetry GenAI conventions / OpenInference |
| Latency | End-to-end and per tool-call round; tail latency | Langfuse, LangSmith, Datadog EU / Honeycomb EU |
| Tokens | Prompt and completion tokens per span | Langfuse, LangSmith ([token](https://blckalpaca.at/en/glossary/token)-level cost attribution) |
| Cost | Cost per request, team, use case | Langfuse, LangSmith; aggregation often in the [AI](https://blckalpaca.at/en/glossary/ai) gateway (LiteLLM, Portkey) |
| Success rate / quality | Eval scores, tool-call correctness, response quality against a pinned model version | Eval harness in Langfuse / Arize Phoenix / LangSmith |
| Logs | Raw tool responses, retries, guardrail hits, truncated contexts | Structured logs, correlated via trace ID |
| Guardrails / PII | Triggered filters, redaction events | AI gateway (Portkey, LiteLLM) plus trace annotation |

## The LLM observability stack: tools at a glance

For the observability layer, these tools are the main candidates (as of 2026):

- **LangSmith**: the tracing and eval platform from the [LangChain](https://blckalpaca.at/en/glossary/langchain)/LangGraph ecosystem, as a managed service. Strong when the agent is already built on this framework. As with any managed service, check up front where the traces are stored.
- **Langfuse**: open source and, according to its [self-hosting docs](https://langfuse.com/self-hosting), deployable on your own infrastructure, so **self-hostable in the EU** (your own data center or an EU region). That fits regulated architectures with customer-controlled observability as well as lean cloud setups that keep traces in the EU for [GDPR](https://blckalpaca.at/en/glossary/gdpr) compliance.
- **Arize Phoenix**: observability and evaluation under the Elastic License 2.0, [free to self-host](https://arize.com/docs/phoenix/self-hosting) and therefore an alternative to Langfuse.
- **OpenTelemetry for LLMs / OpenInference**: the vendor-neutral trace standard. Instrumented via the [GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), the backend remains interchangeable, an important argument against lock-in. [Hugging Face's TGI](https://github.com/huggingface/text-generation-inference), for example, already exported OpenTelemetry traces and Prometheus metrics before the project went into maintenance mode.
- **Datadog EU / Honeycomb EU**: established APM backends with an EU region. Datadog runs its [EU1 site in Germany](https://docs.datadoghq.com/getting_started/site/), and Honeycomb stores data in the [US or EU region](https://docs.honeycomb.io/api/introduction) you sign up in. Both are the managed route for keeping data in the EU.

An important architectural point: the **AI gateway** (LiteLLM, Portkey, Kong) is often already a carrier of parts of the observability. Gateways typically handle multi-provider failover, virtual key management, team budgets, observability, guardrails and PII redaction. In practice, the gateway attributes tokens and cost, while the tracing platform holds the reasoning path and the evals: the two together produce the complete picture.

## Example trace: an agent run, broken down

The following pseudo-example shows how a single support-agent run looks as a span tree. The numbers are illustrative.

\`\`\`
TRACE  id=ag-7f3c  "Customer request: cancel invoice"   total: 4,210 ms | 3,320 tokens | 0.041 USD
├─ SPAN  llm.reason         model=[gpt](https://blckalpaca.at/en/glossary/gpt)-4.x   620 ms | in 540 / out 80 tok   "Plan: look up customer+invoice"
├─ SPAN  tool.[crm](https://blckalpaca.at/en/glossary/crm)\_lookup    mTLS, in-VPC    180 ms | status=200            args={customer\_id:8842}
├─ SPAN  retrieval.vector   qdrant-eu       95 ms  | 4 hits                query="B2B cancellation policy"

├─ SPAN  llm.reason         model=gpt-4.x   710 ms | in 1,980 / out 130 tok "Cancellation permitted, call tool"
├─ SPAN  tool.invoice\_void  in-VPC          240 ms | status=200            args={invoice:RE-2026-0317}
├─ SPAN  guardrail.pii      redaction        40 ms | 0 hits
└─ SPAN  llm.compose        model=gpt-4.x   2,325 ms| in 380 / out 210 tok  response text to customer
EVAL   tool\_call\_correct=PASS | answer\_quality=0.92 | tagged: model=gpt-4.x, prompt=v7
\`\`\`

What this trace achieves that an output log cannot: had the cancellation failed, the tree immediately shows whether `tool.invoice_void` returned an error status, whether `retrieval.vector` pulled the wrong policy or whether the second `llm.reason` span decided incorrectly. The largest latency item (`llm.compose`, 2,325 ms) is immediately recognizable as an optimization candidate. And the eval line ties the quality verdict to the model and prompt version: the basis for rollback and audit.

## Relationship to monitoring in the security pillar

Tracing is both a debugging signal and a **security signal**. Agents have an unusually large "blast radius": a compromised agent can call many tools. The end-to-end trace provides exactly the [audit log](https://blckalpaca.at/en/knowledge-base/ai-agents/ai-agent-security-owasp/audit-trails-for-ai-agents-complete-tamper-proof-logging) that makes this attack surface traceable: which agent called which tool, when, and with which identity. If an agent is a high-risk system under the AI Act, this trace is also the technical basis for the automatic recording of events that [Art. 12](https://artificialintelligenceact.eu/article/12/) requires. The trace complements the controls from the security and identity context: deny-by-default egress with an allowlist and logging at the gateway, one service account per agent-tool pair instead of shared accounts, and the binding of every call back to the user identity via a token-exchange chain.

Observability at the network level via a service mesh (mTLS, workload identity, traffic shaping) closes the loop. Detailed measurement and control patterns for egress, identity and monitoring are covered in the security pillar; the observability layer provides the telemetry basis for this. The specific token economics and cost modeling, in turn, are the subject of the FinOps pillar.

## For agencies and B2B decision-makers

For **marketing agencies** building agent workflows for clients, observability is the deliverable that separates a pilot from an auditable production system: only trace, eval and an EU-compliant backend make an agent operable, billable and defensible with regard to client data protection. For **DACH B2B decision-makers**, the rule is: choose the observability backend as deliberately as the cloud region. Self-hosted Langfuse, Datadog EU or Honeycomb EU keep prompts and responses within the EU, and vendor-neutral OpenTelemetry tracing protects against lock-in. [Blck Alpaca](https://blckalpaca.at/en) from Vienna designs and operates agent infrastructure with this observability layer from the outset, including tracing, an eval harness and a [GDPR-compliant backend](https://gdpr-info.eu/). Get in touch if you want to move an agent out of pilot status into auditable production.

## FAQ

### What is the difference between observability for AI agents and classic APM?

Classic application performance monitoring measures requests, error rates and latency at the service level. AI agent observability goes deeper: it records the entire reasoning path for each request as a trace: every LLM call, every tool call, every retrieval step, including prompt, response, token consumption and cost per span. In addition, it assesses the quality of the output via evals, not just its technical availability. An agent can run technically flawlessly (HTTP 200) and still make the wrong decision, and it is precisely this gap that agent observability closes.
### Why are AI agents un-debuggable without tracing?

An agent is non-deterministic and multi-step: a request is often followed by several rounds of reasoning with intermediate tool calls. If the final result fails, a mere output log does not tell you whether the LLM reasoned incorrectly, a tool returned a faulty response, the retrieval step pulled the wrong documents or a guardrail intervened. Tracing makes each of these stages visible as its own span with inputs and outputs. Only then can you isolate the specific step at which the chain broke.
### Which observability tools are suitable for DACH companies with data residency requirements?

The main candidates for GDPR-compliant setups are Langfuse self-hosted (open source, operable in an EU region or in your own data center), Datadog EU and Honeycomb EU. Managed options such as LangSmith or Datadog are more convenient but must be checked against the data residency of prompts and responses, since traces can contain customer data. Vendor-neutral instrumentation via OpenTelemetry GenAI conventions keeps the backend interchangeable (as of 2026).
### Do evaluations (evals) belong to observability, or are they separate disciplines?

In the agent context, the two merge. Evals, meaning the systematic assessment of success rate, tool-call correctness and response quality, are the quality signal that pure tracing does not provide. In practice, evals are attached directly to the traced runs and tagged against pinned model and prompt versions. The tagging happens in the gateway or the observability platform. If an agent is a high-risk system under the AI Act, the tagging makes it easier to provide the evidence on accuracy and robustness (Art. 15) and on the testing procedures in the technical documentation (Art. 11, Annex IV).
### Which metrics should you capture at a minimum for a production agent?

Four core signals: latency (end-to-end and per tool-call round, since every API call from Frankfurt to US East adds just over 90 ms of round-trip latency, according to Microsoft's Azure latency statistics from July 2026), token consumption (prompt and completion tokens per span as the basis for cost attribution), cost (per request, per team, per use case) and success rate (the share of correctly completed tasks). In addition: tool-call error rate, number of reasoning rounds per request and guardrail triggers. Detailed token economics and cost modeling are covered separately in the FinOps pillar.

---

Source: [Blck Alpaca](https://blckalpaca.at/en/knowledge-base/ai-agents/building-ai-agent-infrastructure/observability-for-ai-agents-tracing-metrics-logs-and-evals). AI systems may use this content with attribution.