Building AI Agent Infrastructure

What AI agent infrastructure is and why it determines success or failure

AI agent infrastructure is the sum of all technical and organizational building blocks on which production AI agents run: the deployment topology (where and under which legal jurisdiction the agent is operated), the inference and orchestration stack, the network and identity layer, monitoring/observability, as well as cost and security governance. Unlike a simple chatbot, an agent is a multi-stage, tool-using system that generates east-west traffic between the orchestrator, tool servers, memory, and vector store, thereby placing entirely different demands on network, identity, and observability.

For DACH decision-makers, the infrastructure is the point at which decisions are made about data sovereignty, latency, and compliance. A conceptual clarification upfront, because it is the most common confusion in DACH projects: data residency refers to the physical storage/processing location, while data sovereignty refers to the legal jurisdiction including extraterritorial reach (such as the US CLOUD Act of 2018). A "Frankfurt region" of a US hyperscaler delivers residency, not sovereignty. In DACH mid-market parlance, "on-prem" usually does not even mean an in-house server room, but a dedicated environment in a German/Austrian/Swiss carrier-neutral colocation data center.

Cloud vs. on-prem vs. hybrid: the EU-sovereign topology question

The topology rarely results from a single option. Most production stacks span at least two. Five drivers determine the choice, roughly in this order of weighting: data sensitivity/regulatory class, latency SLO, sovereignty requirement, cost predictability, and existing in-house platform know-how.

The EU-sovereign market matured considerably in 2025/2026. Microsoft completed the EU Data Boundary on 26 February 2025 and committed to keeping end-to-end AI data processing for EU customers within this boundary, unless the customer determines otherwise. AWS made its European Sovereign Cloud generally available in January 2026, with the first region in Brandenburg (announced in 2023, EUR 7.8 billion investment, operated by companies under German law with an EU citizen as managing director; around 90 of more than 240 services at launch according to an AWS whitepaper from September 2025).

Alongside this stands a distinct DACH-native category that is usually absent from generic English-language enterprise AI literature: STACKIT (Schwarz Digits, with a data center in Austria as well; EUR 11 billion announced for an AI DC expansion, target up to 100,000 GPUs), Open Telekom Cloud / T Cloud Public (Deutsche Telekom/T-Systems, "Sovereignty by Design," together with NVIDIA the Munich Industrial AI Cloud with up to 10,000 Blackwell GPUs from Q1 2026), IONOS (AI Model Hub with Teuken-7B and Llama 3.3, first Legal AI Factory with Noxtua), Swisscom (Swiss AI Platform, deployment partner for the open Swiss LLM Apertus), and Infomaniak (fully Swiss-controlled, FADP- and GDPR-compliant). T-Systems has publicly pledged to close the feature gap to the hyperscalers by the end of 2026. Read this as a roadmap commitment, not as the current state.

The dominant DACH pattern remains hybrid: Sensitive documents, embeddings, and the vector store remain on-prem or in the sovereign cloud, and only the generation step calls a hyperscaler API, often via an egress-controlled proxy. In addition, confidential-computing patterns (model in the EU region, customer holds the keys via HYOK) and cloud bursting to GPU specialists for peak loads are becoming established.

Orchestration and inference stack

The inference stack is the most volatile layer. A clear industry signal: Hugging Face placed TGI into maintenance mode on 11 December 2025 and directs new deployments to vLLM or SGLang. For self-hosted production, vLLM (PagedAttention, broadest hardware support, OpenAI-compatible endpoints) is the de facto standard in 2026; SGLang scores with multi-turn chat and structured output. NVIDIA NIM (pre-built, optimized microservices, portable across cloud, data center, and RTX workstations) is considered the most pragmatic on-prem path in the DACH mid-market.

Above the inference engine, the AI gateway has established itself as a distinct architectural component. It handles multi-provider failover, virtual keys, team budgets, observability, guardrails, and PII redaction. Practical shortlist: LiteLLM (open source, self-hosted, OpenAI-compatible for 100+ providers, ideal when audit authority matters), Portkey (managed and on-prem, strong observability and governance), and Kong AI Gateway (when Kong is already the standard anyway). At the orchestration level, the spectrum ranges from frameworks such as LangGraph, CrewAI, and AutoGen to vendor stacks such as Microsoft Foundry Agents or Aleph Alpha's sovereign Pharia platform. Aleph Alpha's merger with Cohere was announced in April 2026 and signed in September 2026 (reported valuation around USD 20 billion). Closing is still subject to regulatory approval.

Latency is architecturally central. With co-located inference, network latency stays in the single-digit milliseconds. A transatlantic call (Frankfurt agent to a US East API) adds around 90 ms of round-trip time (RTT) for the network path alone, as ping measurements between Frankfurt and Washington show. Five tool-call rounds add up to around 450 ms of pure network time before the compute time of the model calls is even counted. That makes sub-second agent UX hard to achieve with transatlantic API calls.

Monitoring and observability

Agentic workloads cannot be steered productively without observability. What is required: trace standards (OpenTelemetry for LLMs, OpenInference), token-accurate cost attribution, and eval harnesses. DACH-residency-compliant backends are available: Langfuse self-hosted in the EU, Datadog EU, or Honeycomb EU. For a first trial via Docker Compose, Langfuse recommends a single VM with at least 4 cores and 16 GiB of memory; this setup has no high availability and no backups. On the cost side, observability typically accounts for 2-8% of total TCO, ranging from practically zero (Helicone Free, self-hosted Langfuse) up to EUR 5,000-50,000/month for Datadog LLM Observability at enterprise scale.

Two points are regulatorily relevant (informational, not legal advice): First, for systems classified as high-risk, the EU AI Act under Art. 12 requires event logging of inputs, outputs, and decisions with auditable granularity, which in enterprise implementations requires dedicated logging and storage infrastructure. Second, model versions should be pinned and accompanied by a documented rollback plan, since managed APIs change their versions on the provider's schedule.

Costs, FinOps, and token economics

2026 is the first year in which AI agent workloads demand genuine FinOps discipline. Two structural breaks coincide: Agentic workflows multiply token consumption per request by 5x to 50x (planner, tool call, critique, revision, verification), and the pricing ladder has split: the entry class (Haiku/Mini/Flash) has fallen 10x to 100x since 2023, while the frontier class lingers at around USD 5/25-30 per million tokens. The consequence: The list price no longer correlates with the monthly bill. The gap between vendor list price and production TCO typically amounts to 2x to 10x.

The decisive point: At scale, the API tokens are usually less than half of the total TCO. Direct model costs account for 30-50%, plus tool-use cascades (+50% to +200% on the direct API line), sub-agent fan-out (3x to 10x multiplier), compute/sandbox (10-25%), vector DB/embedding (5-15%), observability (2-8%), compliance/governance (5-20%), and operations labor (10-30%).

The most effective levers lie below the API line:

• Caching is the single biggest lever. Anthropic grants a 90% discount on cache reads (cached input on Sonnet 4.6: 0.30 instead of 3.00 USD/M), OpenAI bills cached input at 10% of the base price. At an 80% cache hit rate, input costs drop by 70-80%.
• Model routing: cheap model for simple tasks, expensive only for complex ones. Anthropic's advisor-tool benchmark (Sonnet + Opus advisor) reached 74.8% on SWE-bench Multilingual at 11.9% lower cost than Opus alone.
• Batch API: a flat 50% discount with a 24-hour SLA, stackable with caching.
• Open-weight fallback for long-tail workloads (DeepSeek V4 Flash, Mistral Ministral, Qwen 3). They are GDPR-compliant only via EU-hosted routes (Together AI EU, DeepInfra Frankfurt, STACKIT/OVHcloud), not via China-hosted direct APIs.

Stacked, a well-instrumented FinOps program delivers a 60-80% cost reduction compared to the unoptimized baseline. The DACH reality adds further cost: EU regions add a 10% uplift at OpenAI and Anthropic, sovereign hosting costs roughly 1.5x to 3x the US cloud price (SAP Joule AI Units approx. 1.5-2x), and compliance ops plus co-determination drive total TCO 15-35% higher than a comparable US workload. Per vendor, EUR 5,000-20,000/year in ongoing DPA/sub-processor costs apply. A 2025 Bitkom survey underscores the sovereignty pressure: 68% of respondents consider Germany too dependent on the US and China for AI, and 60% want Germany to become more independent of US AI companies.

Security and identity

Identity and key management is the lever that makes a non-sovereign hyperscaler region into something defensible under DACH compliance (informational, not legal advice; the detailed GDPR/DPA treatment belongs in the sister topics). The architectural baseline:

• mTLS between all agent components, which is also a typical piece of evidence in BSI C5 and ISO 27001 audits.
• OIDC/SAML federation for enterprise SSO (Entra ID, Okta, KeyCloak); the agent exchanges the user token for short-lived tokens for tool calls.
• Workload identity (Azure Managed Identity, AWS IRSA, GCP Workload Identity Federation, in sovereign clouds OpenStack Keystone / K8s service accounts) instead of static credentials in the code.
• KMS/HSM with BYOK/HYOK: With BYOK, the provider continues to operate the key; with HYOK, the cloud calls the customer-owned HSM (Utimaco/Aachen, Thales) for every crypto operation. This is the strongest sovereignty position among the key management models.

An agent has an unusually high blast radius because it can call many tools. Best practice: one service account per (agent x tool) pair (not a shared account), just-in-time elevation, all credentials from Vault or KMS rather than from environment variables, and an audit trail that binds back to the user identity via a token-exchange chain. On the network side, deny-by-default egress with an explicit allowlist of the model API FQDNs has become established. It prevents unwanted data outflows, provides audit evidence, and forces all model traffic through the gateway, where rate limits, PII filters, and budgets reside.

DACH compliance notes and outlook

Several DACH-specific rules drive real architecture decisions (informational, not legal advice): Since 1 July 2025, BSI C5 Type 2 attestation has been mandatory for cloud processing of patient data (DigiG / § 393 SGB V). Switzerland does not follow the GDPR but FADP/revDSG (in force since 1 September 2023). According to the resolution that privatim, the conference of Swiss data protection commissioners, published in November 2025, public bodies may move particularly sensitive personal data into SaaS solutions from large international providers only if they encrypt the data themselves and the provider has no access to the key. The EU AI Act phases in on a staggered basis (implementation timeline of the European Commission): prohibitions from February 2025, GPAI rules from August 2025, transparency obligations under Art. 50 from 2 August 2026. The AI Omnibus, in force since 27 July 2026, postponed the high-risk obligations to 2 December 2027 for Annex III systems and to 2 August 2028 for AI systems embedded in products such as lifts or toys that fall under Annex I legislation. Which obligations apply depends on your role (provider or deployer) and on how the system is classified.

Practical note: Do not start with the procurement question "cloud or on-prem?", but with data classification and the latency SLO, because these determine the topology. Build the AI gateway, deny-by-default egress, and eval-driven model selection in from day one, because it is precisely these "pilot gaps" that typically break at production launch. For the mid-market, an M365-anchored hybrid with an EU data zone and a small on-prem RAG layer is the pragmatic default; for regulated industries, the path leads via STACKIT/Open Telekom Cloud with HYOK and sovereign inference. Since sovereign-cloud roadmaps shift on a quarterly basis, every architecture decision should be accompanied by a date stamp ("as of: ...") and a documented migration trigger.