Prompt Engineering for AI Agents

What is prompt engineering for agents?

Prompt engineering for agents refers to the engineering discipline by which an AI Agent consistently exhibits the desired behavior across multiple inference turns. Whereas classic prompt engineering (2022–2023) meant writing a single, clever instruction string, for an agent the context window is no longer static text but a dynamically composed system state: system prompt, tool definitions, tool results, conversation history, retrieved RAG chunks, scratchpad notes, and structured state.

Andrej Karpathy described this shift on 25 June 2025 as "the delicate art and science of filling the context window with just the right information for the next step," one day after Shopify CEO Tobi Lütke had coined the same term. Anthropic formalized it on 29 September 2025 in "Effective context engineering for AI agents" as "the natural progression of prompt engineering." Important for context: context engineering subsumes prompt engineering, it does not replace it. A good system prompt remains a necessary condition; it is simply no longer sufficient.

This hub page provides an overview of the five central building blocks: system prompts, tool descriptions, planning loops (ReAct), context-window management, and evaluation.

System-prompt architecture

The system prompt is the only piece of context an agent sees in every turn — it is code: versioned, reviewable, diffable. In 2026, production system prompts consistently structure themselves into four layers:

Anthropic recommends separating these sections via XML tags or Markdown headers — the model parses structured prompts more reliably. Anthropic calls the right length "the right altitude": practitioner reports converge on 500–3,000 tokens for the core (excluding tool schemas). Both extremes are harmful: overly long prompts with 47 numbered rules cause the model to apply later rules less often (lost-in-the-middle within its own system prompt); overly vague prompts ("You are a helpful assistant") leave too much to model inference.

A central shift concerns absolute rules: OpenAI's GPT-5 guidance (2026) explicitly warns to use ALWAYS/NEVER only for genuine invariants — "for judgment calls, such as when to search, ask for clarification, use a tool, or keep iterating, prefer decision rules instead." For DACH teams the following additionally applies: always set the response language explicitly (otherwise frontier models default to English for technical content), and explicitly pin the level of formality (Sie/Du) to avoid style drift in the loop.

Tool descriptions: where agents actually fail

When a production agent acts incorrectly, the cause, according to Anthropic's own engineering experience, lies "in most cases" not with the model but with the tool definition. The guiding question is: "If a human engineer can't definitively say which tool should be used in a given situation, an AI agent can't be expected to do better."

What is decisive is that tool descriptions are part of the prompt budget. Each tool adds 100–300 tokens "always-on"; a 10-tool catalog costs 1,000–3,000 tokens per call. The most impactful, most often forgotten component is the when-not-to-use clause: if search_web and query_internal_db exist in parallel, this clause determines tool selection. Tool overlap — two tools that could plausibly answer the same query — is the one problem that no prompt, however good, can solve.

The empirical evidence is clear. With Anthropic's tool_search tool and defer_loading: true for rarely used tools, tool-selection accuracy in internal MCP evals rose on Opus 4 from 49% to 74%, and on Opus 4.5 from 79.5% to 88.1% (Anthropic, November 2025), with around 85% token savings. Beyond roughly 10 active tools, measurable degradation begins. The production pattern is therefore 3–7 always-loaded tools plus tool search for the rest.

Further robust conventions: verb-noun names (get_user, send_email), field-level descriptions with semantics, documented return formats and failure modes, search-focused rather than list-all tools, as well as hard response-token limits (Anthropic guideline ~25,000 tokens per tool return). DACH practice: tool names and parameters in English (interoperability), descriptions in the agent's runtime language.

Planning loops: ReAct and its successors

Planning is the structure of an agent's decision loop. Four patterns dominate in 2026, with clear trade-offs:

ReAct interleaves reasoning and action (Thought → Action → Observation) and is the robust default for non-reasoning models. With reasoning models the picture shifts: interleaved thinking (Anthropic Claude with Extended Thinking, OpenAI o-series/GPT-5) lets the model re-plan between tool calls and thereby collapses many classic "ReAct-in-a-loop" implementations into a single API call. In practice, production agents use hybrids: a ReAct loop with an explicit planning step at the start and verification before irreversible actions.

Disciplined termination is the most important safeguard — infinite loops were the most common production bug class in 2025–2026. Robust loops combine max-iterations (hard cap, typically 10–30 general, 50–100 for coding), a success criterion (e.g., a submit_final_answer tool), cost caps, repeated-state detection against tool thrashing, and a human-escalation path.

Context-window management

Long context windows are available in 2026 (Claude Opus 1M, Gemini 2M tokens), but not uniformly usable. Chroma's "Context Rot" study (July 2025, 18 frontier models) demonstrates: all models degrade with increasing input length. Three mechanisms compound: lost-in-the-middle (Liu et al., Stanford/TACL 2024 — models attend to the beginning and end, poorly in the middle), attention dilution, and distractor interference. As a heuristic, the effective capacity is 30–50% of the nominal for reasoning-heavy and 60–80% for retrieval-heavy tasks — filling a 1M window completely is wasteful with a quality penalty.

Lance Martin (LangChain) coined the canonical four-pillar taxonomy for this, which Anthropic and Manus have adopted:

• Write — persist information outside the window (scratchpads, todo.md, memory store)
• Select — bring the right tokens in per step (RAG, tool filtering, sub-agent dispatch)
• Compress — keep only task-relevant tokens (summarization, Anthropic context editing)
• Isolate — split context across sub-agents and schema fields

Three levers dominate the economics. Prompt caching is the most important: Anthropic cache reads cost roughly 10% of the standard input rate (≈90% discount), OpenAI offers around 50%. An arXiv study (February 2026, "Don't Break the Cache") measures across agentic workloads 41–80% lower API costs and 13–31% shorter time-to-first-token through strategic cache-block control. Pruning removes old turns and stale tool results. Compaction compresses at 70–85% capacity — in Claude Code via /compact, which according to Anthropic preserves "architectural decisions, unresolved bugs, and implementation details" and discards redundant tool outputs. Sub-agent dispatch acts as a compaction primitive: a sub-agent explores in its own window and returns only a 1,000–2,000-token summary.

Evaluation: if you don't measure, you've done nothing

The most brutal insight for tech leads is: context-engineering changes are validated by evals, not by intuition. Hamel Husain's most-cited advice — "Look at your data" — means concretely: read 50–100 real production traces, label failures freely, cluster them into a taxonomy, write a code eval or an LLM-as-judge eval per common mode, and integrate these into CI/monitoring.

Husain warns here against pure eval-first development: "Write evaluators for errors you discover, not errors you imagine." The practicable middle path starts with a small end-to-end eval (10–50 representative tasks), iterates, and builds specific sub-evals for real failure modes. A sobering, empirically substantiated finding: many popular prompt tips ("You are an expert," "Think step by step," "I'll tip you $200") show minimal or no improvement in rigorous evals — on reasoning models, "think step by step" is already default behavior and manually often counterproductive.

Production maturity means: evals run automatically on every change to context, tools, or retrieval — a PR eval (20–50 tasks) blocks merge, a pre-deploy eval (200–2,000 tasks) blocks deploy, a post-deploy eval on production traces performs drift detection. Structured-output enforcement closes the loop: OpenAI Structured Outputs (GA since August 2024) and Anthropic tool use with JSON schema deliver 100% schema adherence and replace the earlier "parse JSON and hope."

DACH relevance and compliance

For DACH teams (Germany, Austria, Switzerland), three hard engineering constraints are added. First, German tokenization: compound nouns and inflection produce 30–50% more tokens per equivalent content than English. A 200K window therefore holds only ~130–150K tokens of German content — higher costs, but also a higher caching ROI, because the 90% discount applies to a larger token count.

Second, GDPR discipline within the context window: personal data does not belong in it unfiltered. Patterns are pseudonymization before context injection (resolving real names only at the tool layer), a PII-redaction layer before the RAG inject, and an auditable session-state reset. Third, EU AI Act logging under Art. 12, which becomes fully applicable to high-risk systems as of 2 August 2026 (provisional or phased applicability — this classification is informational and not legal advice). On the engineering side this means: system-prompt version, tool-catalog version, retrieved documents (or IDs + hashes), user input, tool calls, tool results, and final output must be persisted in an audit-capable and at the same time GDPR-deletable manner — recommended per tool call with a run correlation ID.

Outlook and practical note

In 2026, prompt engineering for agents is neither folklore nor a mere rebranding, but the response to the shift from one-shot calls to multi-stage agentic loops. Anyone building a production agent operates this discipline — the only question is whether deliberately and reproducibly, or unconsciously and fragilely. Several fields remain in motion and should be read as a "current snapshot": the reasoning-model conventions, the multi-agent-vs.-single-agent heuristic (read-heavy parallel works, write-heavy does not), as well as prompt-optimization frameworks such as DSPy, which are suitable for narrowly scoped sub-tasks but do not yet form a production standard for complete agent loops.

The practical entry point is undramatic and well documented: pull the system prompt from the vendor playground and version it in Git, schema validation on every output, enable prompt caching on the stable parts, set hard cost caps, and read 20 real traces weekly. Anthropic's guiding principle captures the entire goal more precisely than any tooling discussion: "Find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome." That is engineering — and then: look at your data.