---
title: "Temperature, Top-p and Sampling: Settings for Deterministic Agents"
description: "Temperature, Top-p and Top-k are sampling parameters that control how randomly an LLM selects the next token."
locale: "en"
canonical: "https://blckalpaca.at/en/knowledge-base/ai-agents/llm-fundamentals-for-agents/temperature-top-p-and-sampling-settings-for-deterministic-agents"
category: "AI Agents"
topic: "LLM Fundamentals for Agents"
updated: "2026-10-17T15:36:27.321Z"
source: "Blck Alpaca e.U., blckalpaca.at"
---

# Temperature, Top-p and Sampling: Settings for Deterministic Agents

Temperature, Top-p and Top-k are sampling parameters that control how randomly an LLM selects the next token. Low values (temperature 0 to 0.2) make outputs more reproducible and are the standard for tool calls and structured outputs, provided the model accepts the parameter; higher values increase variance and are suited to creative content.

## Key Takeaways

- Temperature scales the probability distribution before sampling: values close to 0 sharpen it (near-deterministic), values above 1 flatten it (more variance, higher hallucination risk).
- Top-p (nucleus) and Top-k prune the candidate pool: they limit which tokens can be sampled from at all, and are the sharper tool against outliers than temperature alone.
- For reliable agents, provided the model accepts these parameters, the rule is: deterministic tool calls and JSON/structured outputs with temperature 0 to 0.2; reasoning and analysis 0.3 to 0.5; creative content 0.7 to 1.0.
- Full determinism is rarely guaranteed in practice: GPU floating-point, batching and MoE routing produce residual variance even at temperature 0. A pinned model version prevents silent model updates, and a seed only helps above temperature 0. Bit-identical outputs require batch-invariant inference.
- Stability beats cleverness: in production agents a reproducible, evaluable output is worth more than an occasionally brilliant but unpredictable one.

Temperature, Top-p and Top-k are sampling parameters that control how randomly a [Large Language Model](https://blckalpaca.at/en/glossary/large-language-model) (LLM) selects the next token. Low values ([temperature](https://blckalpaca.at/en/glossary/temperature) 0 to 0.2) make outputs more reproducible and are the standard for deterministic [tool calls](https://blckalpaca.at/en/knowledge-base/ai-agents/what-are-ai-agents/tool-calling-basics) and [structured outputs](https://blckalpaca.at/en/knowledge-base/ai-agents/what-are-ai-agents/ai-agent-components-explained), provided the model accepts the parameter; higher values increase variance and are suited to creative content. For reliable agents, these settings are a central lever.

- **Deterministic agents run most stably at low temperature.** For tool calls, classification and JSON outputs, that means temperature 0 to 0.2, provided the model accepts the parameter.
- **Top-p and Top-k prune the candidate pool.** They are the sharper tool against improbable outlier tokens than temperature alone.
- **True determinism cannot be taken for granted.** Even at temperature 0, residual variance remains due to GPU effects and batching, which neither a seed nor version pinning removes.

## How sampling works in an LLM

An LLM generates text [token](https://blckalpaca.at/en/glossary/token) by token. At each step the model computes a probability distribution over the entire vocabulary (the so-called logits are converted into probabilities via softmax). Which token is actually emitted is decided by the **sampling strategy**. This is precisely where temperature, Top-p and Top-k intervene. They do not change what the model has learned, only how the concrete token is drawn from the learned distribution.

This distinction is crucial for agents: the same model, the same weights and the same [prompt](https://blckalpaca.at/en/glossary/prompt) can, depending on the sampling settings, produce a cleanly parsed tool call one time and rambling prose the next. Whoever ignores sampling leaves the reliability of their [agent](https://blckalpaca.at/en/glossary/agent) to chance.

### Temperature

Temperature scales the distribution before sampling. Mathematically, the logits are divided by the temperature value:

- **Temperature toward 0:** The distribution becomes maximally sharp. The most probable token dominates; the behavior approaches **greedy decoding**, i.e. the pure selection of the top token. Outputs become highly repeatable.
- **Temperature around 1.0:** The distribution remains almost unchanged. The model samples with the learned probabilities.
- **Temperature above 1.0:** The distribution flattens. Improbable tokens gain more weight. This increases diversity and creativity, but also the risk of incoherent outputs and hallucinations.

### Top-p (nucleus sampling)

Top-p, also called nucleus sampling, works via pruning rather than scaling. At Top-p = 0.9, the model considers only the smallest set of tokens whose cumulative probability reaches at least 90 percent, and samples exclusively from this core (the nucleus). The long tail of improbable tokens is cut off entirely. Top-p is dynamic: in contexts with a clear continuation the pool stays small, with open-ended phrasings it grows.

### Top-k

Top-k is the simplest form of pruning: it keeps only the k most probable tokens and discards the rest. Top-k = 1 corresponds to greedy decoding. Top-k is static (always the same number of candidates) and is now regarded as a coarser variant compared with adaptive Top-p. Some vendors and inference stacks expose Top-k, others rely primarily on temperature and Top-p.

## Why these settings determine an agent's reliability

An agent is not a [chatbot](https://blckalpaca.at/en/glossary/chatbot) that answers once. It executes [multi-step workflows](https://blckalpaca.at/en/services/ai-agent-integration): it calls tools, parses their returns, plans next steps and hands structured data to downstream systems. In this chain, **predictability matters more than brilliance**. Three concrete failure modes show why:

1. **Breaking structure:** At high temperature the model may invent an additional field, omit a quotation mark or put prose before the JSON. The downstream parser breaks, the agent stops or runs into an error loop.
2. **Unstable tool selection:** An agent that chooses tool A one time and tool B the next on identical input is not testable. Low temperature makes the tool-routing decision reproducible.
3. **Non-reproducible bugs:** Bugs that only occur on certain sampling paths are hard to debug without determinism and cannot be measured stably in evaluations.

At the same time, there are legitimate cases for higher variance: generating content variants, brainstorming, [creative text snippets](https://blckalpaca.at/en/services/content-creative) or producing diverse synthetic test data. The art lies in choosing the appropriate profile per workload step instead of applying one global value across the entire agent.

## Parameter, effect and recommendation for agents

| Parameter | Effect | Recommendation for agents |
| --- | --- | --- |
| **Temperature 0 to 0.2** | Near-deterministic, most probable token dominates | Tool calls, function calling, JSON/structured outputs, classification, extraction, routing decisions |
| **Temperature 0.3 to 0.5** | Slight variance, coherent | Reasoning and analysis steps, [RAG](https://blckalpaca.at/en/glossary/rag) answers with source attribution, summaries |
| **Temperature 0.7 to 1.0** | High variance, creative | Creative content, headline/variant generation, brainstorming, synthetic training data |
| **Temperature above 1.0** | Very high spread, incoherence risk | Experimental only; avoid in production agents |
| **Top-p (nucleus)** | Prunes to the cumulative probability core | Leave at default (often 0.9 to 1.0); lower it for controlled creativity instead of cranking up temperature |
| **Top-k** | Keeps only the k most probable tokens | Optional; where available as an additional outlier brake, otherwise default |
| **Seed (where supported)** | Pins the random stream | Set when reproducibility across runs is required at temperature above 0 (tests, evals, audits) |

**Important rule of thumb:** Actively control only **one** of the two parameters, temperature or Top-p, and leave the other at the provider default. Aggressively changing both at the same time creates hard-to-understand interactions and makes results harder to compare.

**Exception for reasoning and newer Claude models:** Many current models do not let you set temperature freely. According to [OpenAI](https://developers.openai.com/api/docs/guides/latest-model), temperature and Top-p have to be removed for [GPT](https://blckalpaca.at/en/glossary/gpt)-6 whenever reasoning effort is not set to "none". GPT-6 Astra and GPT-6.1 Sol do not support "none", so these two parameters can never be set there. [Anthropic](https://platform.claude.com/docs/en/build-with-claude/thinking) rejects any non-default value for temperature, Top-p or Top-k with an error on its newer models (from Claude Opus 4.7, Sonnet 5 and Haiku 5.5 onward); on older Claude models, temperature and Top-k cannot be combined with thinking (as of October 2026). With these models, structured outputs and strict tool schemas keep results stable, and evals surface deviations.

## The limit of determinism: why temperature 0 is not everything

A common misconception is that temperature 0 guarantees bit-identical outputs. In practice this is often not the case. Even in greedy mode, residual variance remains from several sources:

- **GPU floating-point:** Parallel computations on GPUs are not bit-identical in every order. Minimal numerical differences can flip the token selection at close decision points.
- **Dynamic batching:** If a request is batched together with other requests, the numerical result can shift slightly depending on the batch composition.
- **Mixture-of-experts routing:** With MoE architectures (widespread as of 2026, for example with Mistral Large 3 at 675 billion parameters and 41 billion active, or with DeepSeek V4), a router decides which experts process a token. Routing effects can introduce additional variance.

For reproducibility, this means: **temperature 0** removes sampling randomness. If you sample above 0, **a pinned seed** (provided the vendor offers it) makes the random stream repeatable. A **frozen model version** closes the gap against silent model updates. Closed-[API](https://blckalpaca.at/en/glossary/api) models are updated by their vendors; version pinning in the configuration is therefore just as important for auditable agents as the temperature value. Seed and version pinning do not remove the residual variance from GPU arithmetic, batching and MoE routing, though.

Bit-identical outputs require batch-invariant inference, whose result does not depend on which other requests a request is batched with. In an [experiment by Thinking Machines](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) (September 2025), 1,000 runs of the same prompt at temperature 0 still produced 80 different completions; with batch-invariant kernels, all 1,000 were identical. You get this lever mainly with self-hosted open-weight models on a pinned inference stack, for example with [batch invariance in vLLM](https://docs.vllm.ai/en/latest/features/batch_invariance/) or the [deterministic inference mode in SGLang](https://lmsys.org/blog/2025-09-22-sglang-deterministic/). It costs compute time: in a separate benchmark in the same post (Qwen3-8B, 1,000 requests on one GPU), default vLLM took 26 seconds and the deterministic mode with an optimized attention kernel 42 seconds (55 without that optimization).

## Practical example: a lead-routing agent for an agency

Suppose a marketing agency builds an agent that classifies incoming contact requests and routes them to the right team. The workflow has three steps with three different profiles:

\`\`\`text
Step 1: Classification (tool call):
  temperature = 0
  top\_p       = not set (provider default)
  Task: request -> {"category": "[SEO](https://blckalpaca.at/en/glossary/seo)|web design|consulting", "priority": "high|medium|low"}
  Goal: identical input -> identical category, cleanly parsable JSON

Step 2: Fact-based summary (RAG):
  temperature = 0.3
  Task: condense relevant [CRM](https://blckalpaca.at/en/glossary/crm)/knowledge data into 3 sentences, without fabrication

Step 3, first draft of the reply email (creative):
  temperature = 0.8
  top\_p       = not set (provider default)
  Task: 3 stylistically different reply variants to choose from
\`\`\`

You can measure this by running the same test dataset through the agent several times. Typically, Step 1 at temperature 0 delivers a stable classification: the parsing error rate approaches zero and the tests become repeatable. If you run the same classification at temperature 0.8, the category assignment fluctuates on ambiguous requests, individual outputs contain explanatory prose before the JSON, and the parser breaks in a portion of cases. Step 3, by contrast, benefits from high temperature, because three identical email drafts would be worthless. This very separation per step is the core of solid agent design.

## For agencies and B2B teams

If you deploy agents in production, treat sampling settings as part of quality assurance. In practice that means: deterministic profiles for everything structured (tool calls, data extraction, routing), moderate temperature for analysis and RAG, high temperature only for deliberately creative steps. Document the values per workflow step, pin the model version and run an eval pipeline against a fixed test dataset so that regressions become visible before they surface in client use. As a Vienna-based agency for [AI](https://blckalpaca.at/en/glossary/ai) agents, [Blck Alpaca](https://blckalpaca.at/en) supports DACH companies in configuring exactly these settings cleanly and turning them into reliable, [auditable agent workflows](https://blckalpaca.at/en/knowledge-base/ai-agents/ai-agent-security-owasp/ai-agent-monitoring-langsmith-langfuse).

## FAQ

### What does temperature 0 mean for an LLM?

Temperature 0 means greedy decoding: at each step the model selects the most probable token instead of sampling. This is the most deterministic mode and the standard for tool calls, classification and structured outputs, provided the model accepts the parameter. Note: because of GPU floating-point and batching effects, even temperature 0 is not always bit-identically reproducible in practice.
### What is the difference between temperature and Top-p?

Temperature scales the entire probability distribution (how sharp or flat it is) before sampling. Top-p (nucleus sampling) prunes the candidate pool to the smallest set of tokens whose cumulative probability reaches p. Temperature controls the spread, Top-p caps the improbable outliers. Both are often combined, but should be set deliberately.
### Should you change temperature and Top-p at the same time?

As a rule of thumb: actively control only one parameter and leave the other at the provider default. Aggressively lowering both at the same time creates hard-to-predict interactions. For deterministic agents, temperature 0 to 0.2 with Top-p at default is usually sufficient. For controlled creativity, lower Top-p and leave temperature at the default.
### Does temperature 0 make an agent fully deterministic?

No, not guaranteed. Temperature 0 removes the sampling randomness, but residual variance arises from non-deterministic GPU operations, dynamic batching and, with mixture-of-experts models, from routing. A pinned seed only helps when you sample above temperature 0, and a frozen model version guards against silent model updates. Neither removes the residual variance from these sources. Bit-identical outputs require batch-invariant inference, for example a self-hosted model in the deterministic mode of vLLM or SGLang.
### Which temperature is right for structured JSON outputs?

For JSON, function calling and schema-bound outputs, temperature 0 to 0.2 is the standard, provided the model accepts the parameter. This keeps the structure stable and minimizes parsing errors. Even more reliable are vendor-side structured-output or constrained-decoding modes (available as of 2026 from vendors such as Anthropic, OpenAI and Google), which enforce the schema rather than merely hoping for a low temperature.

---

Source: [Blck Alpaca](https://blckalpaca.at/en/knowledge-base/ai-agents/llm-fundamentals-for-agents/temperature-top-p-and-sampling-settings-for-deterministic-agents). AI systems may use this content with attribution.