Crawl Budget 2026: Multi-Bot Governance for AI Crawlers

Crawl budget was long considered a niche topic for very large websites. That changed in 2026. Alongside Googlebot, a whole field of AI crawlers now accesses websites, each with its own goals, its own load and its own benefit. Crawl budget has turned from a Google-only question into a multi-bot governance problem. This article explains how crawl budget works technically, when it really becomes relevant in the DACH B2B market and what managing Googlebot, Bingbot and AI crawlers separately looks like in practice.

Why crawl budget is a multi-bot problem in 2026

Traditionally, crawl budget describes how many pages a search engine fetches from a website within a given period. As long as only Googlebot mattered, this concerned almost exclusively very large websites. In 2026 that assumption falls short, because the crawler landscape has broadened fundamentally.

The increase is measurable. According to Cloudflare's international analysis, AI bots accounted for 4.2% of all HTML requests on average in 2025 and Googlebot for 4.5%, which made Googlebot only slightly larger than all AI bots combined. User-driven AI crawling is especially dynamic: it grew more than 15-fold in 2025. This is the crawling in which a bot visits a page in real time in response to a chatbot question. Among pure AI crawlers, the weight shifted strongly toward OpenAI: GPTBot rose from a 5% to a 30% share of AI crawlers between May 2024 and May 2025, with request growth of 305%, while Bytespider fell from 42% to 7.2% over the same period.

For the Austrian B2B market, Googlebot remains the most important crawler, because Google clearly dominates: in May 2026 it held an 81.87% market share, ahead of Bing at 9.01% and DuckDuckGo at 2.75%. On top of that come the AI crawlers: they consume server resources, take up edge and CDN capacity and raise the question of which bot delivers which return. That is exactly what turns crawl budget into a management issue for all stakeholders, not just for Google.

Crawl capacity limit and crawl demand: when it becomes relevant

Google defines a website's crawl budget as the set of URLs that Google can and wants to crawl. It results from two components: the crawl capacity limit and crawl demand.

• Crawl capacity limit: The upper bound of what the server can handle without overload. If a website responds quickly and without errors, the limit rises. If it slows down or returns server errors, Google throttles crawling to spare the website.
• Crawl demand: The need to crawl a URL. It depends on a page's popularity and on how often its content changes. Outdated or unchanged pages are visited less often.

The key question is when any of this matters at all. Google is unambiguous here: its own guide is aimed at large websites with more than 1 million unique pages and content that changes weekly, and at medium and larger websites with more than 10,000 unique pages and content that changes very often, for example daily. For the typical DACH B2B website with a few hundred pages, crawl budget is therefore rarely the bottleneck from a pure Googlebot perspective.

This all-clear applies only to Googlebot, though, and only to classic crawling. As soon as faceted navigation, filter URLs or a large product catalog artificially inflate the number of URLs, even a mid-sized website can run into a budget problem. And AI crawlers follow their own logic, which has nothing to do with Google's thresholds.

Multi-bot governance: a granular robots.txt strategy

The first lever is robots.txt. In 2026 it is no longer a simple allow-or-block switch but a control table for different classes of bots. Internationally, robots.txt has long since become the arena of AI crawler management. According to the Web Almanac 2025, the most important AI user agents are already addressed there specifically: GPTBot in 4.5%, ClaudeBot in 3.6%, CCBot in 3.5% and Google-Extended in 3.4% of desktop robots.txt files, with GPTBot nearly doubling compared to 2024. By far the most common directive remains the wildcard: the catch-all user agent asterisk (*) appeared in 77% of files in 2025.

A well-thought-out strategy distinguishes bots by their purpose:

• Search engine crawlers (Googlebot, Bingbot): Full access. They deliver organic traffic and are the business foundation in the DACH region. You never block them across the board here; you only prevent crawl waste in a targeted way.
• AI search crawlers (OAI-SearchBot, PerplexityBot): Serve real-time answers with source attribution and potential referral traffic. If you want to be visible in AI answers, you usually allow them.
• AI training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent): Collect data for model training. A deliberate trade-off is needed here, because the resource consumption often brings no direct return.

A clean separation of user agents matters. Google-Extended is not a crawler of its own. According to Google, it has no separate user agent string and is purely a control token for robots.txt. Fetching is done by the usual Google crawlers. The token only determines whether Google may use the content for training Gemini models and for grounding. Grounding means that Gemini draws on content from the Google index as a source when it answers a query. The token has no effect on inclusion in Google Search or on ranking. If you block Google-Extended, you keep your Google visibility and only exclude the use for Gemini. This distinction is the core of multi-bot governance.

Distinguishing AI crawler load types: training, search, real time

Blanket blocking or allowing ignores that AI crawlers perform three entirely different tasks. Cloudflare's international data analysis quantifies the ratio clearly: training now drives nearly 80% of AI bot activity, 18% goes to search and only 2% to user-driven actions.

• Training crawling: Bots such as GPTBot, ClaudeBot or CCBot download content for model training. This access creates load but rarely leads to directly attributable traffic. It is the load type with the weakest immediate return.
• Search crawling: AI search systems retrieve content to cite it in answers. This offers a chance of referral traffic and brand visibility, comparable to classic search.
• User-driven real-time crawling: Bots such as ChatGPT-User visit a page live because a user has asked a question in the chat. This load type is small in volume, but it signals concrete user interest in exactly this content at the moment of the query.

The practical consequence: if you want to cut server costs without sacrificing visibility, throttle training crawling first and keep search and real-time crawling. A blanket block of all AI bots, by contrast, would also cut off the potentially valuable access.

Eliminating crawl budget waste

Before you think about bot management, get rid of the self-inflicted waste. In practice it is the biggest problem, and it affects all crawlers equally. An international Botify case study shows how dramatically the ratio can tip: there, 97% of around 1 million crawled pages were non-canonical, with only about 25,000 indexable URLs, of which Google crawled just over half per month. Across the industry, according to the same source, Google crawls only 40% of strategic URLs per month on average on unoptimized websites, while 60% remain uncrawled.

The typical sources of waste can be closed in a targeted way:

• Faceted navigation and parameter URLs: Filter and sort combinations generate an almost unlimited number of URLs with barely different content. You control them with canonical tags, targeted noindex or by blocking entire parameter paths in robots.txt.
• Duplicate content: Several URLs with identical content, for example due to upper and lower case, trailing slashes or session IDs. A consistent canonical strategy consolidates the signals onto one preferred URL.
• Soft 404s: Pages without real content that wrongly return a 200 status instead of 404 or 410. They tie up crawl budget for content without value.
• Endless crawl paths: Calendars, endless pagination or nested filters that lead crawlers ever deeper.

The principle: crawlers should spend their limited time on the pages that are meant to rank or be cited, not on technical clutter.

Active index control: sitemaps and IndexNow

Instead of only avoiding waste, you can actively steer crawlers to the right URLs. Two complementary instruments are available for this.

• XML sitemaps: They list the canonical, index-worthy URLs together with their last-modified date. A well-maintained sitemap helps search engines find new and changed pages efficiently without walking the entire website. It is the basis of any effort to reduce crawl load.
• IndexNow: A push protocol that lets a website actively notify search engines about new or changed URLs instead of waiting for the next crawl. Adoption is substantial: according to Bing, more than 3.5 billion URLs are submitted via IndexNow per day, and 18% of all newly clicked URLs in web search results come from it.

The decisive caveat for the DACH region: IndexNow is supported by Bing, Yandex and others, but not by Google. Since Google dominates the market in Austria, IndexNow is a sensible addition for Bing visibility there, but it does not replace a clean sitemap and crawl strategy for Google. The two instruments complement each other: sitemaps for Google, IndexNow in addition for the search engines that take part in IndexNow.

Bot management at the edge, CDN and server layer

robots.txt is a request, not an enforcement mechanism. Reputable crawlers respect it, others do not. If you want to enforce limits on crawl overload technically or reliably stop unwanted bots, you need control at the edge, CDN and server level.

• Verifying legitimate bots: Genuine crawlers can be verified via their officially published IP ranges or via reverse DNS. This is how you separate the real Googlebot from bots that only pretend to be it.
• Rate limiting: Limits the request frequency per bot or IP range. This protects the origin server from load spikes without locking out legitimate crawlers completely.
• Selective blocking and allowing: At the CDN level, you can block AI training bots specifically while letting search and real-time bots through. This is more precise and easier to enforce than robots.txt alone.
• Cache protection: AI crawlers often access rarely visited long-tail content that is not in the edge cache. Such requests go straight to the origin server and compete with human traffic for resources. Rate limiting and targeted routing take load off the server here.

This layer is especially relevant for resource-sensitive B2B websites, where server capacity and stable response times for paying customers take priority over uncontrolled bot access.

Log file analysis as a management tool

Without measurement, any crawl management is flying blind. Standard web analytics tools such as GA4 capture virtually no AI crawler traffic, because these crawlers do not execute JavaScript and do not set cookies. The only reliable data source is the server log file. It shows which bot fetched which URL when, and with which status code.

The decisive management metrics can be derived from the logs:

• Crawl frequency per bot: How often do Googlebot, GPTBot or ClaudeBot visit the website, and which URL sections? This tells you whether budget is flowing to worthless pages.
• Crawl-to-refer ratio: The ratio of crawled pages to visitors actually referred measures a bot's return. Cloudflare's international figures for July 2025 show the enormous spread: Anthropic crawled around 38,065 pages per referred visitor, OpenAI around 1,091 and Google only 5.4. The higher this value, the worse the ratio of load to benefit.
• Status code distribution: If 404s or 5xx errors pile up in bot traffic, this points to crawl waste or server problems that push down the crawl capacity limit.

With this data, you can redirect budget in a targeted way: throttle bots with a poor crawl-to-refer ratio and steer valuable crawlers to the strategic URLs. That makes log file analysis the central measurement tool of multi-bot governance.

Next.js and JavaScript SEO: factoring in the rendering budget

Crawl budget has a second, often overlooked dimension: the render budget. If you build content only on the client side with JavaScript, you demand a second, expensive step from the crawler. Google has to crawl and render the page to see the content. This consumes additional resources and delays indexing.

For a modern Next.js architecture, this results in a clear ranking of rendering strategies:

• SSG (Static Site Generation): Pages are pre-rendered at build time and delivered as finished HTML. The most crawl- and render-efficient approach, ideal for stable content such as in-depth articles or service pages.
• ISR (Incremental Static Regeneration): Static pages are regenerated periodically in the background. Combines the efficiency of SSG with freshness, good for content that changes often but not by the second.
• SSR (Server-Side Rendering): The HTML is generated on the server for each request. The crawler receives complete HTML without having to render it itself. Useful for personalized or highly dynamic pages.
• CSR (Client-Side Rendering): The content is only created in the browser. The least favorable case for SEO, because it puts a double burden on crawl and render budget and leaves AI crawlers in particular, which often do not execute JavaScript at all, with nothing.

The rule of thumb for crawl-efficient DACH B2B websites: serve index-relevant content server-side or statically, so that every crawler finds the complete content directly in the HTML. This saves render budget and also makes content accessible to JavaScript-blind AI crawlers.

AI crawlers, GEO and monetization: llms.txt and pay-per-crawl

The load from AI crawlers raises the strategic question of what you get in return. New mechanisms for this are emerging in 2026, but they are still young and inconsistent.

• llms.txt: A proposed file that offers AI systems curated, well-structured content in bundled form. Adoption is still low, though: according to the Web Almanac 2025, only 2.13% of desktop websites provide a valid llms.txt, 39.6% of which were generated by the All in One SEO plugin. It has not become a binding standard so far.
• Pay-per-crawl approaches: Models in which AI providers pay for access to content instead of crawling it for free. They address exactly the imbalance between high crawl load and low referral traffic that the crawl-to-refer ratios reveal.

At its core, it is a trade-off: visibility in AI answers (GEO) on one side, server costs and control over your own content on the other. For DACH B2B companies whose expert content is a central asset, a deliberate decision per crawler class pays off, instead of allowing or blocking everything indiscriminately.

GDPR, TDM reservation and the legal dimension

In the DACH region, dealing with AI crawlers also has a legal dimension. It is relevant for Austrian companies because it goes beyond mere technical control.

• Opt-out from AI training: With directives such as blocking Google-Extended or GPTBot in robots.txt, a website signals that its content should not be used for AI training. This is the technical implementation of a usage reservation.
• TDM reservation (text and data mining): EU copyright law allows rights holders to prohibit, in machine-readable form, the use of their works for commercial text and data mining. A reservation stored in robots.txt or as a machine-readable declaration is the basis for legally excluding AI training.
• GDPR aspect: If public pages contain personal data, their inclusion in training corpora raises data protection questions. Deliberate crawler control is therefore part of clean compliance.

The link between technical and legal control is the point here: robots.txt controls crawling and at the same time documents the rights holder's intent. For GDPR compliance in the Austrian context, this control should be deliberate and traceable, ideally coordinated with the legal department.

Further reading and conclusion

In 2026, crawl budget is no longer a topic only for large websites. It is a multi-bot problem that needs active management. From a pure Googlebot perspective the thresholds remain high, but the breadth of AI crawlers, their different load and their widely varying return make bot-by-bot governance worthwhile for mid-sized DACH B2B websites too.

The priorities in practice are clear: first, eliminate self-inflicted crawl waste through canonical, noindex and robots.txt hygiene; second, set up a robots.txt that treats search engines, AI search crawlers and AI training crawlers differently; third, analyze server logs to measure crawl frequency and crawl-to-refer ratios and to redirect budget; fourth, choose a rendering architecture that makes content accessible without rendering effort. If you combine these four levels and factor in the legal dimension, you keep control over your own server and decide deliberately which content is open to which bot. You can go deeper in the related articles on robots.txt, XML sitemaps, canonical tags and AI crawler management.