Skip to content
2.7Advanced9 min

Crawl Budget 2026: Multi-Bot Governance for AI Crawlers

Lucas Blochberger··Updated 11 June 2026
Definition

Crawl budget has become a multi-stakeholder bot management challenge in 2026. ChatGPT-User makes 3.6x more requests than Googlebot (133,000+ per site in 55 days). AI crawlers access long-tail content with 70-100% unique access ratio, disrupting CDN caching for human traffic.

Key Takeaways

  • ✓ChatGPT user makes 3.6x more requests than Googlebot
  • ✓133,000+ requests per site in 55 days (ChatGPT user)
  • ✓AI crawler requests have only 11ms average response time
  • ✓AI crawlers access long-tail content, with a 70-100% unique access ratio
  • ✓This disrupts CDN edge cache that human traffic needs
  • ✓1 AI bot visit per 31 human visits (Q4 2025)
  • ✓Log file analysis is non-negotiable, since GA4 is blind to AI crawlers

The days of only managing a crawler relationship with Google are over.

The Volume Problem

An Alli-AI study (January–March 2026, 24.4 million proxy requests across 69 websites) found that ChatGPT users make 3.6x more requests than Googlebot, namely 133,000+ requests per site in 55 days. By Q4 2025, there was 1 AI-bot visit per 31 human visits (versus 1:200 in Q1 2025).

The Caching Problem

A Cloudflare/ETH Zurich paper (ACM Symposium on Cloud Computing 2025) shows that AI crawlers fundamentally disrupt CDN caching. They access long-tail content that humans rarely visit and maintain a 70–100 percent unique-access ratio per RAG-retrieval loop. This displaces edge cache and forces operators to choose: optimization for AI or for human visitors.

Monitoring

GA4 is completely blind to AI crawlers. Log file analysis is non-negotiable. Enterprise tools: Botify LogAnalyzer, Semrush Enterprise Bot Analytics (30+ bot coverage), JetOctopus. For mid-market: Screaming Frog Log Analyzer.

Data & Statistics

Google definiert Crawl Budget aus Crawl Capacity Limit und Crawl Demand; der Leitfaden richtet sich an Websites mit ueber 1 Mio. einzigartigen Seiten (woechentliche Aenderung) bzw. ueber 10.000 (taegliche Aenderung).

Google Search Central - Large Site Owner's Guide to Managing Crawl Budget (2024)

Innerhalb der reinen KI-Crawler stieg GPTBot von Mai 2024 bis Mai 2025 von 5 auf 30 Prozent Anteil (+305 % Requests); Bytespider fiel von 42 auf 7,2 Prozent.

Cloudflare Blog - From Googlebot to GPTBot: who's crawling your site in 2025 (2025)

KI-Bots machten 2025 im Jahresdurchschnitt 4,2 Prozent aller HTML-Requests aus, der Googlebot 4,5 Prozent (knapp groesser als alle KI-Bots zusammen); Menschen 47 %, nicht-KI-Bots 44 % (Jahresende).

Cloudflare Blog - The 2025 Cloudflare Radar Year in Review (2025)

Nutzergetriebenes KI-Crawling (AI 'user action' crawling, z. B. ChatGPT-User) wuchs 2025 um mehr als das 15-Fache.

Cloudflare Blog - The 2025 Cloudflare Radar Year in Review (2025)

Crawl-to-Refer Juli 2025: Anthropic rund 38.065 gecrawlte Seiten pro vermitteltem Besucher, OpenAI rund 1.091, Google 5,4; rund 80 % des KI-Crawlings dienen dem Training, 18 % der Suche, 2 % Nutzeraktionen.

Cloudflare Blog - The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (2025)

AI-Crawler in robots.txt (Desktop 2025): GPTBot 4,5 %, ClaudeBot 3,6 %, CCBot 3,5 %, Google-Extended 3,4 %; der Catch-all-User-Agent (*) erschien in 77 % der Dateien.

Web Almanac 2025 (HTTP Archive) - SEO Chapter (2025)

Nur 2,13 Prozent der Desktop-Websites stellen 2025 eine gueltige llms.txt bereit; 39,6 Prozent dieser Dateien sind dem All-in-One-SEO-Plugin zuzuordnen.

Web Almanac 2025 (HTTP Archive) - SEO Chapter (2025)

In einer Fallstudie waren 97 Prozent von rund 1 Mio. gecrawlten Seiten nicht-kanonisch (nur ~25.000 indexierbar); branchenweit werden bei unoptimierten Websites nur 40 Prozent der strategischen URLs pro Monat gecrawlt.

Botify - All About Crawl Budget Optimization (2020)

Ueber IndexNow werden mehr als 3,5 Milliarden URLs pro Tag eingereicht; 18 Prozent aller neu in den Web-Suchergebnissen geklickten URLs stammen daraus (Dezember 2024).

Bing Webmaster Blog - IndexNow Expands Adoption Across Industries (2024)

Suchmaschinen-Marktanteil in Oesterreich (Mai 2026): Google 81,87 Prozent, Bing 9,01 Prozent, DuckDuckGo 2,75 Prozent.

StatCounter Global Stats - Search Engine Market Share Austria (2026)

FAQ

What is Crawl Budget in simple terms?
Crawl Budget is the number of URLs a search engine can and wants to crawl on a website. Google determines it from two factors: the Crawl Capacity Limit, i.e. how much load the server can handle, and Crawl Demand, i.e. how strong the need is to crawl a page due to its popularity and freshness. In 2026, in addition to Googlebot, numerous AI crawlers consume server resources, which is why Crawl Budget has become a multi-bot governance topic.
At what page count does Crawl Budget become relevant?
From a pure Googlebot perspective, Google's Crawl Budget guidelines target large websites with over 1 million unique pages with weekly changing content, as well as medium to larger websites with over 10,000 unique pages with very frequent, e.g. daily, changing content. For the typical DACH B2B website with a few hundred pages, Crawl Budget is rarely the bottleneck from Google's perspective. However, once faceted navigation or parameter URLs inflate the URL count or numerous AI crawlers access the site, governance becomes sensible even for smaller websites.
How do I control AI crawlers like GPTBot and ClaudeBot?
The first lever is robots.txt, where individual AI crawlers can be specifically allowed or blocked by user agent. Internationally, GPTBot (4.5%), ClaudeBot (3.6%) and Google-Extended (3.4%) are already frequently addressed. A distinction by purpose makes sense: allow AI search crawlers with referral potential, consciously evaluate AI training crawlers. Since robots.txt is only a request, hard limits are additionally enforced at edge or CDN level, for example via rate limiting and bot verification.
What is the difference between training, search and real-time crawling?
Training crawling (e.g. GPTBot, ClaudeBot, CCBot) collects data for model training and generates load without direct referral traffic; according to international Cloudflare analysis, it accounts for around 80 percent of AI bot activity. Search crawling (18%) retrieves content to cite it in AI answers and can bring referral traffic. User-driven real-time crawling (2%, e.g. ChatGPT-User) visits a page live in response to a specific user question. Those wanting to reduce costs should first throttle training crawling.
Why is GA4 not sufficient for analyzing AI crawlers?
GA4 and other JavaScript-based web analytics practically do not capture AI crawlers because they typically do not execute JavaScript and do not set cookies. The only reliable data source is the server log file. It shows which bot retrieved which URL when with which status code. From this, crawl frequency per bot, the crawl-to-refer value and the status code distribution can be derived, the central governance metrics of multi-bot governance.
Does Google support IndexNow?
No. IndexNow is supported by Bing, Yandex and other search engines, but not by Google. According to Bing, more than 3.5 billion URLs per day are submitted via the protocol. Since Google dominates in Austria with 81.87 percent market share, IndexNow is a sensible addition for Bing visibility here, but does not replace a clean XML sitemap and crawl strategy for Google. Both instruments work complementarily.
How does client-side rendering affect Crawl Budget?
Client-side rendering (CSR) burdens crawl and render budget doubly because the search engine must not only crawl the page but also render it to see the content. This consumes additional resources and delays indexing. AI crawlers, which often do not execute JavaScript at all, then do not see the content at all. In a Next.js architecture, static (SSG) or server-side rendered (SSR) content is significantly more efficient because every crawler finds the complete content directly in the HTML.

Related Articles

How does your website perform?

Get a free, AI-powered SEO report of your website by email: technical SEO, on-page, keywords & competitors. No obligation.

Get a free SEO audit →
Previous← Next.js 15 SEO Architecture: SSR, PPR and Server ComponentsNextAI Crawler Management: Compliance Crisis and Enterprise Strategy →