Skip to content
1.3Intermediate9 min

Googlebot and Crawling: How Google Searches the Web

Lucas Blochberger··Updated 10 June 2026
Definition

Googlebot is Google's automated web crawler that systematically visits web pages, reads their content, and follows links to discover and process new and updated pages for the Google index.

Key Takeaways

  • ✓Googlebot is Google's primary crawler for desktop and mobile and the first step before indexing and ranking
  • ✓Mobile-First Indexing means: Googlebot primarily crawls the mobile version, content only present there must be delivered on mobile
  • ✓According to Cloudflare, Googlebot was the most active crawler on the web and increased by 96 percent from May 2024 to May 2025
  • ✓robots.txt controls crawl access for different crawler types but does not prevent indexing; noindex and canonical are responsible for that
  • ✓Crawl budget from crawl capacity limit and crawl demand is primarily relevant for very large or rapidly changing sites
  • ✓AI crawlers like GPTBot, ClaudeBot and PerplexityBot typically do not execute JavaScript, unlike Googlebot with its Web Rendering Service
  • ✓Genuine Googlebot requests can be distinguished from spoofed user agents via reverse DNS verification

Crawling is the first step in the search engine process. Without crawling, there's no indexing; without indexing, no ranking. In 2026, crawling is more complex than ever, as numerous AI crawlers are searching the web alongside Googlebot.

How Googlebot Works

Googlebot is an automated program that visits web pages and reads their content. It starts with known URLs, downloads the HTML code, extracts all links, and adds them to a crawling queue. This process repeats billions of times.

Googlebot exists in two variants: Desktop and Mobile. Since the complete transition to Mobile-First Indexing in 2024, the Mobile crawler is the primary version. Google evaluates websites primarily based on their mobile presentation.

Understanding Crawl Budget

Crawl budget describes how many pages Googlebot crawls on a website within a specific timeframe. It's determined by two factors: crawl rate (how quickly the server responds) and crawl demand (how important and current the content is).

For small websites with fewer than 1,000 pages, crawl budget is rarely a problem. For large websites with tens of thousands of pages, inefficient crawl budget usage can result in important pages not being indexed.

AI Crawlers: The New Dimension

Alongside Googlebot, numerous AI crawlers are searching the web in 2025/2026. GPTBot traffic grew by 305 percent compared to the previous year. Overall, crawler traffic increased by 18 percent. The most important AI crawlers are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended.

A fundamental difference: AI crawlers typically don't execute JavaScript. Websites whose content is only rendered client-side are invisible to AI systems. Server-Side Rendering (SSR) is therefore mandatory.

robots.txt for Crawler Control

The robots.txt file controls which crawlers are allowed to visit which areas of the website. For SEO and GEO, a differentiated strategy is recommended: allow Googlebot and search crawlers unrestricted access, also allow AI search crawlers (OAI-SearchBot, PerplexityBot), and handle training-only crawlers (GPTBot, Google-Extended) optionally based on strategic consideration.

Important: The ChatGPT user crawler has ignored robots.txt since December 2025. Complete blocking of OpenAI crawlers is therefore no longer possible.

Data & Statistics

Der Google-Index umfasst hunderte Milliarden Webseiten und ist weit über 100.000.000 Gigabyte groß

Google - How Search Works (Organizing Information) (2025)

Google hält in Österreich einen Suchmaschinen-Marktanteil von 81,87 Prozent, Bing folgt mit 9,01 Prozent

StatCounter Global Stats - Search Engine Market Share Austria (2026)

In der österreichischen Bevölkerung zwischen 16 und 74 Jahren stehen 95 Prozent Internetnutzer:innen (Onliner) 5 Prozent Nicht-Nutzer:innen (Offliner) gegenüber

Statistik Austria - Nichtnutzung des Internets 2023 (Pressemitteilung 19.09.2024) (2023)

Googlebot war der aktivste Crawler im Web und wuchs von Mai 2024 bis Mai 2025 um 96 Prozent, mit einem Spitzenwert von plus 145 Prozent im April 2025

Cloudflare Blog - From Googlebot to GPTBot: who's crawling your site in 2025 (2025)

83,9 Prozent der robots.txt-Dateien lieferten bei mobilen Abrufen einen Status 200 zurück, 14,1 Prozent antworteten mit einem 404 (2024, mobil)

Web Almanac 2024 (HTTP Archive) - SEO-Kapitel (2024)

Crawl-Budget definiert Google als Zusammenspiel aus Crawl-Kapazitätslimit und Crawl-Demand (set of URLs that Google can and wants to crawl)

Google Search Central - Crawl Budget Management (2025)

Crawl-Budget-Optimierung ist vor allem relevant fuer grosse Sites mit ueber 1.000.000 einzigartigen Seiten (woechentlich wechselnder Inhalt) oder mittlere bis grosse Sites ab 10.000 Seiten mit sehr schnell wechselndem Inhalt

Google Search Central - Crawl Budget Management for Large Sites (2025)

4,7 Prozent der Desktop-Seiten und 3,9 Prozent der mobilen Seiten nutzen ein noindex

Web Almanac 2024 (HTTP Archive) - SEO-Kapitel (Robots directive rules) (2024)

Das mediane Seitengewicht im mobilen Bereich lag bei 2.311 KB, ein Plus von 6,4 Prozent oder 140 KB gegenueber dem Vorjahr

Web Almanac 2024 (HTTP Archive) - Page Weight (2024)

AI Overviews wurden bei 6,49 Prozent der Anfragen im Januar 2025 ausgeloest, im Juli bei 24,61 Prozent und im November bei 15,69 Prozent (US-Daten)

Semrush Blog - Semrush AI Overviews Study 2025 (2025)

Gezielte GEO-Massnahmen koennen die Sichtbarkeit in generativen Suchantworten um bis zu 40 Prozent steigern (internationale Studie)

arXiv - GEO: Generative Engine Optimization (KDD 2024) (2024)

FAQ

What is Googlebot?
Googlebot is Google's automated web crawler. It systematically visits websites, reads their content and follows links to discover new and updated pages for the Google index. It primarily exists as Googlebot Smartphone, which retrieves the mobile view, and as Googlebot Desktop. As part of Mobile-First Indexing, Google today primarily crawls the mobile version of a page.
How do crawling, indexing and ranking differ?
They are the three consecutive phases of Google Search. During crawling, Googlebot discovers and retrieves URLs. During indexing, Google analyzes and stores the content in the index; only indexed pages can rank. During ranking, Google selects the most relevant results from the index for a search query and sorts them for the SERP. Crawling and indexing are therefore prerequisites, while ranking determines the position.
How does Googlebot find new pages?
Googlebot discovers new URLs predominantly through links on already known pages. Three levers influence discoverability: a flat, logical internal linking structure so that important pages are reachable with few clicks; an XML sitemap that directly reports relevant URLs via Google Search Console; and a correct robots.txt that controls crawl access. Orphaned pages without internal links are poorly or not discovered at all.
What is crawl budget and who is it important for?
Crawl budget refers to the number of URLs that Google can and wants to crawl on a site. Google defines it as the interplay of crawl capacity limit (simultaneous connections plus wait time, depending on server performance) and crawl demand (page inventory, popularity, freshness). For smaller websites it is rarely a bottleneck. According to Google, it becomes relevant primarily for very large sites from about one million pages or medium to large sites from 10,000 pages with very rapidly changing content.
What is the difference between robots.txt, noindex and canonical?
The three mechanisms serve different purposes. The robots.txt controls whether a URL may be crawled, but does not prevent indexing: a blocked page can still appear as a URL in the index via links. The noindex tag prevents indexing, but the page must not be blocked by robots.txt at the same time, otherwise Google will never read the tag. The canonical tag determines the preferred version among multiple similar URLs and consolidates their ranking signals.
How do I verify if a request really comes from Googlebot?
Do not rely solely on the user agent, as it can be spoofed. Perform a reverse DNS lookup of the requesting IP address and check whether the hostname ends in googlebot.com or google.com. Then confirm it via forward DNS lookup, which must point back to the same IP. In server log files, genuine Googlebot requests can thus be reliably separated from bots that only pretend to be Googlebot.
How do AI crawlers like GPTBot and ClaudeBot differ from Googlebot?
AI crawlers like GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic) and PerplexityBot collect content for AI systems and answer engines. A key technical difference: they typically do not execute JavaScript, while Googlebot renders pages via its Web Rendering Service. Purely client-side rendered content therefore remains invisible to AI crawlers. Via robots.txt and the Google-Extended token, you can control which of these crawlers receive access.

Related Articles

How does your website perform?

Get a free, AI-powered SEO report of your website by email – technical SEO, on-page, keywords & competitors. No obligation.

Get a free SEO audit →
Previous← How do search engines work? Google, Bing and Co.NextIndexing: How Google Stores and Understands Pages →