Googlebot and Crawling: How Google Searches the Web

Googlebot is what makes it possible for a website to be found on Google at all. Without successful crawling, there is no indexing, without an index no ranking, and without ranking no visibility. For B2B companies in the DACH region, this is more than a technical detail: if you don't show up in organic search, you are missing from your target audience's most important research channel. This article explains how Googlebot works, how you control its behavior, which errors prevent crawling, and how AI crawlers for ChatGPT and Perplexity, along with Google AI Overviews, are currently shifting the picture.

Why crawling determines your visibility

Crawling is the first step in a chain that ends with business success in search. Googlebot visits pages, reads their content, and follows links to discover new and updated URLs. Only then can Google index the content and serve it for relevant search queries. The sheer scale shows why you shouldn't leave this process to chance: according to Google, the Google index contains hundreds of billions of web pages and is well over 100,000,000 gigabytes in size. At that scale, every page competes for the crawler's limited attention.

The DACH market is also highly concentrated. In Austria, Google holds a search engine market share of 81.87 percent according to StatCounter, followed by Bing with 9.01 percent. The potential reach is broad: according to Statistik Austria, among the Austrian population aged 16 to 74, 95 percent were internet users and only 5 percent were non-users. If you want to be found online in Austria, you are therefore optimizing primarily for Googlebot. And crawling volume is growing: according to Cloudflare, Googlebot was the most active crawler on the web and grew by 96 percent from May 2024 to May 2025, with a peak increase of 145 percent in April 2025.

How Googlebot works: crawling, indexing, ranking

Googlebot is Google's automated web crawler. It comes in two main variants: Googlebot Smartphone, which fetches the mobile view, and Googlebot Desktop. Under mobile-first indexing, Google today primarily crawls the mobile version of a page. If you deliver content or structured data only in the desktop version, you risk it not being captured. You can verify genuine Googlebot requests: run a reverse DNS lookup on the requesting IP, check whether the hostname ends in googlebot.com or google.com, and then confirm it with a forward DNS lookup. This exposes fake user agents posing as Googlebot.

Google Search works in three interlocking phases:

• Crawling: Googlebot discovers URLs through links, sitemaps, and pages it already knows, and fetches their content. For modern pages, Google then renders the content like a browser to capture JavaScript-generated content.
• Indexing: Google analyzes text, images, and key tags, detects duplicates and canonicals, and stores the page in the index. Only indexed pages can rank at all.
• Ranking: For a search query, Google selects the most relevant results from the index and orders them for the SERP. Crawling and indexing are therefore prerequisites, while ranking determines the position.

How Googlebot finds URLs and how you control it

Googlebot discovers new pages mainly through links. Every crawled page yields new URLs for it to follow. This gives you three key levers to actively influence discoverability:

• Internal linking: A flat, logical link structure ensures that important pages are reachable from the homepage in a few clicks. Orphan pages without internal links are found poorly or not at all.
• XML sitemap: The sitemap lists all relevant URLs and submits them directly via Google Search Console. It does not replace good internal linking, but it helps especially on large or poorly linked sites.
• robots.txt: This file in the root directory controls which areas a crawler may fetch. It is the most important tool for controlling crawl access, but it is not a tool for deindexing.

robots.txt deserves particular care, because mistakes here have far-reaching effects. According to the Web Almanac 2024, 83.9 percent of robots.txt files returned a 200 status on mobile requests, while 14.1 percent responded with a 404. A missing or faulty file is therefore not unusual. What matters is keeping the tools cleanly apart: robots.txt blocks crawling, but it does not prevent indexing of URLs that are already known.

Understanding and optimizing crawl budget

Crawl budget is the number of URLs that Google can and wants to crawl on a site. Google defines it as the interplay of two factors: the crawl capacity limit and crawl demand. The capacity limit is the maximum number of simultaneous connections plus the delay between fetches. If a server responds quickly, the limit goes up; with server errors or slow responses, it goes down. Crawl demand depends on the perceived inventory of pages, the popularity of the URLs, and their freshness.

For most smaller websites, crawl budget is not a limiting factor. According to Google, it matters mainly for large sites with more than one million unique pages whose content changes weekly, or medium to large sites with 10,000 or more pages and very rapidly changing content. Google describes these figures as rough estimates, not fixed thresholds. For affected sites, the key is not to waste budget: improve server performance, eliminate soft 404s and infinite loops, consolidate duplicates, and exclude unimportant URL parameters. Every URL crawled unnecessarily comes at the expense of important pages.

Controlling crawling and avoiding errors

The most common cause of crawling problems is confusing the control tools. Three mechanisms serve three different purposes and must not be mixed up:

• robots.txt: controls whether a URL may be crawled. A page blocked by robots.txt can still end up in the index if other pages link to it, in which case it appears without content, only as a URL.
• noindex: a meta robots tag or HTTP header that prevents indexing. For Google to read the tag, the page must not also be blocked by robots.txt. According to the Web Almanac 2024, 4.7 percent of desktop pages and 3.9 percent of mobile pages use noindex.
• canonical: signals the preferred version among several similar URLs and consolidates ranking signals instead of excluding content entirely.

Typical crawl traps result from technical negligence. Faceted navigation, filters, and session IDs generate an endless number of URL variants that eat up the budget. Duplicate content spreads signals across competing URLs. Soft 404s, meaning error pages that return status 200 instead of 404, pretend to be valid content and keep getting crawled. If you spot these patterns early and resolve them cleanly with status codes, canonical, and noindex, you keep crawling efficiency high.

Rendering and JavaScript SEO

Modern websites often deliver content only via JavaScript. Googlebot can generally handle this, because it renders pages in a second step with the Web Rendering Service (WRS), executing the code like a browser. This rendering is resource-intensive, though, and can be delayed. Content that is central to understanding the page should therefore be rendered server-side or delivered prerendered so that it is captured reliably and quickly.

Page size and performance have a direct effect on crawling. The heavier and slower a page, the more resources each fetch costs and the fewer pages Googlebot gets through in the same time window. The trend points the wrong way: according to the Web Almanac 2024, the median mobile page weight was 2,311 KB, an increase of 6.4 percent or 140 KB over the previous year. Lean code, optimized images, and a fast server response therefore do more than improve speed: they directly optimize crawling.

Crawl diagnostics with tools and log files

You can measure crawling instead of guessing at it. Two data sources are essential for this. The first is Google Search Console, which gives you a free inside view of how your own domain is crawled:

• Crawl stats: show the number of requests, average response times, and response codes over time, and reveal server problems.
• URL Inspection: tells you for a single URL whether it was crawled and indexed, which canonical Google selected, and what the rendered page looks like.
• Index coverage (Page indexing): lists which pages are indexed and why others were excluded, for example due to noindex, duplicates, or crawl errors.

The second source is server log file analysis. Log files record every real Googlebot request and therefore show which URLs are actually crawled and how often, where budget is wasted, and which important pages the crawler neglects. Combined with reverse DNS verification, you can cleanly separate genuine Googlebot requests from fake ones. For large sites, log file analysis is the most precise crawl diagnostic tool available.

From Googlebot to AI crawlers: GEO in DACH B2B

Alongside Googlebot, a new generation of bots crawls the web for AI systems. GPTBot collects content for OpenAI and ChatGPT, ClaudeBot for Anthropic's Claude, and PerplexityBot for the answer engine Perplexity. One key technical difference: these AI crawlers usually do not execute JavaScript. Content that is rendered only on the client side remains invisible to them, while Googlebot can still capture it via the WRS. Google-Extended is a token in robots.txt, not a separate crawler. It only controls whether Google may use content it has already crawled for training and grounding Gemini, and it has no effect on Google Search. There is no separate crawler for AI Overviews either: Google only links to pages there that are in the index and eligible to be shown with a snippet.

This shift has tangible consequences, because AI answers are increasingly taking over search behavior. According to a Semrush study, AI Overviews in the US were triggered for 6.49 percent of queries in January 2025, 24.61 percent in July, and 15.69 percent in November. Generative Engine Optimization (GEO) aims to get your content cited in these AI answers. An international study shows that targeted GEO measures can increase visibility in generative search answers by up to 40 percent.

For B2B companies in the DACH region, this creates a twofold task. First, a deliberate decision in robots.txt about which AI crawlers get access, because access secures visibility but costs server capacity and control over your content. Second, delivering core content without JavaScript dependency and with a clear, extractable structure, so that both Googlebot and AI crawlers can process it.

Further reading

Crawling is the invisible foundation of every search strategy. If you understand Googlebot, you take care of clean technical prerequisites first: an error-free robots.txt, an up-to-date XML sitemap, flat internal linking, and fast, lean pages. The next steps build on that. Useful follow-up topics are indexing and how to control it with noindex and canonical, how ranking factors work, optimizing your Google Search Console setup for ongoing diagnostics, and Generative Engine Optimization as a discipline in its own right for visibility in ChatGPT, Perplexity, and Google AI Overviews. Together they form the foundation on which organic and AI-driven visibility in the DACH region is built. If you don't want to keep checking the technical foundation yourself, you can have it handled as part of ongoing SEO support. GEO support covers visibility in AI answers.