Indexing: How Google Stores and Understands Pages

Most conversations about search engine optimization revolve around rankings. But before a page can sit in position 1 or position 50, it has to clear one hurdle: it has to be in the Google index. Without indexing there is no ranking, no clicks, and no visibility. For B2B companies in the DACH region, this is no technical footnote. It is the basic prerequisite for a website to generate any business from organic search at all.

Why indexing is critical for B2B visibility

In Austria, there is practically no way around Google. In September 2026, its market share stood at 80.84 percent, far ahead of Bing at 10.93 percent and DuckDuckGo at 2.38 percent. If you want to build organic visibility here, you are in effect optimizing for a single search engine. That leaves practically no alternative to Google indexing: a page that is not in the Google index does not exist for most of the Austrian search market.

The reach is considerable. At the start of 2025, Austria had 8.69 million internet users, a penetration rate of 95.3 percent. These people research vendors, solutions, and technical terms on Google. In B2B, where buying decisions take a long time and rest on prior research, findability in search helps decide whether a company makes the shortlist at all.

Set the right expectation: indexing is the entry ticket, not the prize. It makes sure a page can appear as a search result at all. Whether it then gets visitors is decided by ranking. An analysis of around 14 billion pages shows how big that gap is: 96.55 percent of all pages get zero organic traffic from Google, and another 1.94 percent get only one to ten visits a month (international study, Ahrefs). Being indexed is necessary, but far from sufficient.

How Google indexes pages: crawling, indexing, ranking

Google Search consists of three clearly separate phases. Google itself stresses this separation and states plainly: Google doesn't guarantee that it will crawl, index, or serve your page. Each phase is a hurdle of its own.

Crawling: First, Google has to learn that a URL exists. Because there is no central registry of all web pages, Google constantly looks for new and updated pages, a process called URL discovery. Google finds new URLs through internal links, sitemaps, or external references. Then Googlebot, the program that does the fetching, retrieves billions of pages from the web. Crawling only means that Google has visited the page and loaded its code.

Indexing: Only then does the actual understanding begin. After a page is crawled, Google tries to understand what the page is about. This stage is called indexing, and it includes processing and analyzing the textual content and key content tags and attributes. Google renders the page the way a browser does, which means it also runs JavaScript to see what users actually get to see.

It analyzes text, images, and headings and recognizes entities, meaning people, places, products, or concepts, and the relationships between them. On this semantic basis, Google assigns the page to topics and possible search queries and stores the result in the Google index, a huge database. Just how big it is came to light through testimony in the US antitrust case against Google: the index comprised around 400 billion documents (as of 2020, according to Google Vice President Pandu Nayak) (US source).

Ranking: Only once a page is in the index can it be retrieved for a search query, evaluated for relevance and quality, and sorted in the search results (SERP). This is where ranking factors and quality signals such as E-E-A-T come into play.

The key point: indexing isn't guaranteed, and not every page that Google processes will be indexed. A page can be crawled and still never end up in the index.

Controlling indexing: practical best practices

Google does not index everything automatically, and not everything right away. With the right tools, you control which pages get into the index and speed up the process.

Use Google Search Console: Search Console is your main control panel. The URL Inspection tool shows for every single URL whether it is indexed and, if not, why. The page indexing report (Pages) lists all URLs by status and names concrete reasons for exclusions. With the Request indexing feature, you submit new or updated pages directly for review.

Submit an XML sitemap: The sitemap is your list of recommendations to Google. It lists all important URLs and, especially on large or poorly linked websites, helps Google discover new pages faster. Submit the sitemap in Search Console and keep it up to date.

Use robots.txt deliberately: robots.txt controls crawling, not indexing. In exceptional cases, a page blocked by robots.txt can still end up in the index, for example when many external links point to it, but Google then cannot read its content. If you want to reliably keep a page out of the index, do not block it in robots.txt. Set a noindex tag instead.

Set meta robots and canonical tags correctly: With the noindex tag, you deliberately exclude individual pages from the index, such as internal search results pages, thank-you pages, or thin filter pages. When several similar URLs exist, the canonical tag tells Google the preferred version and consolidates the signals on one page. Both tags are standard on the web: according to the Web Almanac 2024, 69 percent of desktop pages use canonical tags and 45.5 percent use meta robots tags, with 4.7 percent setting a noindex directive (international data, HTTP Archive).

Maintain internal linking and technical hygiene: Strong internal linking leads Googlebot to new pages and signals their importance. Orphan pages without internal links are often never found in the first place. Then there are hygiene factors: HTTPS, a clean site structure, and solid Core Web Vitals make crawling and indexing easier, even though they do not force indexing.

Common mistakes: why pages don't get indexed

Many indexing problems are self-inflicted. These causes come up most often in practice.

Duplicate content: Identical or very similar content under several URLs is one of the most common blockers. Google then usually picks just one version and ignores the rest. In Search Console, these cases appear as “Duplicate, Google chose different canonical than user” or as “Duplicate without user-selected canonical.” The fix: consistent canonical tags and avoiding unnecessary URL variants.

Crawled, currently not indexed: This message means that Google visited the page but deliberately did not add it to the index. A common cause is poor quality or a lack of added value: to Google, the page simply isn't worth a spot in the index. More substantial content, a clear search intent, and better internal linking help.

Discovered, currently not indexed: Here Google knows the URL but has not crawled it yet. This often points to a crawl budget issue on very large sites or to doubts about quality. Crawl budget only matters for a few sites, though: Google describes it mainly as an issue for large sites with one million or more unique pages, or sites with 10,000 or more pages whose content changes very frequently. For typical B2B websites, crawl budget is rarely the real problem.

Soft 404: A page technically returns status code 200 (OK), but to Google its content looks like an error page, for example empty categories or placeholders without content. Such pages do not get indexed. Give real error pages the correct 404 status code and fill useful pages with real content.

Accidental blocks: A noindex tag from the staging environment that was forgotten after a relaunch, or a faulty robots.txt, can knock entire sections out of the index. After every relaunch, systematically check meta robots, robots.txt, and canonical tags.

Measuring and monitoring indexing

Indexing is an ongoing state, not a one-time status. These metrics and tools keep it in view.

Index coverage in Search Console: The Pages report is the most important source. Watch the ratio of indexed to non-indexed URLs and, above all, how it changes over time. A sudden drop in indexed pages is a warning sign and calls for an immediate root cause analysis.

URL inspection for individual pages: For individual, strategically important pages, the URL Inspection tool shows the exact status, the canonical Google selected, and any problems. Use it whenever a key page simply refuses to rank.

Spot check with a site: query: A quick, rough check is a Google search for site:yourdomain.com. It shows approximately which pages are in the index. Search Console remains the more reliable source, though, because the site: query is only an estimate.

Put indexing in context with ranking signals: Indexed pages without traffic are normal, as the 96.55 percent without traffic cited above show. So never measure indexing in isolation; always look at it together with rankings, impressions, and clicks. A page that is indexed but earns no impressions has a relevance or quality problem, not an indexing problem.

GEO and AI search: the new layer of indexing

AI-powered search gives indexing additional weight. Google increasingly shows AI Overviews, AI-generated answer boxes above the classic results. According to an analysis of more than 10 million keywords, AI Overviews appeared for 6.49 percent of queries in January 2025, peaked at 24.61 percent in July, and settled at around 16 percent in November (international study, Semrush). These answers draw on indexed content. What is not in the index cannot be cited in an AI Overview.

At the same time, generative engines are establishing themselves as a research channel of their own. According to OpenAI, ChatGPT reached 800 million weekly active users (US source). This is where generative engine optimization (GEO) comes in: optimizing content so it gets surfaced in AI answers. A scientific study shows that GEO can increase visibility in generative engine responses by up to 40 percent (research paper, KDD 2024).

In practice, the basic logic stays the same: indexability and technical cleanliness are the foundation. Content has to be retrievable, unambiguous, and well structured so that classic search, AI Overviews, and generative engines can capture it at all. If you get sloppy here, you fail in every channel at once.

Practical checklist: getting a new B2B page indexed

These steps reliably get a new page into the index and keep it there.

1. Check indexability: Make sure no noindex tag is set and robots.txt does not block the page. Check the canonical tag; it should point to the page itself.

2. Link internally: Link to the new page from relevant pages that are already indexed. That way Googlebot finds it quickly and recognizes its importance.

3. Update the sitemap: Add the URL to the XML sitemap and submit the sitemap in Search Console.

4. Request indexing: Inspect the URL in the URL Inspection tool and click Request indexing. This triggers a prioritized review.

5. Ensure quality: Offer substantial, original content with a clear search intent. This is the most effective lever against the “Crawled, currently not indexed” status.

6. Monitor continuously: Check the page indexing report regularly, react to drops, and after every relaunch systematically check robots directives and canonicals.

Further reading

Indexing is at the core of how search engines work, which ties it closely to the other SEO fundamentals. Once you understand indexing, look next at crawl control, at ranking factors and quality signals such as E-E-A-T, and at the difference between SEO and SEA (search engine advertising). For technically complex or large B2B websites, it is also worth taking a deeper look at crawl budget management and log file analysis to see what Googlebot actually fetches. With the rise of AI Overviews and generative engines, GEO joins as a complementary discipline that builds on the same indexable technical foundation.