XML Sitemaps: Best Practices for Large Websites

A broken or outdated XML sitemap costs small websites very little visibility. On large sites with tens of thousands or millions of URLs, it decides which content makes it into the index at all. If Google doesn't discover content, that content doesn't rank, and it doesn't show up in AI answers either, which increasingly draw on indexed sources. This article shows how to structure sitemaps for large websites correctly, generate them in Next.js, monitor them, and protect them against the most common errors.

Why XML sitemaps are business-critical for large websites

An XML sitemap is a machine-readable list of a website's indexable URLs. It complements internal linking rather than replacing it: search engines use the sitemap to discover new and changed content faster, especially on deeply nested or weakly linked pages. For small sites with a few hundred pages, the sitemap is a convenience feature. Beyond a certain size, it becomes your main tool for steering crawling.

Google names two concrete thresholds at which crawl budget becomes an issue: large sites with more than one million unique pages and content that changes weekly, and medium to large sites with more than 10,000 pages and content that changes very rapidly every day. Below these thresholds, Google usually crawls all relevant pages anyway. Above them, the sitemap becomes the most important lever for signaling to the crawler which URLs have priority and when they changed.

In the DACH region, this mainly affects larger online stores, marketplaces, publishers, and job boards. In Austria, 31 percent of companies already made sales via e-commerce in 2023, up from 26 percent in 2022. With growing product catalogs, filter pages, and multilingual country versions, the number of URLs quickly reaches a scale where a clean sitemap architecture determines the indexing rate.

Sitemap basics, limits, and index architecture

Every individual sitemap file is subject to hard technical limits. A file may contain no more than 50,000 URLs and must not exceed 50 MB uncompressed. If you hit either limit, you have to split the sitemap and combine the parts through a sitemap index file. The file must be UTF-8 encoded and use the required tags correctly.

At its core, a valid sitemap consists of these elements:

• urlset: The root element with the sitemap namespace that wraps all URL entries.
• loc: The full, absolute URL of the page, including the protocol. Required for every entry.
• lastmod: The timestamp of the last content change in W3C Datetime format. Optional, but valuable as a crawl signal.

For large sites, the sitemap index file comes into play: a sitemap of sitemaps. Limits apply here too: a sitemap index file may contain up to 50,000 loc tags, and you can submit up to 500 sitemap index files per property in Search Console. Mathematically, this tiering covers catalogs with billions of URLs, which makes it sufficient for practically any real website.

The decisive strategic question for large sites is sharding, meaning the logic you use to distribute URLs across individual sitemap files. Three approaches work well in practice:

• By page type: Separate sitemaps for products, categories, articles, and static pages. This lets you see in Search Console, per content type, how many URLs were submitted and how many are indexed.
• By language or country: For international sites, one sitemap per language or country version. This makes it easier to diagnose indexing gaps per market.
• By recency: A separate sitemap for frequently changed or new URLs. Crawlers that fetch this small file often find fresh content faster.

Well-planned sharding is not an end in itself. It makes indexing diagnosable. If a particular sitemap shows an unusually low indexing rate, narrow the problem down to one page type or market right away instead of searching the entire inventory.

Using lastmod, changefreq, and priority correctly

Of the three optional time and priority tags, only one matters in practice. Google uses the lastmod value only if it is consistently and verifiably accurate, for example when checked against the page's actual last modification. This is the most important rule for working with sitemaps: lastmod is a trust signal, not a marketing field.

If you set lastmod to the current date across the board on every crawl or every deployment without the content having changed, you devalue the signal. Google recognizes the unreliability and then ignores lastmod completely. Set the timestamp only for genuine content changes. A price adjustment or a new paragraph justifies a new lastmod; a changed tracking script in the footer does not.

Google largely ignores the changefreq and priority tags today. They were designed to signal crawl frequency and relative importance, but in practice they proved unreliable and easy to manipulate. You can omit these tags without any downside. Focus your maintenance on honest lastmod values and the right selection of included URLs instead.

Sitemap generation in Next.js: sitemap.ts and generateSitemaps()

In the Next.js App Router, maintaining XML files by hand or using third-party plugins is unnecessary. The sitemap.ts file convention in the app directory generates the sitemap programmatically. The function returns an array of objects with url and an optional lastModified, which Next.js renders into valid XML. This lets you generate the sitemap dynamically from CMS data, a database, or an API.

For large sites, generateSitemaps() takes over. This function splits the URL inventory into multiple sitemap files and serves the routes under /.../sitemap/[id].xml. The official Next.js documentation explicitly demonstrates the split using blocks of 50,000, noting that Google's limit is 50,000 URLs per sitemap. Each block is addressed by an id, so /product/sitemap/1.xml, for example, serves the second block.

Keep three aspects in mind for the architecture:

• Static versus dynamic: Sitemaps for content that rarely changes can be generated statically at build time. Frequently changing inventories, such as product or article URLs, should be generated dynamically at runtime or via revalidation.
• ISR and revalidate: With Incremental Static Regeneration, you set a revalidation interval so the sitemap is rebuilt at defined intervals. This keeps it current without recalculating it on every request.
• Sitemap index routing: On very large sites with multiple content types, you combine several generated sitemap groups. Make sure a top-level index file references all child sitemaps and that the URLs are consistently absolute.

After every deployment, verify that the delivered sitemap actually returns valid XML with the correct content type and that the lastmod values come from the real change dates in the data source, not from the build time.

Submission, IndexNow, and AI crawler control

A sitemap only delivers value once search engines know about it. There are two established ways, and they complement each other. First, the Sitemap directive in robots.txt, which points to the sitemap index file and is read by all major crawlers. Second, manual submission in Google Search Console and Bing Webmaster Tools, which also provides status data and error reports.

The robots.txt file is a frequently underestimated factor here. According to the 2024 Web Almanac, 83.9 percent of robots.txt files returned a 200 status on mobile requests, while 14.1 percent returned a 404. A missing or broken robots.txt means the Sitemap directive it contains doesn't take effect. So first check whether the file is served cleanly at all.

IndexNow has established itself as a complement to the traditional sitemap: a push protocol that actively notifies search engines about changed URLs instead of waiting for the next crawl. According to IndexNow.org, participating search engines include Bing, Yandex, Seznam, Naver, and Yep. Google is not among them. According to the Bing Webmaster Blog, more than 3.5 billion URLs are submitted via IndexNow per day, and 18 percent of all newly clicked URLs in search results were discovered via IndexNow. For frequently updated large sites, IndexNow is an effective real-time channel that complements the sitemap. For Google, the XML sitemap remains the way to actively report new and changed URLs.

AI crawler traffic is a growing factor. According to Cloudflare, in May 2025 around 50 percent of AI and search crawler traffic came from Googlebot, 7.7 percent from GPTBot, and 5.4 percent from ClaudeBot, while total crawler traffic grew 18 percent year over year. Traditional SEO tool crawlers are on the rise too: AhrefsBot appeared in 8.8 percent of robots.txt files in 2024, up from 5.3 percent in 2022. For large sites, this means crawl budget is spread across more and more bots. The sitemap helps direct the relevant crawler's scarce budget toward the important URLs. If you're aiming for visibility in AI answers (Generative Engine Optimization), control AI crawlers deliberately instead of blocking them across the board, and also look into formats like llms.txt.

Special sitemaps and internationalization

Beyond the standard sitemap, there are specialized variants. Image and video sitemaps declare media content explicitly, and news sitemaps cover news publications with their own freshness requirements. For most large sites, however, hreflang annotation in the XML sitemap is the most relevant special case.

With many language and country versions, you can maintain hreflang relationships either in the HTML head or centrally in the sitemap. The sitemap approach has a clear advantage on large sites: it manages all language alternatives in one place instead of duplicating them in the markup of thousands of pages. Each URL entry lists all of its language variants via xhtml:link elements. This reduces sources of error and makes maintenance considerably easier.

Despite these advantages, hreflang remains underused in practice. According to the Web Almanac, 9.6 percent of sites used hreflang on desktop in 2022, compared to 9.0 percent in 2021. For DACH companies with versions for Austria, Germany, and Switzerland, however, correct hreflang maintenance is critical to avoid duplicate content problems between the largely identical German-language versions and to serve users the right country version.

Common mistakes with large sitemaps

Most sitemap problems on large sites come from polluted sitemaps, not missing ones. These are the most frequent errors:

• Non-indexable URLs in the sitemap: Pages with a noindex tag, URLs blocked by robots.txt, or 404 pages don't belong in the sitemap. They send conflicting signals and waste crawl budget. The sitemap must contain only indexable URLs.
• Canonicalized or redirected URLs: List only the canonical target URL. A URL that points to another via canonical or triggers a 301 redirect doesn't belong in the sitemap, because it isn't the final address.
• Outdated lastmod values or lack of freshness: A sitemap that isn't updated after content changes loses its value as a discovery signal. On large sites, generation must be automatically tied to the data source.
• Faulty compression: Sitemaps may be served as .gz to save bandwidth. The 50 MB limit, however, refers to the uncompressed size. Incorrectly declared or corrupted compression causes read errors for the crawler.
• Delivery performance: If a large, dynamically generated sitemap is recalculated on every request, the response time can get too long. Crawlers abort slow sitemap fetches. Caching or revalidation at a fixed interval is mandatory.

A frequently overlooked point is consistency between the sitemap and what you actually want indexed. Every URL in the sitemap is a promise to Google: this page is relevant and should be indexed. Keep that promise by generating the sitemap from the same logic that decides indexability.

Metrics and monitoring

The sitemap is not a set-and-forget artifact. On large sites, you need ongoing monitoring, and the most important metric is the ratio of submitted to indexed URLs. Search Console shows, per sitemap, how many URLs were discovered and how many were actually indexed. A large gap between the two values is the primary warning sign.

Pay particular attention to two statuses in Search Console. The status "Discovered - currently not indexed" often points to crawl budget bottlenecks or thin content: Google knows the URL from the sitemap but has deliberately not crawled it yet. The status "Crawled - currently not indexed" means the page was crawled but judged not valuable enough to index. On large sites, both patterns show up first in individual sitemaps sharded by page type, which simplifies diagnosis.

Supplement the Search Console data with your own checks. Regularly pull the list of URLs in your sitemap and check it against HTTP status, canonical status, and the noindex flag. Any URL that doesn't respond with status 200, no redirect, no noindex, and a self-referencing canonical is a candidate for cleanup. On large sites, you can integrate this into the deployment process through log file analysis and automated crawls.

Further reading

XML sitemaps are one building block of technical SEO and take effect in combination with other mechanisms. The robots.txt file controls access and points to the sitemap, canonical tags and 301 redirects define the canonical URL that belongs in the sitemap, and hreflang links international versions. For Next.js projects, it pays to build an end-to-end SEO architecture in which sitemap.ts, robots.ts, and the Metadata API are fed from the same data sources.

For DACH companies with large, multilingual sites, it makes sense to tie the sitemap strategy closely to indexing goals and to the growing volume of AI crawlers. If you reliably get your important URLs into the index, and with that into the data that generative search systems draw on, you lay the groundwork for visibility in traditional and AI-powered search alike.