Crawl Analysis with Screaming Frog

Why crawl analysis is the foundation of every technical SEO audit

Every search engine starts by crawling. Before Google can rank a page, Googlebot has to reach, fetch, and process it. This is exactly where a crawl analysis with Screaming Frog comes in: It shows a website from a crawler's perspective, not a visitor's. Problems that stay invisible in the browser become visible: a 404 behind an internal link, a redirect chain with four hops, a canonical tag that points to the wrong URL.

For B2B companies, this has direct economic relevance. In Austria, Google's market share is 81.87 percent (as of May 2026). A site that Google can't crawl and index cleanly loses by far the most important organic channel. And every new page adds potential technical errors that go undetected without a systematic crawl.

That makes a crawl analysis a recurring control mechanism rather than a one-time audit. It turns an often vague suspicion ("organic traffic is stagnating") into concrete technical findings you can prioritize.

How it works and how to configure it: Screaming Frog basics

Screaming Frog runs as a desktop application on your own computer. That sets the tool apart from cloud-based auditing solutions and gives you full control over configuration and data storage. Two license models are available:

Free vs. paid: The free version crawls up to 500 URLs per crawl and is enough for small sites and quick spot checks. The paid license removes the limit and adds scheduling, custom extraction, API integrations, and JavaScript rendering. For serious technical audits in a B2B environment, the license is standard.

Spider mode vs. list mode: In spider mode, the crawler starts at a seed URL and follows internal links like a search engine. In list mode, it crawls only a predefined list of URLs, for example from an XML sitemap or a GSC export. List mode is suited to targeted checks of individual page clusters.

Memory vs. database storage mode: By default, Screaming Frog starts in database storage mode and stores crawl data on disk. With an SSD, this mode is almost as fast as RAM, scales much higher, and saves crawls automatically. Crawl comparisons are only available in this mode. According to the vendor, memory mode (RAM) is suitable for sites under 500,000 URLs and for computers without an SSD. Above roughly 500,000 URLs, the crawl therefore usually belongs in database mode. Google mainly addresses its crawl budget guide to sites with 10,000 or more unique pages whose content changes daily and to sites with 1 million or more unique pages whose content changes weekly. According to Google, these figures are rough estimates. For sites of this size, it pays to take a dedicated look at crawl budget.

Crawl setup for accurate data capture

A crawl is only as meaningful as its configuration. Before you start, you need to make a few decisions that strongly affect the result.

User agent: Screaming Frog can identify itself as its own crawler, as Googlebot (smartphone or desktop), or as a custom user agent. Running the crawl as Googlebot Smartphone uncovers mobile-specific content delivery and cloaking.

Respecting robots.txt: By default, the tool respects robots.txt. For a complete audit, it often makes sense to ignore it so you also see blocked areas, or to test a custom robots.txt before it goes live. Also check which status code the robots.txt file itself returns. According to the Web Almanac 2024, 14.1 percent of mobile and 14.3 percent of desktop requests returned a 404 for robots.txt. Google treats a 4xx status (except 429) as a missing file and then crawls without restrictions. 5xx responses are the critical case: Google then initially stops crawling the site for twelve hours and afterwards uses the last successfully fetched version for up to 30 days.

JavaScript rendering: Many modern B2B sites render content on the client side. In a standard crawl, Screaming Frog only sees the initial HTML. Only JavaScript rendering mode (headless Chrome) executes the JavaScript and captures what a user's device actually sees. This mode is slower and more resource-intensive, but essential for single-page applications and JS frameworks. Comparing a rendered and a non-rendered crawl shows which content and links only appear on the client side, a frequently underestimated indexing risk.

Best practices: from status codes to internal linking

A structured crawl workflow works its way from crawlability to content quality.

Status code and broken link analysis: In the "Response Codes" tab, the tool shows all 404s, 5xx server errors, redirect chains, and redirect loops. Prioritize by link strength: A 404 that a hundred internal links point to is more urgent than a 404 that nothing links to. Redirect chains with multiple hops waste crawl budget and dilute link signals, so shorten them to a direct 301 redirect.

Indexability and crawl control: The "Indexability" column summarizes whether a URL is indexable and, if not, why. Common findings are noindex directives on important pages, canonical mismatches, and conflicting robots directives. According to the Web Almanac 2024, 69 percent of desktop pages and 65 percent of mobile pages use canonical tags, but 0.8 percent have conflicting canonical values, which Google treats as undefined behavior. Screaming Frog systematically uncovers exactly these mismatches.

On-page audit: The crawl captures title lengths, meta description lengths, missing or duplicate H1s, thin content, and missing alt text. Meta descriptions are a classic gap: According to the Web Almanac 2024, only 66.7 percent of desktop pages and 66.4 percent of mobile pages had a meta description. An Ahrefs analysis of more than a million domains even found that meta descriptions were missing on 72.9 percent of the sites examined. An important caveat: Ahrefs stresses that missing meta descriptions are not a direct ranking problem, because Google generates them automatically. Prioritizing findings like these is part of the professional assessment.

Internal linking and site architecture: Using crawl depth, the tool shows how many clicks away from the homepage a page is. Important pages should be reachable within a few clicks. Orphan pages, meaning pages without an internal inlink, become visible when you compare the crawl with the sitemap or GSC data. The inlink distribution reveals which URLs receive internal link equity and which get too few internal links.

Data enrichment and AI visibility

A plain crawl delivers technical findings without traffic context. Only API integration lets you prioritize its findings.

API integration: Screaming Frog connects directly to Google Search Console, GA4, and the PageSpeed Insights API. Through Ahrefs, Majestic, or Moz, you can add link metrics such as backlinks and referring domains. The result is a traffic-weighted crawl: A technical error on a page with high organic traffic clearly gets a higher priority than the same error on a page with no clicks.

Structured data: During the crawl, the tool validates Schema.org markup and reports missing or faulty structured data. Correct markup is a prerequisite for rich results and, increasingly, for processing by AI systems.

AI crawler control: Today, robots.txt governs traditional search engines and AI crawlers such as GPTBot. Screaming Frog can check which directives apply to these user agents. If you want to be cited in AI answers (AI answer inclusion), you have to decide deliberately which AI crawlers get access. Few sites did that in 2024: According to the Web Almanac 2024, only 2.6 percent (mobile) and 2.9 percent (desktop) of robots.txt files explicitly named GPTBot. No other AI crawler was named more often.

Common mistakes in crawl analysis

Crawling JS sites without JavaScript rendering: If you crawl a client-side rendered site without rendering mode, you miss content and links and draw the wrong conclusions.

Findings without prioritization: A crawl export with thousands of rows is not an action plan. Without traffic weighting via GSC or analytics, you waste resources on irrelevant findings.

Ignoring the robots.txt configuration: A crawl that blindly respects robots.txt misses blocked areas. A crawl that ignores it without documenting this produces findings for areas that are not actually crawlable.

Choosing the wrong storage mode: Crawling large sites in RAM mode leads to aborted crawls. In database mode on a conventional hard disk drive (HDD), the disk becomes the bottleneck.

One-off crawl instead of monitoring: A single crawl is a snapshot. Technical problems arise continuously through deployments, content changes, and migrations.

Metrics, measurement, and reporting

A crawl analysis only creates value once you translate its findings into measurable KPIs.

Crawl comparisons over time: In database storage mode, the paid version lets you compare crawls. This shows whether the number of 404s is falling, whether redirect chains have been removed, and whether new indexing problems have appeared.

Scheduling and automation: Scheduled crawls (weekly, for example) provide continuous technical monitoring whose results are exported and fed into reports.

Translating findings into SEO KPIs: Technical findings connect to business metrics. A decline in indexable pages correlates with falling impressions in GSC. Fixed broken links and shortened redirect chains improve crawl efficiency. You can track the impact through organic traffic, the indexing rate, and the visibility index (Sistrix, for example). This turns a technical audit into a demonstrable contribution to SEO ROI.

Going further: from crawl to a prioritized action plan in DACH B2B

A well-designed workflow integrates Screaming Frog into the entire SEO toolset. The crawl provides the technical data foundation. GSC and GA4 provide the traffic context. Sistrix or Ahrefs provide the visibility and competitive perspective.

In practice, a B2B audit in Austria typically follows five steps: First, configure the crawl correctly (user agent, rendering, storage mode). Second, connect GSC, GA4, and a backlink tool to weight the findings. Third, prioritize the findings by severity and traffic relevance. Fourth, derive a concrete, actionable plan that links technical findings to responsibilities. Fifth, monitor the result with scheduled follow-up crawls and compare it with how competitors' visibility develops.

This turns crawl analysis from a simple error scan into a strategic management tool that links technical excellence to measurable business results.