robots.txt and AI: EU Legal Situation and TDM Opt-out

Why robots.txt and AI crawler management now matter legally

For a long time, robots.txt was a purely technical tool for controlling crawlers. With the rise of AI training data, it has become a legal question. On December 10, 2025, the Hamburg Higher Regional Court (OLG Hamburg) ruled in Kneschke v. LAION that a text and data mining reservation (TDM opt-out) is only effective if it is machine-readable. In the case decided, natural-language notices in terms of use or disclaimers were not enough. It is the first appellate ruling in Germany on copyright exceptions for AI training. It is not final. The court allowed an appeal on points of law, and the Federal Court of Justice (Bundesgerichtshof, I ZR 281/25) heard it on September 3, 2026; it intends to announce its decision on December 17, 2026 (as of October 2026). At the hearing, the First Civil Senate indicated a need for clarification under EU law; it may refer questions on text and data mining to the Court of Justice of the European Union.

For B2B websites in Germany and Austria, this shifts priorities. If you want to prevent your content from ending up in AI training data without permission, you should set this reservation technically and in machine-readable form. After the OLG Hamburg ruling, you cannot rely on a natural-language sentence in your legal notice in Germany, and in Austria it is uncertain at best. At the same time, AI crawlers' appetite for data is growing measurably: according to Cloudflare, AI bots accounted for around 4.2 percent of all HTML requests on annual average in 2025, while Googlebot alone made up 4.5 percent (international). This makes robots.txt more than a crawl budget question. It is about enforcing rights and data sovereignty.

A note on the data: the core legal statements in this article refer to EU law and its implementation in Germany and Austria and are marked as such. The crawler and adoption statistics come mostly from international surveys (Cloudflare, HTTP Archive, Pew) and serve as benchmarks.

The legal basis: Art. 4 of the DSM Directive and its implementation in Austria and Germany

The legal basis of the TDM opt-out is Article 4 of the EU Directive on Copyright in the Digital Single Market (DSM Directive, 2019/790). It generally permits text and data mining for commercial purposes too, which includes collecting web content for AI training. The key provision is Art. 4(3): rightholders can prohibit this use by reserving their rights expressly and in an appropriate manner, such as machine-readable means in the case of content made publicly available online. In the text of the article, machine readability is only an example. Recital 18 is stricter, at least in the English language version: for content made publicly available online, a reservation should only be considered appropriate if it uses machine-readable means. The recital explicitly includes metadata and the terms and conditions of a website or service. The German version lacks this "only". Whether a natural-language reservation in the terms and conditions counts as machine-readable is the core of the entire debate.

National implementation has taken place, but the wording differs. In Germany, § 44b of the Copyright Act (UrhG) governs text and data mining. Its paragraph 3 turns the directive's example into a condition of effectiveness: a reservation of use for works accessible online is only effective if it is made in machine-readable form. In Austria, it is § 42h of the Austrian Copyright Act (Urheberrechtsgesetz, UrhG). Under paragraph 6, the exception does not apply if reproduction has been expressly prohibited and the prohibition is made recognizable in an appropriate manner through a reservation of use, for example by machine-readable means for works made publicly available on the internet. Austria thus adopts the wording of the directive. Whether a natural-language notice in the terms and conditions counts as an appropriate reservation there is an open question. This results in the following logic:

• Basic rule: Commercial TDM is permitted. Without an effective reservation, AI providers may use publicly accessible content for training.
• Opt-out instead of opt-in: Protection does not apply automatically. Rightholders must actively object.
• Form online: In Germany, a reservation for online content is only effective if it is machine-readable. In Austria, the law names machine-readable means as an example of an appropriate reservation.

For Austrian companies, this means in concrete terms: protection under § 42h UrhG does not arise by itself. It has to be declared actively, ideally in a form that a machine can read and evaluate. The law names this form explicitly, and it also meets the stricter German standard.

Swiss law has no such rule. Switzerland is not an EU member, so the DSM Directive does not apply there. According to the report of the Swiss group of AIPPI on copyright and AI, the Swiss Copyright Act has no exception created specifically for training AI models. Text and data mining for research purposes falls under Art. 24d URG, with no option to reserve rights. Swiss law has no opt-out modeled on Art. 4(3) of the DSM Directive.

Even so, a machine-readable reservation is not meaningless for Swiss websites. Under Recital 106 of the AI Act, providers that place a general-purpose AI model on the EU market should comply with reservations under Art. 4(3) of the DSM Directive, regardless of the country in which training takes place. Such a reservation can therefore also take effect for content on Swiss websites.

What does machine-readable mean? Common methods and their limits

The law does not define the term machine-readable conclusively. Several methods are common in practice, but not every one of them expresses a copyright reservation. robots.txt and TDMRep can be combined to support the legal effect.

• robots.txt: The file in the root directory uses user-agent directives to control which crawlers may fetch which paths. It is the best-known and most widely used mechanism, and most AI providers accept it as an opt-out signal.
• HTML meta tags and robots directives: Tags such as noindex, nosnippet or max-snippet control indexing and snippets. According to Google, they limit what Search, including its AI features, shows from a page. They are not an explicit TDM reservation, and they cost visibility: noindex removes the page from Google Search entirely.
• HTTP header: The X-Robots-Tag header lets you set the same directives on the server side, including for non-HTML resources such as PDFs or images. The same limitation applies here.
• TDM Reservation Protocol (TDMRep): A specification by the W3C TDM Reservation Protocol Community Group, not an official W3C standard. It expresses the reservation under Art. 4 of the DSM Directive explicitly, via HTTP header, via a file at /.well-known/tdmrep.json or via an HTML meta tag. TDMRep was designed specifically for the legal requirement of Art. 4.

The distinction between purpose and form matters. robots.txt controls crawling on a technical level. TDMRep specifically addresses the copyright reservation. If you want legal certainty, set both, because the OLG Hamburg explicitly requires the machine-readable form without prescribing a single format.

The Kneschke v. LAION decision and its consequences for Germany and Austria

The Hamburg Regional Court dismissed the action on September 27, 2024 (310 O 227/23). On appeal, the OLG Hamburg (case number 5 U 104/24) ruled on December 10, 2025. Photographer Robert Kneschke objected to one of his images ending up in the training dataset of the organization LAION. His opt-out was worded in the terms of use of the image platform, that is, in natural language.

The OLG Hamburg held this reservation to be ineffective. Its reasoning: the notice on the photo agency's website did not have the machine-readable form required by law at the time of the download (§ 44b(3) sentence 2 UrhG). As a result, the exception in § 44b UrhG, which implements Art. 4 of the DSM Directive, applied in LAION's favor. According to Freshfields, the benchmark was the situation at the time of the download in 2021. The court therefore did not decide how such a notice should be assessed with today's technology. The OLG also based the use on the exception for scientific research under § 60d UrhG, for which no reservation is possible. This part holds for LAION as a non-commercial research association, but usually not for commercial AI providers. For them, what counts is § 44b UrhG and therefore the machine-readable reservation. The judgment is not final (see above). According to the analysis by Norton Rose Fulbright, open questions include the burden of proof and what exactly counts as machine-readable.

The consequences for Germany and Austria are significant:

• In Germany, machine readability is mandatory: If you want an effective reservation, you cannot rely on terms and conditions or disclaimers. The opt-out must be implemented technically.
• In Austria, it is strongly recommended: The law names machine-readable means as an example. Whether a notice in the terms and conditions is sufficient remains unresolved; a technical reservation avoids the question.
• First appellate guidance: As the first decision at appellate level in Germany, it sends a signal to Austria as well, even though Austrian courts are not formally bound by it.
• Pressure on rightholders to act: If you have so far objected only in natural language, you cannot rely on that reservation in either Germany or Austria. A technical fix pays off in both countries.

A counterexample from the US shows how the legal situation differs. In Ziff Davis v. OpenAI, the US proceedings compared robots.txt to a "No Trespassing" sign without any legal enforceability of its own. Unlike EU law, US law has no TDM opt-out under Art. 4. Machine-readable reservations there do not have the same copyright effect as in the EU.

EU AI Act Art. 53: the copyright obligation of AI providers

The second pillar alongside the DSM Directive is the EU AI Act. It requires providers of general-purpose AI models (GPAI) to respect copyright reservations. Under Art. 53(1), GPAI providers must put in place a policy to comply with Union copyright law and, in particular, identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Art. 4(3) of Directive 2019/790 (EU). The AI Act has been in force since August 1, 2024, and this obligation has applied since August 2, 2025. For models that were already on the market before that date, a transition period runs until August 2, 2027.

This links the two legal acts into one system. The DSM Directive gives rightholders the tool of the machine-readable reservation. The AI Act requires AI providers to actively identify and comply with these reservations. In practice, the phrase "state-of-the-art technologies" means that providers must evaluate exactly the machine-readable signals a website operator sets: robots.txt, TDMRep and comparable standards.

In addition, there is the voluntary General-Purpose AI Code of Practice, published on July 10, 2025. Its signatories commit in the copyright chapter (Measure 1.3) to deploy crawlers that read and follow robots.txt in accordance with RFC 9309 (Robots Exclusion Protocol). For website operators, this is the decisive bridge: a correctly set robots.txt entry is the most widespread machine-readable signal for a reservation. GPAI providers must identify and comply with such reservations under Art. 53 of the AI Act, and the Code's signatories have additionally committed to robots.txt.

Practical robots.txt configuration for AI crawlers

Effective control starts with knowing the relevant user agents. Every major AI provider runs its own crawlers, often with separate bots for training and for real-time retrieval. Opt-outs are spreading: according to the Web Almanac 2024, desktop robots.txt files contained GPTBot in 2.9 percent, CCBot in 2.7 percent, Google-Extended in 2.5 percent and Anthropic-ai in 2.1 percent (international). In 2024, these shares were still at an early stage, and they have been growing since.

The most important user agents and their function:

• GPTBot: OpenAI's crawler for training. Blocking it prevents inclusion in future training data.
• ChatGPT-User: Fetches pages when a ChatGPT user triggers it. According to OpenAI, robots.txt rules may not apply to it. Whether content appears in ChatGPT search is controlled by OAI-SearchBot.
• ClaudeBot: Anthropic's training crawler. Anthropic-ai is an older associated identifier.
• Google-Extended: Not a separate crawler but a robots.txt token. It controls whether Google uses content for training Gemini models and for grounding in Gemini Apps and Vertex AI.
• CCBot: Common Crawl's crawler, whose open datasets many AI models use as a training source.
• PerplexityBot: Perplexity's crawler for its AI search index.

A typical blocking strategy for training crawlers looks like this:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

The picture in practice is nuanced. According to Cloudflare, in 2025 most full Disallow directives targeted AI crawlers such as GPTBot, ClaudeBot and CCBot, while Googlebot and Bingbot were mostly blocked only partially (international). Operators therefore deliberately block pure AI training bots but let classic search crawlers through so they do not lose organic visibility.

The limits of robots.txt: voluntary compliance and stealth crawlers

robots.txt has a fundamental weakness: compliance is voluntary. The file is a request, not a technical barrier. Reputable providers respect it. In the EU, the AI Act requires GPAI providers to identify and comply with reservations of rights; the signatories of the Code of Practice have additionally committed to following robots.txt. But the file can be ignored technically.

The Perplexity case made this clear. On August 4, 2025, Cloudflare documented that Perplexity used undeclared stealth crawlers that relied on a generic browser impersonating Google Chrome on macOS when its declared crawler was blocked, ignoring robots.txt or in some cases not even fetching it (international). The bots used IP addresses outside the official range and rotated across different ASNs to evade blocks. Cloudflare then removed Perplexity's verified bot status and blocked the stealth crawlers with its own heuristics.

This leads to clear limits:

• No technical enforcement: robots.txt does not prevent access; it only signals a wish.
• Undeclared bots: Crawlers that pose as normal browsers cannot be stopped reliably with user-agent rules.
• IP and ASN rotation: Anyone accessing a site from changing address ranges gets around simple IP blocks.

The reservation still matters for the legal assessment. Even if a crawler ignores it technically, the machine-readable opt-out documents that the rightholder has objected. That is the prerequisite for invoking Art. 4(3) of the DSM Directive at all.

Complementary standards: ai.txt, llms.txt and Really Simple Licensing

Several complementary standards have emerged alongside robots.txt. They pursue different goals, from pure blocking to licensing.

• ai.txt: A separate file intended specifically to express the TDM reservation under Art. 4. It adds an explicitly copyright-related layer to robots.txt.
• llms.txt: Works more like a sitemap for AI systems. It is meant to make content discoverable for AI use in a structured way, so it is a discovery standard rather than a blocking standard. Adoption has been low so far and is at an early stage.
• Really Simple Licensing (RSL): Arguably the most important new standard. RSL 1.0 was published as an official specification on December 10, 2025. According to the announcement, RSL 1.0 defines the additional usage categories ai-all, ai-input and ai-index to give publishers finer control, and includes a new payment option called contribution that can require AI systems to make monetary or in-kind contributions (international). Supporters include Yahoo, Ziff Davis and O'Reilly Media.

The key conceptual difference is between blocking and licensing. robots.txt and ai.txt only say yes or no to crawling. RSL goes further: it allows content to be made available under conditions or for compensation. This creates a middle path between a complete block and free access, which can be attractive for commercial content providers in the B2B sector.

The trade-off: AI training opt-out versus GEO visibility

A central strategic question is often misunderstood. Blocking training crawlers and being found in AI answers are two different things. If you block GPTBot or Google-Extended, you prevent inclusion in training data. That does not necessarily affect whether the content appears in AI search results or overviews.

Google-Extended illustrates this well. It is a robots.txt token for opting out of AI training for Gemini and Vertex AI that does not affect inclusion in Google Search and is not a ranking signal (international). According to Google, the token also controls grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, meaning whether content from the search index flows into the model's answer to a query. If you block Google-Extended, you stay visible in regular search and in AI Overviews. Gemini then uses the content neither for training nor as a basis for its answers.

Visibility in AI answers is also the economically relevant factor, because AI overviews take clicks away. According to an international Pew analysis, users clicked on a traditional result in 8 percent of searches with an AI overview, compared with 15 percent without one, and only 1 percent clicked on a link inside the AI overview (international, USA, March 2025). The click rate almost halves. In this environment, it can make sense for B2B providers to be cited in AI answers just to stay visible, while protecting their own material from uncontrolled training at the same time. Separate directives for training and search crawlers allow exactly this separation.

Technical enforcement beyond robots.txt

Because robots.txt is only a request, a serious strategy needs a second layer: server-side enforcement. This is where edge, CDN and bot management come in.

• WAF and edge blocking: A web application firewall or edge rule can block requests by user agent, IP range or behavior pattern before they reach the origin server. This also works against crawlers that ignore robots.txt.
• Bot management: Behavior-based detection identifies automated access by signals such as request frequency or missing browser characteristics, not just by the declared user agent.
• Cloudflare AI crawler controls: Cloudflare offers dedicated switches to block known AI crawlers centrally, and it responded to the Perplexity case with its own heuristics against stealth crawling.
• Pay-per-crawl approaches: Newer models let you grant AI crawlers access only in exchange for payment. This combines the technical block with a licensing idea, similar to the RSL logic.

This layer is the only one with real technical effect. It does not replace the legal reservation; it complements it. Only the combination of a documented opt-out and actual blocking covers both the legal and the practical side.

Common mistakes in AI crawler management

• Opt-out only in the terms and conditions: The most common mistake and, after the OLG Hamburg ruling, the most serious one. In Germany, this judgment means you cannot rely on a natural-language notice; in Austria it is uncertain at best.
• Treating robots.txt as a barrier: The file does not prevent access. If you think it guarantees technical protection, you are mistaken.
• Mixing up training and search: If you block all AI bots across the board, you may lose GEO visibility without meaning to. Training and search crawlers need separate rules.
• Outdated user-agent lists: AI providers introduce new crawlers and change identifiers. A robots.txt written once goes out of date quickly.
• No server-side backup: Relying on robots.txt alone lets stealth crawlers and undeclared bots pass unhindered.
• Missing documentation: Without proof of when which reservation was set, it is hard to show in a dispute that you actively objected.

Metrics and measurement: making crawler access visible

AI crawler management can be measured, even though there is no single success metric. Useful metrics and checkpoints:

• AI bot share of traffic: Analyze server or CDN logs by AI user agents. A useful benchmark is the average of around 4.2 percent of HTML requests in 2025 measured by Cloudflare.
• Block effectiveness: Check whether blocked crawlers really have no more successful requests. Continued access despite Disallow points to non-compliant or stealth bots.
• Anomaly detection: Watch for atypical patterns such as browser user agents with server-like behavior or requests from rotating ASNs.
• Reservation completeness: Check regularly whether robots.txt, the TDMRep signal and, where used, RSL are consistent and up to date.
• GEO visibility: Track whether and how your content is cited in AI answers and AI Overviews so you can manage the trade-off between protection and discoverability.

To get started, analyzing your own logs together with the bot reports of your CDN is enough. What matters is repeating this regularly, because both the crawler landscape and the legal requirements change quickly.

Recommendations and further reading

For B2B websites, especially those running on Next.js and edge architectures, a layered strategy makes sense. It covers the legal and the technical level at the same time.

• Machine-readable reservation: Set a TDM reservation under Art. 4(3) of the DSM Directive in machine-readable form. robots.txt with targeted Disallow directives for training crawlers forms the basis, complemented by a TDMRep signal via HTTP header or markup.
• Separate crawler logic: Separate training bots such as GPTBot, ClaudeBot, CCBot and Google-Extended from search crawlers such as OAI-SearchBot and PerplexityBot, so that protection against training does not damage your GEO visibility.
• Server-side blocking: Back up the reservation with real enforcement through WAF, edge or your CDN's bot management to catch stealth crawlers and undeclared bots.
• Documentation: Record when which reservation was set. In a dispute, the documented, machine-readable opt-out is the basis for invoking Art. 4.

The legal situation is still young and evolving. The OLG Hamburg has drawn a first line, which the Federal Court of Justice and possibly the Court of Justice of the European Union will now rule on. The AI Act has made providers' obligations binding since August 2025, and for older models from August 2027. New standards such as RSL are emerging quickly. If you act today, implement the layered strategy and keep an eye on developments. Related topics include crawl budget and multi-bot governance for AI crawlers, structured data as an AI understanding layer, GDPR-compliant data processing, and GEO and visibility in AI answers.