2.15Advanced10 min

robots.txt and AI: EU Legal Situation and TDM Opt-out

Lucas Blochberger·7 April 2026·Updated 11 June 2026

Definition

The Hamburg Higher Regional Court ruled on December 10, 2025, that natural language opt-outs in terms of use are insufficient: opt-outs must be machine-readable (robots.txt, TDM Reservation Protocol, ai.txt). A US court (Ziff Davis v. OpenAI) compared robots.txt to a "No Trespassing" sign without legal enforceability.

Key Takeaways

✓Hamburg Higher Regional Court (Dec 2025): Opt-outs must be machine-readable
✓Natural language opt-outs in terms and conditions are legally insufficient
✓EU Copyright Directive Art. 4: Commercial TDM permitted, opt-out possible
✓EU AI Act Art. 53 (from Aug 2, 2025): AI providers must implement copyright compliance
✓US: Ziff Davis v. OpenAI: robots.txt has no legal enforceability (DMCA)
✓Emerging Standards: llms.txt (AI sitemap) and ai.txt (TDM Art. 4 opt-out)
✓GPAI Code of Practice requires robots.txt compliance per RFC 9309

The legal situation for AI crawling has dramatically tightened in 2025, particularly in the EU.

EU Copyright Directive

Articles 3 and 4 form the legal backbone. Article 4 permits commercial text and data mining, but allows rightholders to opt-out "in an appropriate manner, including machine-readable means."

Hamburg Higher Regional Court Ruling

The Hamburg Higher Regional Court ruled on December 10, 2025 (OLG Hamburg 5 U 104/24) that natural language opt-outs in terms of use are insufficient. Opt-outs must be machine-readable: robots.txt, TDM Reservation Protocol Header or ai.txt metadata.

EU AI Act

Article 53 (in force since August 2, 2025) requires AI model providers to implement copyright compliance policies and deploy "state-of-the-art technologies" to detect rights reservations. The GPAI Code of Practice (March 2025) requires signatories to deploy crawlers that read and comply with robots.txt per RFC 9309.

Implementation

Minimal DACH-compliant implementation: robots.txt with specific AI bot rules, TDM-Reservation: 1 HTTP header for blocked content, documentation of machine-readable opt-outs for legal verifiability.

Data & Statistics

OLG Hamburg (5 U 104/24, Urteil vom 10.12.2025): Ein natürlichsprachlicher TDM-Opt-out (in Nutzungsbedingungen/Disclaimer) ist nach Art. 4 Abs. 3 DSM-Richtlinie unzureichend; der Opt-out muss maschinenlesbar sein; Kneschkes Opt-out war unwirksam, weil nicht maschinenlesbar

Norton Rose Fulbright (Inside Tech Law) [Deutschland/Hamburg] (2025)

Art. 53(1)(c) AI Act: GPAI-Anbieter müssen eine Copyright-Policy einführen, die den Rechtevorbehalt nach Art. 4(3) DSM-Richtlinie identifiziert und einhält, auch mittels modernster Technologien; in Kraft seit 2. August 2025 (Art. 113(b))

EU Artificial Intelligence Act (artificialintelligenceact.eu), Volltext Artikel 53 [EU] (2025)

8 Prozent Klickrate auf ein Suchergebnis mit KI-Übersicht gegenüber 15 Prozent ohne (fast halbiert); 1 Prozent klickte auf einen Link innerhalb der KI-Übersicht; Basis: 68.879 Google-Suchen, 900 US-Erwachsene, März 2025

Pew Research Center [international, USA] (2025)

Perplexity nutzte undeklarierte Stealth-Crawler (Chrome/macOS-Imitation, IP-/ASN-Rotation), die robots.txt ignorierten oder teils nicht abriefen; Cloudflare entzog den Verified-Bot-Status und blockierte die Bots (4. August 2025)

Cloudflare Blog [international] (2025)

KI-Bots = 4,2 Prozent aller HTML-Requests (Jahresdurchschnitt 2025); Googlebot = 4,5 Prozent; GPTBot/ClaudeBot/CCBot hatten die meisten vollständigen Disallow-Direktiven in robots.txt; Googlebot/Bingbot überwiegend nur partiell gesperrt

Cloudflare Blog - Radar 2025 Year in Review [international] (2025)

Desktop-robots.txt-Nutzung der KI-Crawler: GPTBot 2,9 Prozent, CCBot 2,7 Prozent, Google-Extended 2,5 Prozent, Anthropic-ai 2,1 Prozent

Web Almanac 2024 (HTTP Archive), SEO-Kapitel [international] (2024)

RSL 1.0 am 10.12.2025 als offizielle Industriestandard-Spezifikation veröffentlicht; Nutzungskategorien ai-all / ai-input / ai-index ergänzen robots.txt; contribution-Zahlungsmodell enthalten; Unterstützer u. a. Yahoo, Ziff Davis, O'Reilly Media

RSL Standard - offizielle Pressemitteilung: Really Simple Licensing (RSL) 1.0 Specification [international] (2025)

Google-Extended: robots.txt-Token zum KI-Training-Opt-out (Gemini/Vertex AI), ohne Auswirkung auf die Aufnahme in die Google-Suche und kein Ranking-Signal

Search Engine Journal [international] (2026)

FAQ

Is an opt-out in the terms and conditions sufficient to prevent AI training?

No. The Hamburg Higher Regional Court ruled on December 10, 2025 (Kneschke v. LAION) that a TDM opt-out is only effective if it is machine-readable. Natural language notices in terms of use or disclaimers are legally insufficient. The reservation must be technically implemented, for example via robots.txt, X-Robots-Tag HTTP header, or TDM Reservation Protocol.

What does machine-readable mean for TDM opt-out?

Machine-readable means that the reservation is in a format that a crawler can automatically read and evaluate. Recognized methods include robots.txt, HTML meta tags such as noindex or nosnippet, the X-Robots-Tag HTTP header, and the TDM Reservation Protocol (TDMRep) developed by the W3C. The law does not prescribe a single format but requires a machine-processable form.

Which robots.txt entries block AI crawlers?

The most important training crawlers are GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and the token Google-Extended (Gemini/Vertex AI). An entry such as 'User-agent: GPTBot' followed by 'Disallow: /' signals the opt-out. For real-time requests, there are separate identifiers such as ChatGPT-User and PerplexityBot. Training and live crawlers should be treated separately.

Must AI providers comply with robots.txt under EU law?

Under the EU AI Act, yes. Article 53(1)(c) requires providers of general-purpose AI models, since August 2, 2025, to recognize and comply with copyright reservations under Article 4(3) of the DSM Directive using state-of-the-art technologies. The accompanying Code of Practice also requires compliance with robots.txt according to the RFC 9309 standard.

Does robots.txt technically prevent access by AI crawlers?

No. robots.txt is a request, not a technical barrier, and compliance is voluntary. The Perplexity case in 2025 (documented by Cloudflare) showed that undeclared stealth crawlers ignore robots.txt, masquerade as normal browsers, and can access via rotating IPs and ASNs. For real enforcement, server-side blocking via WAF, edge, or bot management is required.

What is the difference between AI training opt-out and GEO visibility?

Both are different. Blocking training crawlers such as GPTBot or Google-Extended prevents inclusion in training data. However, it does not necessarily affect whether content appears in AI search results or AI Overviews. Google-Extended, for example, only controls the training of Gemini and Vertex AI and is not a ranking signal; visibility in Google Search remains unaffected. This allows protection to be combined with discoverability.

What is Really Simple Licensing (RSL) and what is it used for?

RSL is an open standard whose version 1.0 was published on December 10, 2025. It supplements robots.txt with usage categories such as ai-all, ai-input, and ai-index, and introduces a contribution option that can require AI systems to make monetary or in-kind contributions. Unlike robots.txt, which only blocks or allows, RSL enables the licensing of content under conditions. Supporters include Yahoo, Ziff Davis, and O'Reilly Media.

How does your website perform?

Get a free, AI-powered SEO report of your website by email: technical SEO, on-page, keywords & competitors. No obligation.

Get a free SEO audit →

Previous← 301 Redirects: Using Redirects Correctly NextHreflang for DACH: AI Platforms Ignore Language Signals →