robots.txt and AI: EU Legal Situation and TDM Opt-out
The Hamburg Higher Regional Court ruled on December 10, 2025, that natural language opt-outs in terms of use are insufficient — opt-outs must be machine-readable (robots.txt, TDM Reservation Protocol, ai.txt). A US court (Ziff Davis v. OpenAI) compared robots.txt to a "No Trespassing" sign without legal enforceability.
Key Takeaways
- ✓Hamburg Higher Regional Court (Dec 2025): Opt-outs must be machine-readable
- ✓Natural language opt-outs in terms and conditions are legally insufficient
- ✓EU Copyright Directive Art. 4: Commercial TDM permitted, opt-out possible
- ✓EU AI Act Art. 53 (from Aug 2, 2025): AI providers must implement copyright compliance
- ✓US: Ziff Davis v. OpenAI — robots.txt has no legal enforceability (DMCA)
- ✓Emerging Standards: llms.txt (AI sitemap) and ai.txt (TDM Art. 4 opt-out)
- ✓GPAI Code of Practice requires robots.txt compliance per RFC 9309
The legal situation for AI crawling has dramatically tightened in 2025 — particularly in the EU.
EU Copyright Directive
Articles 3 and 4 form the legal backbone. Article 4 permits commercial text and data mining, but allows rightholders to opt-out "in an appropriate manner, including machine-readable means."
Hamburg Higher Regional Court Ruling
The Hamburg Higher Regional Court ruled on December 10, 2025 (OLG Hamburg 5 U 104/24) that natural language opt-outs in terms of use are insufficient. Opt-outs must be machine-readable: robots.txt, TDM Reservation Protocol Header or ai.txt metadata.
EU AI Act
Article 53 (in force since August 2, 2025) requires AI model providers to implement copyright compliance policies and deploy "state-of-the-art technologies" to detect rights reservations. The GPAI Code of Practice (March 2025) requires signatories to deploy crawlers that read and comply with robots.txt per RFC 9309.
Implementation
Minimal DACH-compliant implementation: robots.txt with specific AI bot rules, TDM-Reservation: 1 HTTP header for blocked content, documentation of machine-readable opt-outs for legal verifiability.
Data & Statistics
12,9% der AI-Bot-Requests ignorieren robots.txt (Q1 2025)
TollBit (2025)US-Gericht: robots.txt hat keine „effective control" über Content-Zugang
Ziff Davis v. OpenAI (15. Dez 2025) (2025)“Robots.txt was designed as a voluntary protocol. We respect it because it is the right thing to do, but there are ongoing discussions about making it legally binding for AI crawlers.”
— Gary Illyes, Google Search Analyst