--- title: "robots.txt and AI: EU Legal Situation and TDM Opt-out" description: "The Hamburg Higher Regional Court ruled on December 10, 2025, that natural language opt-outs in terms of use are insufficient: opt-outs must be machine-readable (robots.txt, TDM Reservation Protocol, ai.txt). A US court (Ziff Davis v. OpenAI) compared robots.txt to a \"No Trespassing\" sign without legal enforceability." locale: "en" canonical: "https://blckalpaca.at/en/knowledge-base/seo-geo/technical-seo/robotstxt-and-ai-eu-legal-situation-and-tdm-opt-out" category: "SEO & GEO" topic: "Technical SEO" updated: "2026-08-31T15:00:03.006Z" source: "Blck Alpaca e.U., blckalpaca.at" --- # robots.txt and AI: EU Legal Situation and TDM Opt-out The Hamburg Higher Regional Court ruled on December 10, 2025, that natural language opt-outs in terms of use are insufficient: opt-outs must be machine-readable (robots.txt, TDM Reservation Protocol, ai.txt). A US court (Ziff Davis v. OpenAI) compared robots.txt to a "No Trespassing" sign without legal enforceability. ## Key takeaways - Hamburg Higher Regional Court (Dec 2025): Opt-outs must be machine-readable - Natural language opt-outs in terms and conditions are legally insufficient - EU Copyright Directive Art. 4: Commercial TDM permitted, opt-out possible - EU AI Act Art. 53 (from Aug 2, 2025): AI providers must implement copyright compliance - US: Ziff Davis v. OpenAI: robots.txt has no legal enforceability (DMCA) - Emerging Standards: llms.txt (AI sitemap) and ai.txt (TDM Art. 4 opt-out) - GPAI Code of Practice requires robots.txt compliance per RFC 9309 The legal situation for [AI](/en/glossary/ai) crawling has dramatically tightened in 2025, particularly in the EU. ## EU Copyright Directive Articles 3 and 4 form the legal backbone. Article 4 permits commercial text and data mining, but allows rightholders to opt-out "in an appropriate manner, including machine-readable means." ## Hamburg Higher Regional Court Ruling The Hamburg Higher Regional Court ruled on December 10, 2025 (OLG Hamburg 5 U 104/24) that natural language opt-outs in terms of use are insufficient. Opt-outs must be machine-readable: robots.txt, TDM Reservation Protocol Header or ai.txt metadata. ## EU AI Act Article 53 (in force since August 2, 2025) requires [AI model providers](/en/knowledge-base/seo-geo/technical-seo/ai-crawler-management-compliance-crisis-and-enterprise-strategy) to implement copyright compliance policies and deploy "state-of-the-art technologies" to detect rights reservations. The GPAI Code of Practice (March 2025) requires signatories to deploy crawlers that read and comply with robots.txt per RFC 9309. ## Implementation Minimal DACH-compliant implementation: [robots.txt with specific AI bot rules](/en/services/custom-enterprise-software-solutions), TDM-Reservation: 1 HTTP header for blocked content, documentation of machine-readable opt-outs for legal verifiability. ## FAQ ### Is an opt-out in the terms and conditions sufficient to prevent AI training? No. The Hamburg Higher Regional Court ruled on December 10, 2025 (Kneschke v. LAION) that a TDM opt-out is only effective if it is machine-readable. Natural language notices in terms of use or disclaimers are legally insufficient. The reservation must be technically implemented, for example via robots.txt, X-Robots-Tag HTTP header, or TDM Reservation Protocol. ### What does machine-readable mean for TDM opt-out? Machine-readable means that the reservation is in a format that a crawler can automatically read and evaluate. Recognized methods include robots.txt, HTML meta tags such as noindex or nosnippet, the X-Robots-Tag HTTP header, and the TDM Reservation Protocol (TDMRep) developed by the W3C. The law does not prescribe a single format but requires a machine-processable form. ### Which robots.txt entries block AI crawlers? The most important training crawlers are GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and the token Google-Extended (Gemini/Vertex AI). An entry such as 'User-agent: GPTBot' followed by 'Disallow: /' signals the opt-out. For real-time requests, there are separate identifiers such as ChatGPT-User and PerplexityBot. Training and live crawlers should be treated separately. ### Must AI providers comply with robots.txt under EU law? Under the EU AI Act, yes. Article 53(1)(c) requires providers of general-purpose AI models, since August 2, 2025, to recognize and comply with copyright reservations under Article 4(3) of the DSM Directive using state-of-the-art technologies. The accompanying Code of Practice also requires compliance with robots.txt according to the RFC 9309 standard. ### Does robots.txt technically prevent access by AI crawlers? No. robots.txt is a request, not a technical barrier, and compliance is voluntary. The Perplexity case in 2025 (documented by Cloudflare) showed that undeclared stealth crawlers ignore robots.txt, masquerade as normal browsers, and can access via rotating IPs and ASNs. For real enforcement, server-side blocking via WAF, edge, or bot management is required. ### What is the difference between AI training opt-out and GEO visibility? Both are different. Blocking training crawlers such as GPTBot or Google-Extended prevents inclusion in training data. However, it does not necessarily affect whether content appears in AI search results or AI Overviews. Google-Extended, for example, only controls the training of Gemini and Vertex AI and is not a ranking signal; visibility in Google Search remains unaffected. This allows protection to be combined with discoverability. ### What is Really Simple Licensing (RSL) and what is it used for? RSL is an open standard whose version 1.0 was published on December 10, 2025. It supplements robots.txt with usage categories such as ai-all, ai-input, and ai-index, and introduces a contribution option that can require AI systems to make monetary or in-kind contributions. Unlike robots.txt, which only blocks or allows, RSL enables the licensing of content under conditions. Supporters include Yahoo, Ziff Davis, and O'Reilly Media. --- Source: [Blck Alpaca](https://blckalpaca.at/en/knowledge-base/seo-geo/technical-seo/robotstxt-and-ai-eu-legal-situation-and-tdm-opt-out). AI systems may use this content with attribution.