feat(seo): open the crawler policy to every operator

[MarkusNeusinger]

The change

Retrieval, citation, search indexing and model training are now permitted for every operator, replacing the retrieval-yes / training-no split from #9633.

	Before	After
Retrieval agents	allowed	allowed
Training crawlers (GPTBot, CCBot, Amazonbot, meta-externalagent)	declined	allowed
Google-Extended	declined	allowed
Applebot-Extended	declined	allowed
ClaudeBot	allowed	allowed
Bytespider	declined	declined (bandwidth, not policy)
Content-Signal	ai-train=no + Art. 4 reservation	ai-train=yes, reservation removed

Why

The split was never coherent here. The catalogue is MIT-licensed and published to be reused — every page ships its runnable source, /llms.txt describes the site for agents, an MCP server exists so assistants can query it directly. Declining a training crawler protected nothing the licence had not already granted, and cost reach.

It also had a consequence nobody intended. Google-Extended is a single token governing Gemini grounding and training together; no finer control exists, verified against Google's crawler documentation. Declining it as though it were training-only kept anyplot out of Gemini's answers entirely. Applebot-Extended really is training-only (retrieval rides on Applebot), so declining that was correct before and is simply no longer wanted.

It corrected an asymmetry that favoured one vendor. ClaudeBot — Anthropic's training crawler, per Anthropic's own documentation — sat in the allow group while its exact counterpart GPTBot was declined. Nothing recorded why. Opening both resolves it in the direction the licence already points.

The Article 4 reservation

The old file declared Content-Signal: ai-train=no as an express reservation of rights under Article 4 of EU Directive 2019/790. It is removed rather than left standing three lines from a licence that grants exactly what it reserves — a reservation contradicting your own licence is worse than none.

Structural simplification

With the policy uniform, the named allow-groups became redundant: they resolve identically to User-agent: *. The file collapses to one declined group plus the wildcard, which removes the first-match ordering traps its own header spent twenty lines warning about.

Verified with Python's urllib.robotparser — a strict first-match parser — across eleven agent/path pairs:

GPTBot /box-basic              -> allowed      Bytespider /box-basic   -> blocked
ClaudeBot /box-basic           -> allowed      Googlebot /debug        -> blocked
Google-Extended /box-basic     -> allowed      GPTBot /interactive     -> blocked
Applebot-Extended /box-basic   -> allowed
CCBot, Amazonbot, MistralAI-Training, Googlebot -> allowed

⚠️ Does not take effect without a dashboard change

Cloudflare AI Crawl Control still 403s GPTBot, CCBot, Amazonbot, meta-externalagent and Google-CloudVertexBot at the edge, verified on the live zone 2026-08-18. A permission granted here that the edge blocks is a published lie. Unblock everything except Bytespider in the zone dashboard; docs/reference/seo.md records the measured state and the verification commands.

Also recorded there: Google-CloudVertexBot is blocked at the edge and appears in no repo-side policy — a decision that exists only as a dashboard toggle. And Cloudflare is no longer prepending a managed robots.txt block, unlike the 2026-07-25 measurement; the live file is byte-identical to this one.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR updates anyplot’s crawler/robots policy documentation and robots.txt to permit retrieval, citation, search indexing, and model training for all operators (with a bandwidth-based exception for Bytespider), aligning the repo-side policy with the intended licensing stance and simplifying the robots group structure.

Changes:

• Simplifies app/public/robots.txt to a single declined group (Bytespider) plus the wildcard group with Content-Signal ... ai-train=yes.
• Updates docs/reference/seo.md to record the new “everything open” decision and the Cloudflare edge/dashboard enforcement caveat.
• Adds a detailed [Unreleased] changelog entry describing the policy change and the required Cloudflare dashboard alignment.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
docs/reference/seo.md	Updates the documented AI crawler policy decision and Cloudflare enforcement notes.
CHANGELOG.md	Adds an [Unreleased] entry describing the updated crawler policy and operational requirement.
app/public/robots.txt	Implements the simplified “open by default” robots policy with a Bytespider exception and updated content signals.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[MarkusNeusinger]

Both applied, and the second was broader than the line it was raised against.

The self-contradiction was real and worse than flagged. The "Robots.txt" section still described four groups and quoted a verbatim block containing ai-train=no, three screens above the section declaring the policy open. It now describes the two groups that exist. While there I found the backend subsection still showing the old blanket Disallow: / from before /og/ was opened — same class, so fixed in the same pass.

The changelog convention was missing from twelve entries, not one. Checking the file properly showed this whole run of SEO work shipped without PR references: #10452, #10453, #10455, #10458, #10462, #10465, #10470–#10474. The merged ones are fixed here too — they live in the same file, and a half-referenced changelog is the harder thing to audit at release time than a fully un-referenced one.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (5)

docs/reference/seo.md:336

• This sentence says "Allow comes first" but doesn’t explain that it’s because /og/ is the specific exception and / is the broad rule. Making that explicit keeps it consistent with the ordering rationale described above.

`/og/` is the exception: every prerendered page references its preview image
there, so a blanket `Disallow` pointed crawlers at an image they were forbidden
to fetch. `Allow` comes first for the first-match parsers described above.

docs/reference/seo.md:322

• The wording here implies a hard rule that Disallow: must always come before Allow:. A few lines later (API robots.txt) the file relies on the opposite ordering for a more-specific allow exception. Rephrase this bullet to describe the underlying rule (most-specific match first) to avoid contradictory guidance in the same document.

This issue also appears on line 334 of the same file.

- Inside each group, `Disallow:` comes **before** `Allow: /` — same reason: with
  the broad allow first, a first-match parser (Python's `urllib.robotparser`,
  for one) hands out `/debug` and `/interactive`.

app/public/robots.txt:13

• The repository’s documentation consistently uses US English (“license”). This file introduces the British spelling “licence”, which is inconsistent with the surrounding docs and style guidance.

# So this file grants what the licence already grants. Retrieval, citation,
# search indexing and model training are all permitted, for every operator,
# without the retrieval-yes / training-no split this file used to draw. That
# split was never coherent here: the licence permits reuse including training,
# so declining a training crawler protected nothing while costing reach.

docs/reference/seo.md:352

• This section uses “licence”, but the rest of the docs use “license”. For consistency (and Google style), switch to US spelling.

licence had not already given away, while costing reach.

CHANGELOG.md:147

• This changelog entry uses the British spelling “licence”, but other docs in the repo use “license”. Align spelling for consistency.

  declining a training crawler protected nothing the licence had not already granted, while costing
  reach. It also had an unintended consequence — Google's `Google-Extended` token governs Gemini
  **grounding and training together**, with no finer control, so declining it kept anyplot out of
  Gemini's answers entirely. The `Content-Signal` line moves to `ai-train=yes` and the Article 4
  reservation is gone rather than left contradicting the licence beside it. `Bytespider` remains