diff --git a/CHANGELOG.md b/CHANGELOG.md index 65ce0f38cb..a186494bc7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,7 +28,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an search crawlers are included on purpose: per-engine crawl frequency is otherwise only visible by sampling Search Console one URL at a time, which is how a months-long recrawl gap went unnoticed. Hooked in as a router dependency rather than per handler, so the next endpoint added cannot go - unrecorded. + unrecorded (#10471). - **Dead URLs are now visible in analytics** — a new `page_not_found` Plausible event fires when a visitor reaches a URL the app cannot serve, carrying the requested `path` and a `source` that @@ -41,7 +41,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an which redirects such visitors to the hub instead of rendering a 404 — without it the event would have missed the exact case it was built for, since that silent redirect is what a library migration produces. Documented in `docs/reference/plausible.md` (#10453 covers the crawler-facing - half; bots run no JavaScript, so the two never overlap). + half; bots run no JavaScript, so the two never overlap) (#10465). - **Search Console API access, documented and reproducible** — `docs/reference/seo.md` gains a "Search Console API access" section: the domain property (`sc-domain:anyplot.ai`), the Application Default Credentials login that carries the `webmasters.readonly` scope, a @@ -49,7 +49,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an token can never hold the scope, and the localhost callback flow dies with `missing_code` on WSL2 before you finish the consent screen. Machine-specific values move to `.env` (`SEARCH_CONSOLE_PROPERTY`, `SEARCH_CONSOLE_ACCOUNT`, documented in `.env.example`), so a - second machine is one login away from full-mode audits. + second machine is one login away from full-mode audits (#10452). - **Three new verification skills** — `/verify-migrations` runs the Alembic chain against a throwaway Postgres (Docker, or a rootless `pgserver` fallback; single-head check, @@ -137,6 +137,20 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **The crawler policy is open to every operator** — retrieval, citation, search indexing and model + training are now permitted for all of them, replacing the retrieval-yes / training-no split from + #9633. That split was never coherent for an MIT-licensed catalogue published to be reused: + declining a training crawler protected nothing the licence had not already granted, while costing + reach. It also had an unintended consequence — Google's `Google-Extended` token governs Gemini + **grounding and training together**, with no finer control, so declining it kept anyplot out of + Gemini's answers entirely. The `Content-Signal` line moves to `ai-train=yes` and the Article 4 + reservation is gone rather than left contradicting the licence beside it. `Bytespider` remains + declined on bandwidth grounds, not principle: it is documented as ignoring robots.txt and crawling + far harder than this catalogue can justify serving. Verified against Python's `urllib.robotparser` + across eleven agent/path pairs. **Requires a Cloudflare dashboard change to take effect** — the + edge still 403s GPTBot, CCBot, Amazonbot, meta-externalagent and Google-CloudVertexBot, and a + permission the edge blocks is a published lie (#10474). + - **Repository prose now follows the Google developer documentation style guide** — the `write-docs` skill gains a "Writing style" section anchoring [Google style](https://developers.google.com/style) as the baseline for `docs/`, `README.md`, @@ -218,7 +232,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an index none of the 3,913 previews. Link-preview bots were unaffected either way — they do not consult robots.txt, which is why this went unnoticed. `/og/` is now allowed, with `Allow` placed before `Disallow` so first-match parsers see the exception; verified against Python's - `urllib.robotparser`, which is exactly such a parser. + `urllib.robotparser`, which is exactly such a parser (#10472). - **The SEO proxy no longer invents pages** — `/{spec}/{language}/{library}` served HTTP 200 with a self-referencing canonical for *any* language and library string, because neither segment is @@ -228,7 +242,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an URLs from the highcharts Python→JS migration (#8516) were still indexed months after those implementations were deleted. Unknown combinations now return 404, and the 404 is not cached so a later regen becomes visible immediately. The stale URLs are dropped rather than redirected: - measured over 28 days they carry 4.4% of impressions and 10 clicks in total. + measured over 28 days they carry 4.4% of impressions and 10 clicks in total (#10453). - **A trailing slash sent crawlers to a crawl-blocked URL** — `/box-basic/` answered `307` to `http://api.anyplot.ai/seo-proxy/box-basic`: the internal proxy path, on the API host, over plain http, and that host's `robots.txt` disallows everything. FastAPI's `redirect_slashes` built the @@ -237,7 +251,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an written with a trailing slash, which is common, dead-ended there; some of the 48 Search Console "Redirect error" URLs came from this. nginx now normalises the trailing slash to the canonical URL before any routing, the proxy declares the scheme, and `bot-serving-check` fails if a - trailing-slash redirect ever again points at `/seo-proxy` or downgrades to http. + trailing-slash redirect ever again points at `/seo-proxy` or downgrades to http (#10473). - **AI assistants asked about a plot page saw nothing** — seven user-directed fetchers (eight UA patterns; NotebookLM and Mariner each answer to two) were absent from the `$is_bot` map in @@ -251,7 +265,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an assistant to open a page, and Google documents its own as generally ignoring robots.txt — so this map, not the crawler policy, is the only control point for them. The daily `bot-serving-check` now covers all of them, since the failure is invisible to every human - visitor and would otherwise regress unnoticed. + visitor and would otherwise regress unnoticed (#10470). - **Home-page filter params were self-canonicalising** — the bot page built its canonical from the request query string, so `/?spec=point-basic` declared itself a page in its own right rather than @@ -263,7 +277,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an `canonical` slots, previously one value, are now separate — defaulting to identical, so every other handler is unchanged. Verified that `?view=` and `?language=` on implementation and hub pages were already correct. Verified that `?view=` and `?language=` on implementation and - hub pages were already correct. + hub pages were already correct (#10462). - **Meta descriptions were three times too long to survive a search result** — every prerendered page passed the full spec description straight into ``. Measured across 40 live pages: median 424 characters, longest 801, and all 40 over Google's ~155-character @@ -272,7 +286,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an that ends on the last full sentence that fits (falling back to a word boundary), taking the same sample to a median of 142 with none over the limit. Visible body copy and JSON-LD keep the full description — the trim is for the snippet, not the content — and it runs on the raw text before - escaping, so it can never cut through an HTML entity. + escaping, so it can never cut through an HTML entity (#10458). - **Googlebot was walking an infinite redirect on every `/{spec}/{language}` URL** — the handler answered `Location: /seo-proxy/{spec}`, its own internal path. nginx serves crawlers by prepending `/seo-proxy` to the request URI, so the bot fetched `anyplot.ai/seo-proxy/{spec}`, @@ -281,7 +295,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an at `anyplot.ai/seo-proxy/seo-proxy`; a normal user agent never saw it, because only bots take the proxy path. Search Console recorded it as 48 URLs under **Redirect error**. The Location is now the public URL, and a regression test asserts both that it never starts with `/seo-proxy` - and that re-entering with the prefix nginx adds resolves instead of bouncing. + and that re-entering with the prefix nginx adds resolves instead of bouncing (#10455). - **`seo-auditor` could never reach Search Console** — its auth contract probed `gcloud auth print-access-token`, which mints the gcloud CLI credential; that credential's scope set is fixed and can never include `webmasters.readonly`, so every audit since the @@ -291,7 +305,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an auditor was told to collect data it was simultaneously barred from fetching; read-ness is now anchored to the `webmasters.readonly` scope rather than the HTTP verb, which additionally unlocks `urlInspection`. Its Coverage guidance no longer asks for the page-indexing report - buckets, which have no API at all. + buckets, which have no API at all (#10452). - **Model↔migration index drift fixed before it could drop production indexes** — seven migration-created indexes (`ix_specs_issue`, `ix_specs_tags` GIN, `ix_impls_library_id`, diff --git a/app/public/robots.txt b/app/public/robots.txt index 210f603294..657a9163f6 100644 --- a/app/public/robots.txt +++ b/app/public/robots.txt @@ -1,73 +1,63 @@ -# Crawler policy — AI crawlers that RETRIEVE and CITE are welcome: anyplot is -# built to be consumable by AI agents (see /llms.txt and the MCP server at -# https://api.anyplot.ai/mcp/). Training-only scrapers are declined below. +# Crawler policy — everything here is open, to humans and machines alike. # -# Cloudflare's AI Crawl Control can PREPEND its own managed block to the live -# file and additionally answer the blocked user agents with HTTP 403 at the -# edge, so https://anyplot.ai/robots.txt may be longer and stricter than this -# file. The policy is therefore spelled out here in full — including the -# content signals Cloudflare would otherwise supply — so it survives with the -# managed block turned off. Rationale, dashboard steps and how to verify: -# docs/reference/seo.md ("AI crawler policy"). +# The catalogue is MIT-licensed and published to be reused: every plot page +# carries its runnable source, /llms.txt describes the site for agents, and an +# MCP server at https://api.anyplot.ai/mcp/ exists so assistants can query it +# directly. Declining the crawlers that make that content findable would +# contradict the point of publishing it. # -# Two ordering rules in here are load-bearing, both for the same reason — -# a compliant crawler resolves by specificity, simpler parsers by first match: +# So this file grants what the licence already grants. Retrieval, citation, +# search indexing and model training are all permitted, for every operator, +# without the retrieval-yes / training-no split this file used to draw. That +# split was never coherent here: the licence permits reuse including training, +# so declining a training crawler protected nothing while costing reach. +# +# It also had one concrete, unintended consequence. Google offers a single +# token, Google-Extended, that governs Gemini *grounding and training together* +# — there is no finer control. Declining it kept anyplot out of Gemini's +# answers entirely, which is the opposite of what a catalogue built for AI +# consumption wants. Applebot-Extended is training-only and is likewise no +# longer declined, now that training is permitted. +# +# One user agent remains declined, and for an operational reason rather than a +# policy one — see the group below. +# +# Cloudflare's AI Crawl Control enforces at the edge and can be stricter than +# this file: it answers blocked agents with HTTP 403 regardless of what is +# written here. Keep the dashboard in step with this file, or the published +# permission is not the one that applies. Verified edge state and how to check +# it: docs/reference/seo.md ("AI crawler policy"). +# +# Two ordering rules are load-bearing, both for the same reason — a compliant +# crawler resolves by specificity, simpler parsers by first match: # 1. Named groups come FIRST, `User-agent: *` last. With the wildcard on top, -# a first-match parser would read `Allow: /` and never reach the declining -# groups below. +# a first-match parser would read `Allow: /` and never reach the group +# below. # 2. Inside a group, `Disallow:` lines come BEFORE `Allow: /` — otherwise a # first-match parser lets /debug and /interactive through (verified with # Python's urllib.robotparser, which is exactly such a parser). - -# Content signals per contentsignals.org. ANY RESTRICTION EXPRESSED HERE IS AN -# EXPRESS RESERVATION OF RIGHTS UNDER ARTICLE 4 OF EU DIRECTIVE 2019/790. -# search: indexing and returning links/excerpts — yes -# ai-input: retrieval for AI answers, grounding, citation — yes -# ai-train: training or fine-tuning models — no -# -# The signal is repeated in EVERY group below, because a crawler reads only the -# group that matches it: a reservation declared once under `User-agent: *` would -# never reach a named agent — least of all the training collectors it is aimed -# at. # -# AI assistants, AI search and citation crawlers. These send readers back and -# are the reason /llms.txt exists; the *-User agents fetch only because a human -# asked their assistant to open the page. -User-agent: ClaudeBot -User-agent: Claude-User -User-agent: Claude-SearchBot -User-agent: OAI-SearchBot -User-agent: ChatGPT-User -User-agent: PerplexityBot -User-agent: Perplexity-User -Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Disallow: /debug -Disallow: /interactive -Allow: / +# Content signals per contentsignals.org: +# search: indexing and returning links/excerpts — yes +# ai-input: retrieval for AI answers, grounding, citation — yes +# ai-train: training or fine-tuning models — yes +# The signal is repeated in every group, because a crawler reads only the group +# that matches it. -# Training-only collectors: they take the catalogue without ever returning a -# reader, which is what ai-train=no says in prose. GPTBot sits here because it -# is OpenAI's TRAINING crawler — ChatGPT's retrieval path is -# OAI-SearchBot/ChatGPT-User above and stays open. Flip this one group if the -# training stance ever changes. -User-agent: GPTBot -User-agent: CCBot +# Bytespider is declined on bandwidth, not principle. It is documented as +# ignoring robots.txt and crawling far more aggressively than the catalogue can +# justify serving, so this line states the intent while Cloudflare's AI Crawl +# Control does the actual enforcing. If its behaviour changes, this group can +# go — nothing in the licence or the policy above argues against it. User-agent: Bytespider -User-agent: Amazonbot -User-agent: meta-externalagent -Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Disallow: / - -# Opt-out tokens for vendors that crawl under a different user agent -# (Googlebot / Applebot fetch for search and stay allowed). -User-agent: Google-Extended -User-agent: Applebot-Extended -Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference +Content-Signal: search=yes,ai-input=yes,ai-train=yes,use=reference Disallow: / -# Everyone else: search engines, social/link previews, feed readers. +# Everyone else: search engines, AI assistants and their training crawlers, +# social and link previews, feed readers. /debug and /interactive are app +# internals rather than catalogue content and are not worth crawling. User-agent: * -Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference +Content-Signal: search=yes,ai-input=yes,ai-train=yes,use=reference Disallow: /debug Disallow: /interactive Allow: / diff --git a/docs/reference/seo.md b/docs/reference/seo.md index 18ab570196..19a493ea01 100644 --- a/docs/reference/seo.md +++ b/docs/reference/seo.md @@ -293,41 +293,30 @@ Uses **MonoLisa** variable font (commercial, not in repo): ### Frontend (anyplot.ai) -Static file at `app/public/robots.txt`. It carries the full policy — content -signals, the welcomed AI agents, the declined training collectors — so it holds -regardless of what Cloudflare does or does not prepend (see -[AI crawler policy](#ai-crawler-policy)): - -Four groups in this order — welcomed AI agents (`ClaudeBot`, `Claude-User`, -`Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, -`Perplexity-User`), declined training collectors (`GPTBot`, `CCBot`, -`Bytespider`, `Amazonbot`, `meta-externalagent`), opt-out tokens -(`Google-Extended`, `Applebot-Extended`), and finally the wildcard. The first +Static file at `app/public/robots.txt`. It carries the full policy — the content +signals and the one declined agent — so it holds regardless of what Cloudflare +does or does not prepend (see [AI crawler policy](#ai-crawler-policy)): + +Two groups, in this order: `Bytespider`, declined on bandwidth grounds, then the +wildcard that allows everyone else. The retrieval-yes / training-no split this +section used to describe is gone — the policy is open to every operator, so the +named allow-groups it needed became redundant with `User-agent: *`. The declined group verbatim; read the file for the rest: ```txt -User-agent: ClaudeBot -User-agent: Claude-User -User-agent: Claude-SearchBot -User-agent: OAI-SearchBot -User-agent: ChatGPT-User -User-agent: PerplexityBot -User-agent: Perplexity-User -Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference -Disallow: /debug -Disallow: /interactive -Allow: / +User-agent: Bytespider +Content-Signal: search=yes,ai-input=yes,ai-train=yes,use=reference +Disallow: / ``` Three properties of that file are deliberate and should survive any cleanup: -- The `Content-Signal` line is repeated in **every** group, declining ones - included. A crawler reads only the group that matches it, so a reservation - declared once under `User-agent: *` never reaches a named agent — least of all - the training collectors it is aimed at. -- The named groups come **before** the wildcard group. A spec-compliant crawler - picks the most specific match regardless of order, but simpler parsers take - the first match and would read `Allow: /` and stop. +- The `Content-Signal` line is repeated in **every** group. A crawler reads only + the group that matches it, so a signal declared once under `User-agent: *` + never reaches a named agent. +- The named group comes **before** the wildcard. A spec-compliant crawler picks + the most specific match regardless of order, but simpler parsers take the + first match and would read `Allow: /` and stop. - Inside each group, `Disallow:` comes **before** `Allow: /` — same reason: with the broad allow first, a first-match parser (Python's `urllib.robotparser`, for one) hands out `/debug` and `/interactive`. @@ -338,9 +327,14 @@ Dynamic endpoint at `GET /robots.txt`: ```txt User-agent: * +Allow: /og/ Disallow: / ``` +`/og/` is the exception: every prerendered page references its preview image +there, so a blanket `Disallow` pointed crawlers at an image they were forbidden +to fetch. `Allow` comes first for the first-match parsers described above. + **Why block the API?** - APIs should not be indexed by search engines - Prevents crawling of debug endpoints, docs, and API responses @@ -348,58 +342,70 @@ Disallow: / ### AI crawler policy -**Decision (issue #9633):** AI agents that *retrieve and cite* are welcome; -agents that only *collect for training* are declined. anyplot's entire strategy -is to be consumable by AI agents — `/llms.txt`, the MCP server, MIT-licensed -code on every page — so blocking the retrieval side works against the product. -The training reservation stays expressed, and it is the legally load-bearing -part: `Content-Signal: ai-train=no` is an express reservation of rights under -Article 4 of EU Directive 2019/790. +**Decision (2026-08-18, supersedes issue #9633):** everything is open. Retrieval, +citation, search indexing and model training are permitted for every operator. + +The previous policy drew a retrieval-yes / training-no line and expressed the +training half as a reservation of rights under Article 4 of EU Directive +2019/790. That line was never coherent here: the catalogue is MIT-licensed and +published to be reused, so declining a training crawler protected nothing the +licence had not already given away, while costing reach. + +It also had a concrete, unintended consequence. Google offers a single token, +`Google-Extended`, that governs Gemini **grounding and training together** — +there is no finer control, verified against Google's crawler documentation. The +old policy declined it as if it were a training-only token, which kept anyplot +out of Gemini's answers entirely. `Applebot-Extended` genuinely *is* +training-only (retrieval and Siri answers ride on `Applebot`), so declining it +was correct under the old policy and is simply no longer wanted under this one. | Group | Agents | Policy | |---|---|---| -| Retrieval / citation / user-directed | `ClaudeBot`, `Claude-User`, `Claude-SearchBot`, `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `Perplexity-User` | allowed | -| Training collectors | `GPTBot`, `CCBot`, `Bytespider`, `Amazonbot`, `meta-externalagent` | declined | -| Opt-out tokens (vendor crawls under another UA) | `Google-Extended`, `Applebot-Extended` | declined | +| Everything | search engines, AI assistants, their index and training crawlers, social and link previews | allowed | +| Bandwidth exception | `Bytespider` | declined | +| App internals | `/debug`, `/interactive` for all agents | declined | -`GPTBot` is the deliberate borderline call: it is OpenAI's *training* crawler, -so it sits with the declined group, while ChatGPT's retrieval path -(`OAI-SearchBot`, `ChatGPT-User`) stays open. Reversing that is a one-group -edit in `app/public/robots.txt`. Vendor UA roles shift — re-check the current -role of each agent against Cloudflare's AI Crawl Control categories before -changing the policy. +`Bytespider` is the one exception and it is operational, not principled: it is +documented as ignoring robots.txt and crawls far more aggressively than this +catalogue can justify serving. The robots group states the intent; Cloudflare +does the enforcing. If its behaviour changes, the group can go. #### Cloudflare is the enforcement layer -Measured 2026-07-25 on the live zone: Cloudflare prepends a **managed -robots.txt** block (`Disallow: /` for ClaudeBot, GPTBot, CCBot, Google-Extended, -Amazonbot, Applebot-Extended, Bytespider, meta-externalagent, -CloudflareBrowserRenderingCrawler) *and* answers those user agents with a hard -`HTTP 403 Your request was blocked.` at the edge — including `Claude-User` and -`ChatGPT-User`, and including `/llms.txt` itself. The file written for AI agents -was unreachable to every agent it was written for. Googlebot passes (200). +The edge can be stricter than this file and answers blocked agents with `HTTP +403` regardless of what is written here — so **a permission granted in +`robots.txt` that the dashboard still blocks is a published lie**. Keep the two +in step. + +Measured on the live zone 2026-08-18 (zone `anyplot.ai` → **AI Crawl Control** → +Security): + +| State | Agents | +|---|---| +| Blocked at the edge | `GPTBot`, `CCBot`, `Amazonbot`, `meta-externalagent`, `Bytespider`, `Google-CloudVertexBot`, `PetalBot`, `Anchor Browser`, `Arquivo Web Crawler` | +| Passing | `Googlebot`, `bingbot`, `Baidu`, `Applebot`, `Claude-User`, `ClaudeBot`, `Claude-SearchBot`, `ChatGPT-User`, `OAI-SearchBot`, `PerplexityBot`, `Perplexity-User`, `DuckAssistBot`, `MistralAI-User`, `Meta-ExternalFetcher`, `archive.org_bot` | -`api.anyplot.ai` is **not** covered by the block (ClaudeBot gets 200 there), so -the MCP server stayed reachable. +Two things to note. `Google-CloudVertexBot` is blocked at the edge and appears in +no repo-side policy — an undocumented decision that only exists as a dashboard +toggle. And unlike the 2026-07-25 measurement, Cloudflare is **no longer +prepending a managed robots.txt block**: the live file is byte-identical to this +repo's, and enforcement is purely the 403. -Aligning the edge with this policy is a dashboard action (zone `anyplot.ai` → -**AI Crawl Control** / Bots): allow the retrieval group, keep the training -group blocked, and either turn off the managed robots.txt (this repo's file -already carries the content signals) or leave it on and accept that the live -file is stricter than the repo's. +To bring the edge in line with the policy above, unblock everything except +`Bytespider` in the dashboard. Verify afterwards: ```bash -curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/llms.txt # expect 200 -curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; CCBot/2.0; +https://commoncrawl.org/faq/)" https://anyplot.ai/ # expect 403 +curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://anyplot.ai/ # expect 200 once unblocked +curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; Bytespider)" https://anyplot.ai/ # expect 403 curl -sA "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://anyplot.ai/scatter-basic | grep -o '[^<]*' # per-route title, not the SPA shell ``` The last command is the part this repo owns: `app/nginx.conf` maps the AI UAs onto the seo-proxy path, and `.github/workflows/bot-serving-check.yml` guards it daily against the Cloud Run origin (origin, not edge — so it reports on the -nginx map no matter what the zone policy is). +nginx map no matter what the zone policy is, and will never catch edge drift). ## Sitemap