fix(seo): close the gaps an independent audit found

[MarkusNeusinger]

Three findings from a fresh-context audit of today's twelve PRs. I had spotted none of them.

1. The documentation contradicted itself

docs/reference/seo.md's bot-map section still said gptbot, meta-externalagent and amazonbot were "declined in robots.txt", and app/nginx.conf repeated the claim in a comment. Both were written about an hour before #10474 opened the policy, and neither was reconciled — so the page asserted the old policy three screens from the section declaring the new one.

The measured edge-state table was stale the other way round: the dashboard unblock it prescribed had since been carried out, so the table described a state that no longer existed. It now records what is actually blocked (Bytespider, TikTok Spider, and three agents whose rule-compliance is unverified rather than disproven) and says plainly that bot-serving-check tests the origin and will never catch edge drift.

2. A database outage would reopen #10453

With no catalogue to check against, the bot routes answered 200 with a fabricated, self-canonicalising page for any string — the precise defect #10453 removed, surviving in degraded mode.

Degraded pages now carry noindex. I first tried returning 503, which is arguably more correct, and backed it out: it broke eleven tests that use the no-DB path as a rendering harness. noindex keeps the behaviour those tests depend on and removes the indexing risk, which is the part that matters. The path is unreachable in production — but "unreachable" here means one misconfiguration away from indexable.

3. 404s were counted as successful page reads

bot_fetch ran as a router dependency. A dependency executes before the handler and cannot see the response, so every miss was recorded as a read.

It has moved to a middleware and gained a status property. Recording the miss is right — an assistant asking for a URL that no longer exists is how a library migration announces itself — but recording it as a page view is a lie. Filter on status before reading anything else; documented in docs/reference/plausible.md.

Also

A docstring pointed at app/src/router.tsx, which does not exist. Routing lives in app/src/routes/index.tsx.

Verification

• pytest tests/unit — 1640 passed, including: degraded hub and impl pages assert noindex, a companion test asserts normal pages do not, and a bot_fetch test pins status: "404" on a miss
• ruff check + ruff format --check — clean
• grep confirms no remaining reference to the superseded policy in seo.md, nginx.conf or robots.txt

Not in this PR

The audit's other findings are handled elsewhere: the ten-day-red monitor in #10478, and the analytics that recorded nothing in #10477.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR addresses several SEO/analytics correctness gaps: it prevents indexable fabricated bot pages during DB-degraded serving, fixes bot_fetch analytics so misses aren’t recorded as successful reads, and reconciles documentation/comments with the current crawler policy and measured edge state.

Changes:

• Add noindex to bot-served SEO pages when the DB is unavailable to avoid indexing near-duplicate “invented” URLs.
• Move bot_fetch recording from a router dependency to an HTTP middleware and record the response status for accurate analytics.
• Update tests, docs (SEO + Plausible), nginx comments, and CHANGELOG.md to reflect the corrected behavior and current policy.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
api/main.py	Adds middleware to record bot_fetch after response so status can be captured.
api/analytics.py	Extends track_bot_fetch to accept/record HTTP status in Plausible props.
api/routers/seo.py	Adds optional noindex meta injection for DB-degraded bot HTML responses; removes router dependency tracking.
tests/unit/api/test_routers.py	Updates router tests to patch middleware location and asserts degraded pages are noindex.
tests/unit/api/test_analytics.py	Updates analytics unit tests for the new status argument/prop and adds a 404-status assertion.
docs/reference/seo.md	Reconciles crawler mapping text with the open policy and updates the measured edge-state table.
docs/reference/plausible.md	Documents bot_fetch as middleware-based and adds the new status property + filtering guidance.
app/nginx.conf	Updates comment language about mapping vs permission under the current policy.
CHANGELOG.md	Adds an Unreleased “Fixed” entry describing the outage/noindex and bot_fetch status corrections plus doc fixes.

Suppressed comments (1)

api/analytics.py:311

• The path arg description still says “being read”, but the event is now explicitly a request that may be a 404/redirect/etc. Tweaking this text will prevent misinterpretation when analyzing bot_fetch events.

        path: Public path being read, e.g. "/box-basic/python/matplotlib"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[MarkusNeusinger]

Both applied — the wording was left over from when this was a router dependency that only ever saw successful handler runs. docs/reference/plausible.md already described the new semantics, so the code comments were the last place still claiming 'reads'.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

CHANGELOG.md:241

• The changelog entry describes the risk as a “database outage”, but the code paths that return the degraded HTML (and therefore need noindex) are used when the DB is not configured (optional_db returns None only when is_db_configured() is false). Rewording to “degraded/no-database mode” (or “DB misconfiguration”) would better match the actual behavior.

- **A database outage would have reopened the hole #10453 closed** — with no catalogue to check
  against, the bot routes answered 200 with a fabricated page for any string, self-canonicalising,
  exactly the defect that PR removed. Degraded pages now carry `noindex`. The path is unreachable in
  production, where the database is configured, but it is one misconfiguration away from indexable.
  `bot_fetch` also moved from a router dependency to a middleware and gained a `status` property: a