[geo-optimizer] Add robots.txt and llms.txt to docs site to unlock AI bot crawl permissions

[github-actions]

GEO Improvement: Add robots.txt and llms.txt to Docs Site

Source audit: GEO Audit Report — 2026-07-17
Audit date: 2026-07-17
Run: §29597253783

Finding

"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)" — docs-site-audit.json recommendation #2

"Create /llms.txt for AI indexing: geo llms --base-url https://github.github.com/gh-aw. Note: llms.txt is an organizational signal, not a proven ranking factor. It helps structure content for AI systems." — docs-site-audit.json recommendation #3

The docs site (github.github.com/gh-aw/) scores 0/18 on both robots.txt and llms.txt checks. All 20 audited pages are capped around 37–44/100 because these two files are missing site-wide.

Why this matters

Without a robots.txt, AI crawlers like GPTBot and ClaudeBot have no explicit crawl permission, which suppresses ChatGPT citation scores to 50/100. Adding both files would contribute up to +36 points combined toward the docs site GEO score (from 44 → ~80/100), moving it from Foundation into the Good band and significantly improving citation probability on ChatGPT, Perplexity, and Google AI Overview.

Suggested fix

1. Generate robots.txt for the docs site:

   geo robots --base-url https://github.github.com/gh-aw

   Deploy the output as public/robots.txt in the docs site source (likely the Astro project).

2. Generate llms.txt for the docs site:

   geo llms --base-url https://github.github.com/gh-aw

   Deploy the output as public/llms.txt.

3. Commit and deploy — all 20 sitemap pages will immediately benefit since both files are served at the root.

Generated by 🌍 GEO Optimizer Daily Audit · 44.7 AIC · ⌖ 8.92 AIC · ⊞ 4.9K · ◷

• expires on Jul 24, 2026, 8:53 AM UTC-08:00