Agent Scrape Kit 🕷️

Systematic web scraping for AI agents. Two LLM-driven skills that teach agents to do senior-level scraping: discover how a site works, then extract data deterministically.

scraper-discover  →  blueprint.json  →  scraper-run  →  structured_data
   (learn once)                           (run forever)

No browser in the hot path. No LLM tokens on repeat runs. Just clean data extraction.

---

Why This Exists

Most scraping is done ad-hoc: open browser, find the element, extract, repeat for the next site. This works but doesn't scale. Every run pays the same discovery cost.

This kit flips the model:

1. scraper-discover — spends LLM time once to reverse-engineer the site. Produces a blueprint describing the exact API endpoints, auth flow, anti-bot measures, and pagination.

2. scraper-run — reads the blueprint and executes deterministically. No browser. No LLM. Just HTTP calls + data extraction + rate limiting + checkpointing.

The result: you pay for exploration once and get reliable extraction forever.

---

The Skills

scraper-discover — Site Reconnaissance

Systematically classifies a website's architecture and discovers the fastest data extraction path.

What it does:

• Classifies the site into 6 archetypes (SSR, SPA REST, SPA GraphQL, static commerce, anti-bot shielded)
• Probes API endpoints directly (no browser when possible)
• Captures network traffic via Playwright when needed
• Detects auth mechanisms, CSRF tokens, session cookies
• Fingerprints anti-bot protection (Cloudflare, Datadome, PerimeterX, Akamai)
• Identifies pagination patterns (cursor, offset, page number, infinite scroll)
• Validates the blueprint by extracting at least 5 records
• Saves a reusable blueprint JSON

Scripts:

• scripts/recon_capture.py — Playwright-based network capture
• scripts/detect_antibot.py — Anti-bot fingerprinting

scraper-run — Deterministic Extraction

Executes a blueprint to extract all data from a site. No browser. No LLM. Just extraction.

What it does:

• Loads the blueprint, validates it
• Executes pagination automatically (cursor, offset, page number)
• Handles auth refresh (session cookies, JWT, CSRF)
• Respects rate limits (built-in throttling + backoff)
• Checkpoints progress every 500 records
• Self-heals: detects stale endpoints, response shape changes, anti-bot escalation
• Produces clean JSON + CSV output

---

Usage

1. Discover a site

# Scraper-discover handles this conversationally.
# Give it a URL and what you want to extract:
"scraper-discover: scrape all products from target.com, get name, price, SKU"

2. Run the blueprint

# Once the blueprint exists, scraper-run extracts everything:
"scraper-run: extract all products using the target.com blueprint"

3. Monitor changes (optional)

# Cron job setup:
"schedule scraper-run on target.com every 24h and notify me of changes"

---

Blueprint Format

Blueprints are saved to ~/.hermes/scraper/blueprints/<domain>.json.

{
  "site": "target.com",
  "archetype": "spa_with_api",
  "tech_stack": {
    "framework": "nextjs",
    "api": "rest",
    "auth": "session_cookie + csrf_token",
    "anti_bot": "none"
  },
  "endpoints": [{
    "name": "list_products",
    "method": "GET",
    "url": "https://target.com/api/v2/products",
    "pagination": {
      "type": "cursor",
      "request_param": "cursor",
      "response_path": "$.meta.next_cursor"
    },
    "extraction": {
      "data_path": "$.data.items",
      "fields": ["id", "name", "price", "image_url"]
    },
    "rate_limit": {
      "max_rps": 2.0,
      "concurrency": 1
    }
  }],
  "quality": {
    "proven_working": true,
    "validated_records": 5
  }
}

Full schema: schemas/blueprint-v1.json

---

Architecture Anti-Patterns This Avoids

Anti-Pattern	How This Kit Fixes It
Re-extracting via browser every time	Replays the API directly — browser is discovery only
LLM in the hot path	Zero tokens during extraction
Hardcoded selectors that break	Blueprints are self-healing — failures trigger re-discovery
No rate limiting	Blueprint captures limits, runner enforces them
No checkpointing	Every 500 records saved, resume from failure
Mixing discovery and execution	Separate skills with clear boundaries

---

Directory Layout

~/.hermes/scraper/
├── blueprints/          # Reusable site blueprints (JSON)
│   └── target.com.json
├── data/                # Extracted data (JSON/CSV)
│   └── target.com_20250520.json
├── checkpoints/         # Checkpoints for resume
│   └── target.com_latest.json
└── skills/
    └── web-scraping/
        ├── scraper-discover/  # Discovery logic
        └── scraper-run/       # Execution logic

---

License

MIT — use it, modify it, ship it.