Requestory

Your requests tell a story

Load a page in a real browser, record every network request it makes, and surface the hidden-but-public APIs and data you can actually reuse.

I can defeat bot detection on any website, but I shouldn't have to do so. The purpose of Requestory is to handle a lot of the busywork involved in trying to use the Internet in a more practical way and thus uncover data that could potentially save hours on a per-project basis. It's the DevTools Network tab with an opinion: which of these requests is a reusable API, and can I call it without authentication

Requestory only inspects requests your browser would make anyway. The endpoints it surfaces are public by design, the same way an unlisted YouTube video is reachable by anyone with the link.

What it finds

Each detector maps to a real pattern (and to the examples that motivated the project):

Finding	What it is
Hidden JSON APIs	Background XHR/fetch returning structured JSON from API-shaped URLs
Data hidden in HTML	JSON embedded in the page itself - __NEXT_DATA__, JSON-LD, window.<state> = {…}
GraphQL	GraphQL endpoints, with operation names and query/mutation type extracted
Upscalable images	Images served downscaled via tunable ?w=/?q=//256x256/ knobs

Every finding comes with ready-to-run curl / Python / fetch snippets and a Test public access button that re-runs the request server-side with cookies and auth headers stripped - if it still returns data, the endpoint is public and reusable from anywhere.

How it works

URL -> capture.py -> analyze.py -> replay.py -> server.py / __main__.py

• capture drives headless Chromium (Playwright) so JavaScript runs and the page makes the same calls a real visitor would, recording every request/response pair. It only downloads bodies worth analyzing (JSON/HTML/XML), skipping JS and CSS bundles so heavy pages process in seconds.
• analyze scores and categorizes each request. Scoring rewards reusability - a clean public JSON endpoint ranks near the top. Embedded data is pulled and de-duplicated so the same blob shipped from several URLs collapses to the root-most one and the same endpoint hit repeatedly with different session tokens or search terms collapses to one reusable entry. Empty ([]/{}) and 4xx/5xx responses are dropped.
• replay generates snippets and runs the credential-stripped public test.
• server / CLI are two thin frontends over the same pipeline, streaming capture progress in real time.

Setup

Initial setup:

make install && make browser

Web App:

make serve        # http://localhost:8000

Open the page, paste a URL, hit Analyze.

CLI - two modes:

# Discovery: find hidden APIs on a page
cd backend && PYTHONPATH=. python3 -m requestory https://www.npmjs.com/package/react
python3 -m requestory analyze https://example.com --json --static

# Monitoring / self-healing: is a known endpoint still alive and public?
python3 -m requestory check https://api.site.com/v2/feed --expect-json --contains '"items"'
# exit code 0 = working, 1 = broken

The check command is the building block for the monitoring ideas: it replays a known request (credentials stripped by default) and sets its exit code by the result. A scraper can wrap its own requests so that when one starts failing it runs analyze --json, picks the new endpoint, and updates itself - "self-healing" against site redesigns. Or a cron job can check an endpoint on a schedule and alert when it's been broken long enough to mean the API changed.

As a library:

import asyncio
from requestory import capture_url, analyze

async def main():
    report = analyze(await capture_url("https://example.com"))
    for f in report.findings[:10]:
        print(f.score, f.category, f.request.url)

asyncio.run(main())