Enable the captures PR #25 substituted with doc examples - #26
Conversation
Reviewing #25 surfaced two providers where doc-sourced samples were added despite this repo already having capture automation for them, and one publishing gap. This is the groundwork so those captures can be run instead. Scrapfly Crawler API: Crawler deliveries set X-Scrapfly-Webhook-Resource-Type to a constant `crawler` and carry the event name in X-Scrapfly-Crawl-Event-Name, so the receiver wrote every crawler event to one crawler.json. That is what made the events look uncapturable. topic_identifier now accepts an ordered list and requestReceiver takes the first key that resolves, so crawler deliveries resolve by event name while scrape and extraction still fall through to resource-type. capture.ts gains a /crawl trigger covering the four events a clean crawl emits; the four that only fire on a failed or interrupted crawl are excluded from the delete-then-wait cycle so a normal run can't destroy a sample it cannot recreate. Shopify 2026-07: providers/shopify is pinned to 2024-10, which stopped being accessible on 2025-10-16. Adds scripts/shopify/2026-07 alongside the 2024-10 harness: 217 topics extracted from the current docs (22 gone since 2024-10, 31 new), version taken from the directory name, app path and base URL via env instead of editing the file, and the off-by-one that triggered an undefined topic on the last iteration fixed. latest_version stays 2024-10 until the capture has actually run -- compile publishes latest_version and the version list separately, so pointing at an empty version would break consumers. compile.ts: doc_sourced was read from index.json but never published, so consumers could not tell a doc-sourced provider from a captured one without downloading a version file. It now reaches providers.json. Verified: yarn compile passes; receiver resolves crawler, scrape, shopify header and monday body identifiers correctly, and no longer throws on a null body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjdTHaFed88Xd6fBFbJTKf
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…ocked them Running `yarn capture:scrapfly` for the first time surfaced three problems that made the crawler captures impossible, all now fixed: - `/crawl` takes its config as a JSON body with only `key` in the query string. Passing it as query params returned HTTP 400 "Invalid JSON payload", so the trigger never queued a job. - Omitting `webhook_events` does not subscribe to every crawler event. Scrapfly defaults to the four lifecycle events, excluding the per-URL ones, so `crawler_url_*` never arrived. All eight are now named explicitly. - `parseJobId` only read `context.job.uuid`, which is the `/scrape` shape. `/extraction` and `/crawl` return a flat `job_uuid`, so both logged "job (unknown)". `crawler_url_skipped` moves into the clean-crawl set: a crawl bounded by `max_depth` filters every link past the boundary, and the captured `crawler_finished` reports 30 URLs skipped against 2 visited. Only `crawler_url_failed`, `crawler_stopped` and `crawler_cancelled` remain conditional. Also: the delivery window is now overridable via `SCRAPFLY_CAPTURE_TIMEOUT_MS` (default unchanged at 120s), since a crawl emits `crawler_finished` only once the job completes; and expected files are snapshotted before the run and restored if they fail to arrive, so a failed capture can no longer destroy a good sample. The README's claim that crawler deliveries set the resource-type header to `crawler` is corrected — real deliveries send `crawl`. `scrape.json` and `extraction.json` are unchanged. Scrapfly is currently not scheduling any queued scrape or extraction job attached to a webhook; they sit at "Pending to be scheduled" indefinitely. This reproduces on a webhook created from scratch and has been reported to Scrapfly. The crawler path is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019pce9oHWGjdwHsJNya1ovP
Update: captures run, three blockers fixedRan The
Five crawler samples are now committed: Two smaller fixes came out of the run: Also corrected in scrape.json and extraction.json are unchanged, deliberatelyScrapfly is currently not scheduling any queued scrape or extraction job attached to a webhook. They are accepted with a job UUID and then sit at This is not something in this repo:
Reported to Scrapfly. The May samples remain valid, and |
All three findings from @leggetter's review, plus the sweep it asked for. **shopify — 29 files removed.** They went in `2024-10/` because that is the provider's latest_version, but they were read from shopify.dev/docs/api/webhooks/latest, which serves the current stable version. 2024-10 stopped being accessible on 2025-10-16, so they could not have been sourced at it, and 15 of the 29 are inventory_transfers/* and inventory_shipments/* topics that postdate it. The 181 captures in that directory all carry `x-shopify-api-version: 2024-10`; these carried no version header at all. `scripts/shopify/` can capture properly. **scrapfly — 7 crawler files removed**, and providers/scrapfly/README.md is back to its original text. `scripts/scrapfly/` can capture these once requestReceiver.ts resolves an ordered topic_identifier list, which hookdeck#26 adds. Scrapfly documents 8 crawler events, not 7 — `crawler_url_visited` was missing here, which is its own argument for capturing rather than hand-writing them. **Six providers move out of `latest/`:** bridge-xyz `2024-01-01`, chargebee `v2`, docusign `v2.1`, klaviyo `2025-07-15`, microsoft-graph `v1.0`, smartcar `4.0`. The sweep across all 104 found that "has a version field" is too broad a rule — most providers have one and most do not qualify. Object versions (square, courier, github's `package_version`), event-schema versions (checkout's 1.0.0/1.2.0/2.0.0, zendesk, twitch's `subscription.version`) and asset versions (cloudinary's unix timestamp) version the thing carried, not the contract carrying it. The test that holds is whether the field versions the webhook API itself: `api_version`, `apiVersion`, or a field the vendor documents as such — Smartcar's `meta.version` is "Webhook API version" in their docs. Two left in `latest/` deliberately: huggingface's bare `version: 3` has no doc statement of what it versions, and notion's `api_version` appears on 4 of 23 samples. 1,661 -> 1,625 samples across 102 providers. `yarn compile` passes with 112 providers, 0 topic collisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 217 topics documented at 2026-07 triggered via the Shopify CLI and
captured through Hookdeck. Nothing was skipped — every topic in
topics.txt produced a real delivery, each carrying
`x-shopify-api-version: 2026-07`.
`latest_version` moves from 2024-10 to 2026-07 now that the directory
has samples in it. 2024-10 stopped being accessible on 2025-10-16, so
the previous value pointed at a version Shopify no longer serves.
Eleven samples have a `{}` body: Shopify's trigger sends no payload for
those topics. That's the same behaviour as 2024-10, which has seven,
and `shipping_addresses/*` are empty in both. The headers are real
either way.
The 2026-07 README gains the two things that cost time in this run:
the CLI destination path must be `/` because trigger.js appends
`/shopify/<version>` to the address itself — passing `--path` the way
scripts/scrapfly does doubles it and every delivery 404s, and the value
is stored server-side so restarting `hookdeck listen` won't clear it —
and `shopify app webhook trigger` starts a device-code login when the
CLI isn't authenticated, which would stall an unattended run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019pce9oHWGjdwHsJNya1ovP
Update: Shopify 2026-07 captured,
|
`doc_sourced` was read from index.json but never published until this branch. Nothing consumes it yet, so this is the cheapest moment to change the shape — after #25 lands and stamps 104 providers it becomes a migration. Three problems with the boolean: `config.doc_sourced === true` coerced absence to `false`, and `false` is not "unknown" — it is a positive claim that the samples were captured live. Every provider in the repo would have started asserting that the moment the field was published, including ones nobody has verified. `provenance` defaults to `unknown` instead, and an unrecognised `sourced_via` fails the build rather than reaching consumers looking authoritative. Provenance is a property of a version, not a provider. `providers/ scrapfly` alone is mixed: the crawler samples were captured today, the scrape and extraction samples in May, and `screenshot` can never be captured at all because Hookdeck rejects Scrapfly's content-type. One boolean on the provider cannot express that. And docs are not equivalent to capture. Scrapfly's crawler docs state the resource-type header is `crawler`; real deliveries send `crawl`. Samples transcribed from those docs encode a value that does not exist in production. Flattening both into one "verified" bit would hide exactly the discrepancy a consumer needs to see. `sourced_on` records when a version's samples were obtained, using the oldest date where a version is mixed so it never overstates freshness. Recorded for shopify/2026-07 and scrapfly/latest, both captured first-hand. shopify 2023-01 and 2024-10 are left `unknown` rather than inferred from commit dates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019pce9oHWGjdwHsJNya1ovP
hookdeck#26 replaced `doc_sourced` with a `provenance` block keyed by version. 93 index.json files move to it; no payload changes — the per-sample `source` key was already the version-independent half of the same idea. "provenance": { "latest": { "sourced_via": "docs", "sourced_on": "2026-07-29" } } All 1,625 samples here carry `retrieved: 2026-07-29`, so that is the `sourced_on` for every version, and the "oldest where files span dates" rule has nothing to disambiguate yet. The 11 providers that merge into an existing entry deliberately get no `provenance`. After merging, those versions hold captures and doc examples together, and `sourced_via` has no value for "both" — so no claim is the honest record, and it reads as `unknown` rather than as a false `capture`. Merged main, which brings hookdeck#26's captures. Re-checked the additive rule against it: 1,625 doc-sourced files across 102 providers, 0 of them sitting where main now has a capture. shopify and scrapfly were already removed from this branch in the previous commit, which is what hookdeck#26 then captured for real — including `crawler_url_visited`, the eighth crawler event this PR's hand-written set was missing. `yarn compile` passes under hookdeck#26's stricter compile.ts, which now validates `sourced_via` and fails an unrecognised value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 participants
Follow-up work identified while reviewing #25. None of #25's own fixes are here — those are Gareth's to make on that branch.
Two of the providers #25 added doc-sourced samples for already have capture automation in this repo. This is the groundwork so the captures can actually be run, plus one publishing gap.
Scrapfly Crawler API
#25 hand-wrote 7 crawler payloads from Scrapfly's docs rather than using
scripts/scrapfly/. There was a real reason: crawler deliveries setX-Scrapfly-Webhook-Resource-Typeto a constantcrawlerand put the event name inX-Scrapfly-Crawl-Event-Name, sorequestReceiver.tscollapsed every crawler event into a singlecrawler.json. That made them look uncapturable.topic_identifiernow accepts an ordered list, andrequestReceiver.tstakes the first key that resolves. Crawler deliveries resolve by event name; scrape and extraction don't carry that header and fall through to resource-type exactly as before.capture.tsgains a/crawltrigger (page_limit=3,max_depth=1,webhook_eventsunset so it subscribes to everything).crawler_url_failed,crawler_url_skipped,crawler_stoppedandcrawler_cancelledonly fire on a crawl that fails or is interrupted. They're excluded from the delete-then-wait cycle so a normal run can't destroy a sample it can't recreate — the script scrubs and reports one if it happens to land.topic_identifieris internal torequestReceiver.ts—compile.tsnever publishes it — so the list form doesn't reach consumers.Someone with the Scrapfly key needs to run
yarn capture:scrapflyto land the actual files.Shopify 2026-07
providers/shopifyis pinned to2024-10, which stopped being accessible on 2025-10-16. Addsscripts/shopify/2026-07/alongside the existing 2024-10 harness, which stays as the record of how those samples were captured.topics.txt— 217 topics extracted from the current docs. 22 are gone since 2024-10 (purchase_orders/*,suppliers/*,translatable_content/*,checkouts/paidand others), 31 are new.trigger.js— API version read from the directory name, app path and base URL from env instead of editing the file, and a fix for the off-by-one that triggered anundefinedtopic on the last iteration.latest_versiondeliberately stays2024-10.compile.tspublisheslatest_versionand the version list separately, so pointing at a version with no samples in it would break consumers. It gets bumped when the capture runs — that's step 2 in the new README.compile.ts
doc_sourcedis read fromindex.jsonbut never published, so a consumer couldn't tell a doc-sourced provider from a captured one without downloading a version file. One line; it now reachesproviders.json. No effect until #25 lands and there are doc-sourced providers to flag.Testing
yarn compilepasses. Receiver checked against a scratch provider tree:crawler_started.jsonscrape.jsonorders.create.jsoncreate_pulse.jsonuntitled-<hash>.jsonnullbody with identifier configuredThe two
tsc --noEmiterrors inscripts/bigcommerceandscripts/scrapfly/lib.tsare pre-existing onmainand untouched here.🤖 Generated with Claude Code
https://claude.ai/code/session_01XjdTHaFed88Xd6fBFbJTKf
Generated by Claude Code