From aa58f2ac83e43723514bf2efb82eb4442f5b49c6 Mon Sep 17 00:00:00 2001 From: Vivek Date: Thu, 30 Jul 2026 15:58:26 +0530 Subject: [PATCH 1/3] ci: gate the CDN purge on the real Railway deployment status The first version inferred the deploy by watching the origin's /__webjs/version uptime for a restart, on the assumption that every push to main produces a Railway deploy. It does not. Railway marked #1189's own merge commit SKIPPED (it touched only .github and a .md), so no restart ever happened, the wait ran its full 15 minute budget, and the job failed. Left alone it would red-X every docs-only and CI-only commit, and a job that cries wolf on routine commits stops being read. Ask Railway instead of guessing. Its GraphQL API returns, per deployment, both a status and the originating meta.commitHash, so the job now resolves the deployment for github.sha and branches on the real status: SUCCESS purges, SKIPPED exits 0 with a notice (nothing was deployed, so nothing is stale), FAILED / CRASHED / REMOVED exit 0 with a warning (no new content reached the origin), anything in progress keeps polling. An unresolved deployment now warns and leaves the run GREEN rather than failing, which is the whole point. The purge and verify steps become conditional on that outcome, and workflow_dispatch still purges immediately with no wait. Two API details worth keeping: Railway sits behind Cloudflare, which answers `error code: 1010` to automated-looking user agents (python urllib is blocked outright, curl is fine), and a GraphQL error arrives with HTTP 200 plus an `errors` array, so the status code alone proves nothing. Token type is not knowable ahead of time either, since a project token uses Project-Access-Token and an account token uses Authorization: Bearer, so both are tried. Verified by exercising the extraction and branching against the real response shape for both known commits: 93883bbc resolves to SKIPPED (the commit that broke the previous version, now a clean no-op) and 086b3c75 to SUCCESS. Closes #1192 --- .github/workflows/purge-cdn.yml | 170 +++++++++++++++++++++----------- framework-dev.md | 18 +++- 2 files changed, 130 insertions(+), 58 deletions(-) diff --git a/.github/workflows/purge-cdn.yml b/.github/workflows/purge-cdn.yml index e23b0aae2..f1d3b6a5a 100644 --- a/.github/workflows/purge-cdn.yml +++ b/.github/workflows/purge-cdn.yml @@ -1,6 +1,7 @@ name: Purge CDN -# Evicts the Cloudflare edge cache after a site deploy reaches the origin. +# Evicts the Cloudflare edge cache after a site deploy actually reaches the +# origin. # # Why this exists: the site's static assets (/public/tailwind.css, the brand # SVGs) are served with `cache-control: public, max-age=14400` at STABLE urls, @@ -9,19 +10,30 @@ name: Purge CDN # pre-redesign stylesheet after #1179, then the un-fixed logo marks after # #1185), each needing a manual dashboard purge somebody had to remember. # -# Why it waits instead of purging on push: Railway auto-deploys from a push to -# main and the build takes minutes. Purging immediately would evict the cache -# while the origin still serves the OLD bytes, the next visitor would -# repopulate the cache with exactly those old bytes, and the site would be -# stale for another four hours with nothing to show for the run. So the job -# confirms the new build is live on the ORIGIN first, and fails loudly rather -# than purging if it never shows up. +# Why it waits instead of purging on push: Railway builds take minutes, and +# purging before the new bytes are live would evict the cache while the origin +# still serves the OLD ones, so the next visitor repopulates the edge with +# exactly those, leaving the site stale for another four hours. # -# Required secret: CLOUDFLARE_API_TOKEN, scoped to Zone / Cache Purge / Purge -# on the webjs.dev zone only (not an account-wide token). See framework-dev.md. +# Why it asks Railway rather than watching the origin: the first version of +# this workflow inferred the deploy from process uptime, assuming every push to +# main produces one. It does not. Railway marked #1189's own merge commit +# SKIPPED (it touched only .github and a .md), no restart ever happened, and +# the job failed after a 15 minute wait. Inferring the deploy makes every +# docs-only commit a red X, and a job that cries wolf on routine commits is +# worse than no job. So this asks Railway for the deployment matching the +# pushed sha and reads its real status (#1192). +# +# Required secrets: +# RAILWAY_TOKEN a Railway PROJECT token for this project's production +# environment (narrower than an account token). +# CLOUDFLARE_API_TOKEN scoped to Zone / Cache Purge / Purge on the webjs.dev +# zone only. +# Both are REPOSITORY secrets: this job declares no `environment:`, so an +# environment-scoped secret would arrive empty. See framework-dev.md. # # Manual purge: run this workflow from the Actions tab (workflow_dispatch), -# which skips the deploy wait and purges immediately. +# which skips the deploy wait entirely and purges immediately. on: push: @@ -43,77 +55,122 @@ jobs: purge: name: Wait for the deploy, then purge runs-on: ubuntu-latest - # Comfortably above the wait step's own 15-minute budget, so the step's - # explanatory failure always fires before the job-level kill (which would - # leave no diagnosis in the log). + # Comfortably above the wait step's own budget, so the step's explanatory + # exit always happens before the job-level kill (which would leave no + # diagnosis in the log). timeout-minutes: 25 env: - # A zone id identifies a domain, it is not a credential, so it lives in - # the workflow rather than costing a second required secret. + # Ids identify resources and are not credentials, so they live in the + # workflow rather than costing extra secrets. ZONE_ID: 46692b879a06f3a6a987b99915560393 - # The Railway service domain for @webjsdev/website. Polled directly so - # the readiness check bypasses the very cache this job is about to - # purge (asking webjs.dev could be answered from the edge). + RAILWAY_API: https://backboard.railway.com/graphql/v2 + RAILWAY_PROJECT_ID: 1d05ab75-b64e-4920-94ef-1ca7d04778fc + RAILWAY_ENVIRONMENT_ID: 2e30b2f6-d166-48d3-9ae7-17176fb0942f + RAILWAY_SERVICE_ID: 2eb044b7-8698-43d3-959b-37c8925eb653 + # The Railway service domain for @webjsdev/website, used only by the + # post-purge smoke check so it can compare the edge against the origin. ORIGIN: https://webjs.up.railway.app - # How long to wait for the deploy before giving up, in seconds. + # How long to wait for the deployment to reach a terminal status. WAIT_BUDGET: 900 steps: - - name: Wait for the new build to be live on the origin + - name: Resolve the Railway deployment for this commit + id: deploy if: github.event_name == 'push' + env: + RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }} + SHA: ${{ github.sha }} run: | set -euo pipefail + if [ -z "${RAILWAY_TOKEN:-}" ]; then + echo "::error::RAILWAY_TOKEN is not set. Add it as a REPOSITORY secret" + echo "(a Railway project token for the production environment)." + exit 1 + fi + + # Railway's API sits behind Cloudflare, which answers `error code: + # 1010` to clients whose user agent looks automated (python urllib is + # blocked outright), so this must go through curl. + # + # A project token authenticates with `Project-Access-Token`, an + # account token with `Authorization: Bearer`. Which one applies + # depends on the token the owner created, so try both and keep + # whichever answers without a GraphQL error. + query='query deployments($input: DeploymentListInput!, $first: Int) { deployments(input: $input, first: $first) { edges { node { id status createdAt meta } } } }' + payload=$(jq -n --arg q "$query" \ + --arg p "$RAILWAY_PROJECT_ID" --arg e "$RAILWAY_ENVIRONMENT_ID" --arg s "$RAILWAY_SERVICE_ID" \ + '{query: $q, variables: {first: 30, input: {projectId: $p, environmentId: $e, serviceId: $s}}}') + + railway_query() { + local out + for hdr in "Project-Access-Token: ${RAILWAY_TOKEN}" "Authorization: Bearer ${RAILWAY_TOKEN}"; do + out=$(curl -sS --max-time 20 -X POST "$RAILWAY_API" \ + -H "$hdr" -H "Content-Type: application/json" --data "$payload" 2>/dev/null || true) + # A GraphQL error comes back with HTTP 200 and an `errors` array, + # so the status code alone proves nothing. + if [ -n "$out" ] && [ "$(printf '%s' "$out" | jq -r '.errors == null' 2>/dev/null || echo false)" = "true" ]; then + printf '%s' "$out" + return 0 + fi + done + return 1 + } + started=$(date +%s) deadline=$(( started + WAIT_BUDGET )) + echo "Looking for the Railway deployment of ${SHA} (up to $(( WAIT_BUDGET / 60 ))m)..." - # /__webjs/version (#239) reports {version, build, node, uptime}. - # `uptime` resets when Railway restarts the process for a deploy, so - # "uptime < seconds elapsed since this run began" proves the restart - # happened AFTER this push, rather than matching some earlier deploy - # that was already live when the run started. - # - # The loop is deadline-driven rather than a fixed iteration count so - # the budget stays 15 minutes whatever the per-request latency is; a - # count multiplied by (curl timeout + sleep) can silently drift past - # the job's own timeout and lose the diagnostic message below. - echo "Waiting up to $(( WAIT_BUDGET / 60 ))m for a process restart on $ORIGIN newer than this run..." + status="" while [ "$(date +%s)" -lt "$deadline" ]; do - elapsed=$(( $(date +%s) - started )) - # `|| true` on every extraction: under `set -e` a failing jq (a - # non-JSON body from a mid-deploy error page) would abort the step - # at the assignment instead of retrying, turning a transient blip - # into a hard failure. - body=$(curl -fsS --max-time 10 "$ORIGIN/__webjs/version" 2>/dev/null || true) - uptime=$(printf '%s' "$body" | jq -r '.uptime // empty' 2>/dev/null || true) - build=$(printf '%s' "$body" | jq -r '.build // empty' 2>/dev/null || true) - - if [ -n "$uptime" ]; then - if [ "$(jq -n --argjson u "$uptime" --argjson e "$elapsed" '$u < $e' 2>/dev/null || echo false)" = "true" ]; then - echo "Origin restarted ${uptime}s ago, within the ${elapsed}s since this run began." - echo "Live build: $build" - exit 0 - fi - echo " origin uptime ${uptime}s >= elapsed ${elapsed}s, deploy not live yet" + resp=$(railway_query || true) + if [ -z "$resp" ]; then + echo " Railway API not answering cleanly yet" else - echo " origin not answering with a version yet (a mid-deploy restart looks like this)" + # A brand new push takes a few seconds to appear, so an empty + # match keeps polling rather than concluding anything. + status=$(printf '%s' "$resp" \ + | jq -r --arg sha "$SHA" '[.data.deployments.edges[]?.node | select((.meta.commitHash // "") == $sha) | .status] | first // ""' \ + 2>/dev/null || echo "") + case "$status" in + SUCCESS) + echo "Deployment SUCCESS: new bytes are live, purge is warranted." + echo "should_purge=true" >> "$GITHUB_OUTPUT" + exit 0 ;; + SKIPPED) + echo "::notice::Railway SKIPPED this commit, so the origin is unchanged." + echo "Nothing is stale at the edge, so there is nothing to purge." + echo "should_purge=false" >> "$GITHUB_OUTPUT" + exit 0 ;; + FAILED|CRASHED|REMOVED) + echo "::warning::Deployment status ${status}: no new content reached the origin." + echo "Skipping the purge, which would only cost a cold cache for no benefit." + echo "should_purge=false" >> "$GITHUB_OUTPUT" + exit 0 ;; + "") + echo " no deployment recorded for this sha yet" ;; + *) + echo " deployment ${status}, still in progress" ;; + esac fi sleep 15 done - echo "::error::No new deploy observed on $ORIGIN within $(( WAIT_BUDGET / 60 ))m." - echo "Refusing to purge: evicting the cache while the origin still serves the" - echo "previous build would repopulate the edge with stale bytes, which is the" - echo "exact failure this workflow exists to prevent." - echo "If the deploy landed later, re-run this workflow from the Actions tab to purge." - exit 1 + # Deliberately not a failure. The whole point of #1192 is that this + # job must not cry wolf; an unresolved deployment is reported and + # left for a manual run rather than red-Xing a routine commit. + echo "::warning::No terminal deployment status for ${SHA} within $(( WAIT_BUDGET / 60 ))m." + echo "Not purging. If the deploy landed later, run this workflow from the" + echo "Actions tab to purge manually." + echo "should_purge=false" >> "$GITHUB_OUTPUT" - name: Purge the Cloudflare cache + if: github.event_name == 'workflow_dispatch' || steps.deploy.outputs.should_purge == 'true' env: CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} run: | set -euo pipefail if [ -z "${CLOUDFLARE_API_TOKEN:-}" ]; then - echo "::error::CLOUDFLARE_API_TOKEN is not set. Add it as a repository secret," + echo "::error::CLOUDFLARE_API_TOKEN is not set. Add it as a REPOSITORY secret," echo "scoped to Zone / Cache Purge / Purge on the webjs.dev zone." exit 1 fi @@ -141,6 +198,7 @@ jobs: echo "Cache purged for the webjs.dev zone." - name: Confirm the edge now matches the origin + if: github.event_name == 'workflow_dispatch' || steps.deploy.outputs.should_purge == 'true' run: | set -euo pipefail # The purge is asynchronous across pops, so give it a moment before diff --git a/framework-dev.md b/framework-dev.md index c5fb62316..2e62894a4 100644 --- a/framework-dev.md +++ b/framework-dev.md @@ -21,11 +21,25 @@ Net: edit the `HEALTHCHECK` for the Docker contract, keep `railway.json` for the `webjs.dev` sits behind Cloudflare, and the site's static assets (`/public/tailwind.css`, the brand SVGs) are served with `cache-control: public, max-age=14400` at STABLE urls. Without an eviction the edge therefore keeps serving the PREVIOUS copy for up to four hours after a deploy. That shipped two visible regressions in one day (a pre-redesign stylesheet after #1179, then the un-fixed logo marks after #1185), and staleness is per-asset rather than all-or-nothing, so the site can look half-updated. -`.github/workflows/purge-cdn.yml` handles this on every push to `main`. It does NOT purge immediately: Railway auto-deploys from the same push and the build takes minutes, so an immediate purge would evict the cache while the origin still served the old bytes and the next visitor would repopulate the edge with exactly those bytes. Instead the job polls `GET /__webjs/version` (#239) on the Railway ORIGIN (bypassing the cache it is about to purge) until the reported `uptime` is lower than the time elapsed since the run began, which proves the process restarted for THIS push, then issues a zone-wide `purge_everything`. On timeout it fails loudly rather than purging, because a mistimed purge is the precise failure the workflow exists to prevent. +`.github/workflows/purge-cdn.yml` handles this on every push to `main`. It does NOT purge immediately: a Railway build takes minutes, so an immediate purge would evict the cache while the origin still served the old bytes and the next visitor would repopulate the edge with exactly those. Instead the job asks Railway's GraphQL API for the deployment whose `meta.commitHash` matches the pushed sha and reads its real status, then issues a zone-wide `purge_everything` only on `SUCCESS`. + +It is a SEPARATE workflow rather than a step in `release.yml` on purpose: `release.yml` fires only on pushes touching `changelog/**` (an npm package release), while the site redeploys on ordinary merges. None of the four website merges on 2026-07-30 touched `changelog/`, so a purge living there would have fired for none of them. + +Reading the deployment status rather than inferring it is the second design (#1192). The first version watched the origin's `/__webjs/version` `uptime` for a restart, assuming every push to `main` produces a deploy. It does not: Railway marked #1189's own merge commit `SKIPPED` (it touched only `.github` and a `.md`), no restart ever happened, and the job failed after a 15 minute wait. Every docs-only commit would have been a red X, and a job that cries wolf on routine commits stops being read. So the status now decides: + +| Deployment status | Action | +|---|---| +| `SUCCESS` | purge | +| `SKIPPED` | no purge, notice. Railway did not deploy, so the origin is unchanged and nothing is stale | +| `FAILED` / `CRASHED` / `REMOVED` | no purge, warning. No new content reached the origin | +| in progress, or no record yet | keep polling | +| nothing terminal within 15 minutes | no purge, warning, run still GREEN. Use the manual run below | The purge is zone-wide rather than a path list: all four hostnames (`webjs.dev`, `example-blog.webjs.dev`, `docs.webjs.dev`, `ui.webjs.dev`) are proxied inside the one `webjs.dev` zone, so a single call covers them, and a zone purge cannot silently miss an asset the way a hand-maintained list does. Purging evicts only; nothing is deleted. -- **Required secret:** `CLOUDFLARE_API_TOKEN`, scoped to **Zone / Cache Purge / Purge** on the `webjs.dev` zone only. Do not reuse a broad account-wide token. +- **Required secrets**, both REPOSITORY secrets. The job declares no `environment:`, so an environment-scoped secret arrives EMPTY and the step fails on the explicit "not set" branch: + - `CLOUDFLARE_API_TOKEN`, scoped to **Zone / Cache Purge / Purge** on the `webjs.dev` zone only. Do not reuse a broad account-wide token. + - `RAILWAY_TOKEN`, a Railway **project** token for this project's `production` environment (narrower than an account token). A project token authenticates with the `Project-Access-Token` header and an account token with `Authorization: Bearer`, so the workflow tries both and keeps whichever answers without a GraphQL error. - **Manual purge:** run the "Purge CDN" workflow from the Actions tab (`workflow_dispatch`), which skips the deploy wait and purges straight away. Use this if a deploy landed after the wait timed out. - **Checking staleness by hand:** compare the edge against the origin rather than trusting `cf-cache-status`, since a `HIT` on fresh content is fine and only differing bytes are a problem. From 0d571e8e9bf4359be7d5ee1f55b3d099db381676 Mon Sep 17 00:00:00 2001 From: Vivek Date: Thu, 30 Jul 2026 20:09:31 +0530 Subject: [PATCH 2/3] ci: fail loudly on a bad Railway token, parse either meta shape Review of the new gating step found two defects of the same class: both turned a misconfiguration into a GREEN run that silently never purges, which is precisely the failure the workflow exists to catch. 1. `meta` may arrive as an object or as a JSON-encoded string, and this was not verifiable against the live API beforehand. Indexing a string with `.commitHash` is a jq ERROR rather than a null, so the unguarded filter would have swallowed every response, polled to the deadline and reported green without ever purging. Normalize both shapes, and leave a null `meta` resolving to empty so it keeps polling instead of crashing. 2. A wrong, expired, or mis-scoped RAILWAY_TOKEN produced an auth error on every poll, which read as "no deployment yet" and ended in the same green no-op. Track whether ANY clean response ever arrived: none means the credential or endpoint is wrong, so exit 1 and print the server's own error body (never the token). "Answered fine but the sha never reached a terminal status" stays a warning and stays green, which is the case #1192 was about. Also dump the deployments actually seen on the timeout path, so a field-name or response-shape mismatch is diagnosable in the log instead of looking identical to "the deploy never happened". Verified: object, string, and null `meta` all resolve without a jq crash, and the bad-token path exits 1 carrying the API's message. --- .github/workflows/purge-cdn.yml | 47 ++++++++++++++++++++++++++++++--- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/.github/workflows/purge-cdn.yml b/.github/workflows/purge-cdn.yml index f1d3b6a5a..03ca76232 100644 --- a/.github/workflows/purge-cdn.yml +++ b/.github/workflows/purge-cdn.yml @@ -103,6 +103,7 @@ jobs: railway_query() { local out + : > /tmp/railway-last-error for hdr in "Project-Access-Token: ${RAILWAY_TOKEN}" "Authorization: Bearer ${RAILWAY_TOKEN}"; do out=$(curl -sS --max-time 20 -X POST "$RAILWAY_API" \ -H "$hdr" -H "Content-Type: application/json" --data "$payload" 2>/dev/null || true) @@ -112,6 +113,9 @@ jobs: printf '%s' "$out" return 0 fi + # Keep the server's own words for the diagnostic below. This is + # an error body, never the token. + printf '%s\n' "${out:0:400}" >> /tmp/railway-last-error done return 1 } @@ -121,16 +125,27 @@ jobs: echo "Looking for the Railway deployment of ${SHA} (up to $(( WAIT_BUDGET / 60 ))m)..." status="" + got_response=false while [ "$(date +%s)" -lt "$deadline" ]; do resp=$(railway_query || true) if [ -z "$resp" ]; then echo " Railway API not answering cleanly yet" else + got_response=true # A brand new push takes a few seconds to appear, so an empty # match keeps polling rather than concluding anything. - status=$(printf '%s' "$resp" \ - | jq -r --arg sha "$SHA" '[.data.deployments.edges[]?.node | select((.meta.commitHash // "") == $sha) | .status] | first // ""' \ - 2>/dev/null || echo "") + # `meta` may arrive as an object OR as a JSON-encoded string, + # and this could not be checked against the live API ahead of + # time. Indexing a string with `.commitHash` is a jq ERROR, not + # a null, so the unguarded form would swallow every response, + # poll to the deadline and silently never purge. Normalize both + # shapes before reading the sha. + status=$(printf '%s' "$resp" | jq -r --arg sha "$SHA" ' + [ .data.deployments.edges[]?.node + | . as $n + | ((.meta // {}) | if type == "string" then (fromjson? // {}) else . end) as $m + | select(($m.commitHash // "") == $sha) + | $n.status ] | first // ""' 2>/dev/null || echo "") case "$status" in SUCCESS) echo "Deployment SUCCESS: new bytes are live, purge is warranted." @@ -155,6 +170,32 @@ jobs: sleep 15 done + # Never a single clean response means the credential or the endpoint + # is wrong, not that the deploy is slow. That MUST be loud: a job + # that quietly stops purging is exactly how the edge goes stale + # without anyone noticing, which is the bug this workflow exists to + # prevent. + if [ "$got_response" != "true" ]; then + echo "::error::No valid response from the Railway API in $(( WAIT_BUDGET / 60 ))m." + echo "RAILWAY_TOKEN is likely missing the project, expired, or of the wrong type." + echo "Last response body from the API:" + sed 's/^/ /' /tmp/railway-last-error 2>/dev/null | head -10 || true + exit 1 + fi + + # Dump what the API actually returned. Without this, a field-name or + # response-shape mismatch is indistinguishable from "the deploy never + # happened", and the job would quietly stop purging forever while + # still reporting green. + if [ -n "${resp:-}" ]; then + echo "Most recent deployments the API reported for this service:" + printf '%s' "$resp" | jq -r ' + .data.deployments.edges[]?.node + | ((.meta // {}) | if type == "string" then (fromjson? // {}) else . end) as $m + | " \(.status) \(($m.commitHash // "")[0:8]) \(.createdAt)"' \ + 2>/dev/null | head -5 || true + fi + # Deliberately not a failure. The whole point of #1192 is that this # job must not cry wolf; an unresolved deployment is reported and # left for a manual run rather than red-Xing a routine commit. From 3c369bf9fc2120fff4a2987c2d96ff1b32d6e685 Mon Sep 17 00:00:00 2001 From: Vivek Date: Thu, 30 Jul 2026 22:28:39 +0530 Subject: [PATCH 3/3] ci: make the purge diagnostic name the missing field in full The timeout-path dump sliced its own fallback string to eight characters, so a response with no commitHash printed "")[0:8]) \(.createdAt)"' \ + | (($m.commitHash // "") | if . == "" then "" else .[0:8] end) as $short + | " \(.status) \($short) \(.createdAt)"' \ 2>/dev/null | head -5 || true fi