Skip to content

ci: gate the CDN purge on the real Railway deployment status #1193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vivek7405 merged 3 commits into from
Jul 30, 2026
+172 −58
Merged

ci: gate the CDN purge on the real Railway deployment status #1193

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aa58f2a
ci: gate the CDN purge on the real Railway deployment status
vivek7405 Jul 30, 2026
0d571e8
ci: fail loudly on a bad Railway token, parse either meta shape
vivek7405 Jul 30, 2026
3c369bf
ci: make the purge diagnostic name the missing field in full
vivek7405 Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
212 changes: 156 additions & 56 deletions .github/workflows/purge-cdn.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
name: Purge CDN

# Evicts the Cloudflare edge cache after a site deploy reaches the origin.
# Evicts the Cloudflare edge cache after a site deploy actually reaches the
# origin.
#
# Why this exists: the site's static assets (/public/tailwind.css, the brand
# SVGs) are served with `cache-control: public, max-age=14400` at STABLE urls,
Expand All @@ -9,19 +10,30 @@ name: Purge CDN
# pre-redesign stylesheet after #1179, then the un-fixed logo marks after
# #1185), each needing a manual dashboard purge somebody had to remember.
#
# Why it waits instead of purging on push: Railway auto-deploys from a push to
# main and the build takes minutes. Purging immediately would evict the cache
# while the origin still serves the OLD bytes, the next visitor would
# repopulate the cache with exactly those old bytes, and the site would be
# stale for another four hours with nothing to show for the run. So the job
# confirms the new build is live on the ORIGIN first, and fails loudly rather
# than purging if it never shows up.
# Why it waits instead of purging on push: Railway builds take minutes, and
# purging before the new bytes are live would evict the cache while the origin
# still serves the OLD ones, so the next visitor repopulates the edge with
# exactly those, leaving the site stale for another four hours.
#
# Required secret: CLOUDFLARE_API_TOKEN, scoped to Zone / Cache Purge / Purge
# on the webjs.dev zone only (not an account-wide token). See framework-dev.md.
# Why it asks Railway rather than watching the origin: the first version of
# this workflow inferred the deploy from process uptime, assuming every push to
# main produces one. It does not. Railway marked #1189's own merge commit
# SKIPPED (it touched only .github and a .md), no restart ever happened, and
# the job failed after a 15 minute wait. Inferring the deploy makes every
# docs-only commit a red X, and a job that cries wolf on routine commits is
# worse than no job. So this asks Railway for the deployment matching the
# pushed sha and reads its real status (#1192).
#
# Required secrets:
# RAILWAY_TOKEN a Railway PROJECT token for this project's production
# environment (narrower than an account token).
# CLOUDFLARE_API_TOKEN scoped to Zone / Cache Purge / Purge on the webjs.dev
# zone only.
# Both are REPOSITORY secrets: this job declares no `environment:`, so an
# environment-scoped secret would arrive empty. See framework-dev.md.
#
# Manual purge: run this workflow from the Actions tab (workflow_dispatch),
# which skips the deploy wait and purges immediately.
# which skips the deploy wait entirely and purges immediately.

on:
push:
Expand All @@ -43,77 +55,164 @@ jobs:
purge:
name: Wait for the deploy, then purge
runs-on: ubuntu-latest
# Comfortably above the wait step's own 15-minute budget, so the step's
# explanatory failure always fires before the job-level kill (which would
# leave no diagnosis in the log).
# Comfortably above the wait step's own budget, so the step's explanatory
# exit always happens before the job-level kill (which would leave no
# diagnosis in the log).
timeout-minutes: 25
env:
# A zone id identifies a domain, it is not a credential, so it lives in
# the workflow rather than costing a second required secret.
# Ids identify resources and are not credentials, so they live in the
# workflow rather than costing extra secrets.
ZONE_ID: 46692b879a06f3a6a987b99915560393
# The Railway service domain for @webjsdev/website. Polled directly so
# the readiness check bypasses the very cache this job is about to
# purge (asking webjs.dev could be answered from the edge).
RAILWAY_API: https://backboard.railway.com/graphql/v2
RAILWAY_PROJECT_ID: 1d05ab75-b64e-4920-94ef-1ca7d04778fc
RAILWAY_ENVIRONMENT_ID: 2e30b2f6-d166-48d3-9ae7-17176fb0942f
RAILWAY_SERVICE_ID: 2eb044b7-8698-43d3-959b-37c8925eb653
# The Railway service domain for @webjsdev/website, used only by the
# post-purge smoke check so it can compare the edge against the origin.
ORIGIN: https://webjs.up.railway.app
# How long to wait for the deploy before giving up, in seconds.
# How long to wait for the deployment to reach a terminal status.
WAIT_BUDGET: 900

steps:
- name: Wait for the new build to be live on the origin
- name: Resolve the Railway deployment for this commit
id: deploy
if: github.event_name == 'push'
env:
RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }}
SHA: ${{ github.sha }}
run: |
set -euo pipefail
if [ -z "${RAILWAY_TOKEN:-}" ]; then
echo "::error::RAILWAY_TOKEN is not set. Add it as a REPOSITORY secret"
echo "(a Railway project token for the production environment)."
exit 1
fi

# Railway's API sits behind Cloudflare, which answers `error code:
# 1010` to clients whose user agent looks automated (python urllib is
# blocked outright), so this must go through curl.
#
# A project token authenticates with `Project-Access-Token`, an
# account token with `Authorization: Bearer`. Which one applies
# depends on the token the owner created, so try both and keep
# whichever answers without a GraphQL error.
query='query deployments($input: DeploymentListInput!, $first: Int) { deployments(input: $input, first: $first) { edges { node { id status createdAt meta } } } }'
payload=$(jq -n --arg q "$query" \
--arg p "$RAILWAY_PROJECT_ID" --arg e "$RAILWAY_ENVIRONMENT_ID" --arg s "$RAILWAY_SERVICE_ID" \
'{query: $q, variables: {first: 30, input: {projectId: $p, environmentId: $e, serviceId: $s}}}')

railway_query() {
local out
: > /tmp/railway-last-error
for hdr in "Project-Access-Token: ${RAILWAY_TOKEN}" "Authorization: Bearer ${RAILWAY_TOKEN}"; do
out=$(curl -sS --max-time 20 -X POST "$RAILWAY_API" \
-H "$hdr" -H "Content-Type: application/json" --data "$payload" 2>/dev/null || true)
# A GraphQL error comes back with HTTP 200 and an `errors` array,
# so the status code alone proves nothing.
if [ -n "$out" ] && [ "$(printf '%s' "$out" | jq -r '.errors == null' 2>/dev/null || echo false)" = "true" ]; then
printf '%s' "$out"
return 0
fi
# Keep the server's own words for the diagnostic below. This is
# an error body, never the token.
printf '%s\n' "${out:0:400}" >> /tmp/railway-last-error
done
return 1
}

started=$(date +%s)
deadline=$(( started + WAIT_BUDGET ))
echo "Looking for the Railway deployment of ${SHA} (up to $(( WAIT_BUDGET / 60 ))m)..."

# /__webjs/version (#239) reports {version, build, node, uptime}.
# `uptime` resets when Railway restarts the process for a deploy, so
# "uptime < seconds elapsed since this run began" proves the restart
# happened AFTER this push, rather than matching some earlier deploy
# that was already live when the run started.
#
# The loop is deadline-driven rather than a fixed iteration count so
# the budget stays 15 minutes whatever the per-request latency is; a
# count multiplied by (curl timeout + sleep) can silently drift past
# the job's own timeout and lose the diagnostic message below.
echo "Waiting up to $(( WAIT_BUDGET / 60 ))m for a process restart on $ORIGIN newer than this run..."
status=""
got_response=false
while [ "$(date +%s)" -lt "$deadline" ]; do
elapsed=$(( $(date +%s) - started ))
# `|| true` on every extraction: under `set -e` a failing jq (a
# non-JSON body from a mid-deploy error page) would abort the step
# at the assignment instead of retrying, turning a transient blip
# into a hard failure.
body=$(curl -fsS --max-time 10 "$ORIGIN/__webjs/version" 2>/dev/null || true)
uptime=$(printf '%s' "$body" | jq -r '.uptime // empty' 2>/dev/null || true)
build=$(printf '%s' "$body" | jq -r '.build // empty' 2>/dev/null || true)

if [ -n "$uptime" ]; then
if [ "$(jq -n --argjson u "$uptime" --argjson e "$elapsed" '$u < $e' 2>/dev/null || echo false)" = "true" ]; then
echo "Origin restarted ${uptime}s ago, within the ${elapsed}s since this run began."
echo "Live build: $build"
exit 0
fi
echo " origin uptime ${uptime}s >= elapsed ${elapsed}s, deploy not live yet"
resp=$(railway_query || true)
if [ -z "$resp" ]; then
echo " Railway API not answering cleanly yet"
else
echo " origin not answering with a version yet (a mid-deploy restart looks like this)"
got_response=true
# A brand new push takes a few seconds to appear, so an empty
# match keeps polling rather than concluding anything.
# `meta` may arrive as an object OR as a JSON-encoded string,
# and this could not be checked against the live API ahead of
# time. Indexing a string with `.commitHash` is a jq ERROR, not
# a null, so the unguarded form would swallow every response,
# poll to the deadline and silently never purge. Normalize both
# shapes before reading the sha.
status=$(printf '%s' "$resp" | jq -r --arg sha "$SHA" '
[ .data.deployments.edges[]?.node
| . as $n
| ((.meta // {}) | if type == "string" then (fromjson? // {}) else . end) as $m
| select(($m.commitHash // "") == $sha)
| $n.status ] | first // ""' 2>/dev/null || echo "")
case "$status" in
SUCCESS)
echo "Deployment SUCCESS: new bytes are live, purge is warranted."
echo "should_purge=true" >> "$GITHUB_OUTPUT"
exit 0 ;;
SKIPPED)
echo "::notice::Railway SKIPPED this commit, so the origin is unchanged."
echo "Nothing is stale at the edge, so there is nothing to purge."
echo "should_purge=false" >> "$GITHUB_OUTPUT"
exit 0 ;;
FAILED|CRASHED|REMOVED)
echo "::warning::Deployment status ${status}: no new content reached the origin."
echo "Skipping the purge, which would only cost a cold cache for no benefit."
echo "should_purge=false" >> "$GITHUB_OUTPUT"
exit 0 ;;
"")
echo " no deployment recorded for this sha yet" ;;
*)
echo " deployment ${status}, still in progress" ;;
esac
fi
sleep 15
done

echo "::error::No new deploy observed on $ORIGIN within $(( WAIT_BUDGET / 60 ))m."
echo "Refusing to purge: evicting the cache while the origin still serves the"
echo "previous build would repopulate the edge with stale bytes, which is the"
echo "exact failure this workflow exists to prevent."
echo "If the deploy landed later, re-run this workflow from the Actions tab to purge."
exit 1
# Never a single clean response means the credential or the endpoint
# is wrong, not that the deploy is slow. That MUST be loud: a job
# that quietly stops purging is exactly how the edge goes stale
# without anyone noticing, which is the bug this workflow exists to
# prevent.
if [ "$got_response" != "true" ]; then
echo "::error::No valid response from the Railway API in $(( WAIT_BUDGET / 60 ))m."
echo "RAILWAY_TOKEN is likely missing the project, expired, or of the wrong type."
echo "Last response body from the API:"
sed 's/^/ /' /tmp/railway-last-error 2>/dev/null | head -10 || true
exit 1
fi

# Dump what the API actually returned. Without this, a field-name or
# response-shape mismatch is indistinguishable from "the deploy never
# happened", and the job would quietly stop purging forever while
# still reporting green.
if [ -n "${resp:-}" ]; then
echo "Most recent deployments the API reported for this service:"
printf '%s' "$resp" | jq -r '
.data.deployments.edges[]?.node
| ((.meta // {}) | if type == "string" then (fromjson? // {}) else . end) as $m
| (($m.commitHash // "") | if . == "" then "<no commitHash>" else .[0:8] end) as $short
| " \(.status) \($short) \(.createdAt)"' \
2>/dev/null | head -5 || true
fi

# Deliberately not a failure. The whole point of #1192 is that this
# job must not cry wolf; an unresolved deployment is reported and
# left for a manual run rather than red-Xing a routine commit.
echo "::warning::No terminal deployment status for ${SHA} within $(( WAIT_BUDGET / 60 ))m."
echo "Not purging. If the deploy landed later, run this workflow from the"
echo "Actions tab to purge manually."
echo "should_purge=false" >> "$GITHUB_OUTPUT"

- name: Purge the Cloudflare cache
if: github.event_name == 'workflow_dispatch' || steps.deploy.outputs.should_purge == 'true'
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
set -euo pipefail
if [ -z "${CLOUDFLARE_API_TOKEN:-}" ]; then
echo "::error::CLOUDFLARE_API_TOKEN is not set. Add it as a repository secret,"
echo "::error::CLOUDFLARE_API_TOKEN is not set. Add it as a REPOSITORY secret,"
echo "scoped to Zone / Cache Purge / Purge on the webjs.dev zone."
exit 1
fi
Expand Down Expand Up @@ -141,6 +240,7 @@ jobs:
echo "Cache purged for the webjs.dev zone."

- name: Confirm the edge now matches the origin
if: github.event_name == 'workflow_dispatch' || steps.deploy.outputs.should_purge == 'true'
run: |
set -euo pipefail
# The purge is asynchronous across pops, so give it a moment before
Expand Down
18 changes: 16 additions & 2 deletions framework-dev.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,25 @@ Net: edit the `HEALTHCHECK` for the Docker contract, keep `railway.json` for the

`webjs.dev` sits behind Cloudflare, and the site's static assets (`/public/tailwind.css`, the brand SVGs) are served with `cache-control: public, max-age=14400` at STABLE urls. Without an eviction the edge therefore keeps serving the PREVIOUS copy for up to four hours after a deploy. That shipped two visible regressions in one day (a pre-redesign stylesheet after #1179, then the un-fixed logo marks after #1185), and staleness is per-asset rather than all-or-nothing, so the site can look half-updated.

`.github/workflows/purge-cdn.yml` handles this on every push to `main`. It does NOT purge immediately: Railway auto-deploys from the same push and the build takes minutes, so an immediate purge would evict the cache while the origin still served the old bytes and the next visitor would repopulate the edge with exactly those bytes. Instead the job polls `GET /__webjs/version` (#239) on the Railway ORIGIN (bypassing the cache it is about to purge) until the reported `uptime` is lower than the time elapsed since the run began, which proves the process restarted for THIS push, then issues a zone-wide `purge_everything`. On timeout it fails loudly rather than purging, because a mistimed purge is the precise failure the workflow exists to prevent.
`.github/workflows/purge-cdn.yml` handles this on every push to `main`. It does NOT purge immediately: a Railway build takes minutes, so an immediate purge would evict the cache while the origin still served the old bytes and the next visitor would repopulate the edge with exactly those. Instead the job asks Railway's GraphQL API for the deployment whose `meta.commitHash` matches the pushed sha and reads its real status, then issues a zone-wide `purge_everything` only on `SUCCESS`.

It is a SEPARATE workflow rather than a step in `release.yml` on purpose: `release.yml` fires only on pushes touching `changelog/**` (an npm package release), while the site redeploys on ordinary merges. None of the four website merges on 2026-07-30 touched `changelog/`, so a purge living there would have fired for none of them.

Reading the deployment status rather than inferring it is the second design (#1192). The first version watched the origin's `/__webjs/version` `uptime` for a restart, assuming every push to `main` produces a deploy. It does not: Railway marked #1189's own merge commit `SKIPPED` (it touched only `.github` and a `.md`), no restart ever happened, and the job failed after a 15 minute wait. Every docs-only commit would have been a red X, and a job that cries wolf on routine commits stops being read. So the status now decides:

| Deployment status | Action |
|---|---|
| `SUCCESS` | purge |
| `SKIPPED` | no purge, notice. Railway did not deploy, so the origin is unchanged and nothing is stale |
| `FAILED` / `CRASHED` / `REMOVED` | no purge, warning. No new content reached the origin |
| in progress, or no record yet | keep polling |
| nothing terminal within 15 minutes | no purge, warning, run still GREEN. Use the manual run below |

The purge is zone-wide rather than a path list: all four hostnames (`webjs.dev`, `example-blog.webjs.dev`, `docs.webjs.dev`, `ui.webjs.dev`) are proxied inside the one `webjs.dev` zone, so a single call covers them, and a zone purge cannot silently miss an asset the way a hand-maintained list does. Purging evicts only; nothing is deleted.

- **Required secret:** `CLOUDFLARE_API_TOKEN`, scoped to **Zone / Cache Purge / Purge** on the `webjs.dev` zone only. Do not reuse a broad account-wide token.
- **Required secrets**, both REPOSITORY secrets. The job declares no `environment:`, so an environment-scoped secret arrives EMPTY and the step fails on the explicit "not set" branch:
- `CLOUDFLARE_API_TOKEN`, scoped to **Zone / Cache Purge / Purge** on the `webjs.dev` zone only. Do not reuse a broad account-wide token.
- `RAILWAY_TOKEN`, a Railway **project** token for this project's `production` environment (narrower than an account token). A project token authenticates with the `Project-Access-Token` header and an account token with `Authorization: Bearer`, so the workflow tries both and keeps whichever answers without a GraphQL error.
- **Manual purge:** run the "Purge CDN" workflow from the Actions tab (`workflow_dispatch`), which skips the deploy wait and purges straight away. Use this if a deploy landed after the wait timed out.
- **Checking staleness by hand:** compare the edge against the origin rather than trusting `cf-cache-status`, since a `HIT` on fresh content is fine and only differing bytes are a problem.

Expand Down
Loading