Skip to content

ci: wire the DEV-2203 suites — post-deploy smoke, weekly canary, monthly matrix - #189

Open
danielzytohoc wants to merge 5 commits into
feat/DEV-2203-ai-live-specfrom
chore/DEV-2203-wire-workflows
Open

ci: wire the DEV-2203 suites — post-deploy smoke, weekly canary, monthly matrix#189
danielzytohoc wants to merge 5 commits into
feat/DEV-2203-ai-live-specfrom
chore/DEV-2203-wire-workflows

Conversation

@danielzytohoc

@danielzytohoc danielzytohoc commented Aug 14, 2026

Copy link
Copy Markdown

Last in the DEV-2203 stack. Stacked on #187 — merge the stack bottom-up (#181#182#183#184/#185#186#187 → this).

The coverage layers this creates

Layer Trigger Containers
PR CI (unchanged) every PR 0 — deterministic only, now incl. style-panel/version-pin/download/share-UI
Post-deploy @smoke both deploy workflows, via workflow_call 1 — one render per engine, share fixture, docs example, one Style round-trip
Weekly canary Mon 05:00 UTC, prod, AI on ~8, strictly serialized
Monthly full matrix 1st, 05:00 UTC ~50, --workers=2, unchanged discipline

e2e-live.yml

  • New dispatch inputs: ai (LLM answer checks), pkg_pr_new_ref (one real container at a pkg.pr.new build — DEV-2198 validation days).
  • Folds in the three orphaned live specs nothing ever ran: row-striping, preview-recovery's live half, import-live.
  • Steps split by what they hold: sandpack/API-read specs run parallel; container suites strictly --workers=1; the authed share step pre-checks E2E_BROKER_TOKEN against the broker and turns rot into a ::warning, not a red run.
  • Concurrency: mode-keyed as before, smoke calls keyed by run id, deployed/canary runs never cancellable (cancelled runs strand container sessions — cleanup lives in finally).

Notes

  • runner/AGENTS.md CI/CD table updated.
  • All four YAML files parse-checked. Before enabling the cron in anger: one manual dispatch against prod with everything on is the real proof — I'd do it right after the stack merges.
  • The smoke jobs add ~8–10 min after each deploy; the API deploy's smoke boots exactly the container image that deploy just rebuilt.

Note

Medium Risk
Changes deploy concurrency and adds scheduled prod E2E that consume real container pool capacity and LLM budget on the weekly canary; mis-timed crons or cancellation policy regressions could strand Sandboxes or starve live traffic, but no application runtime code changes.

Overview
Adds post-deploy @smoke E2E to both API and authoring deploy workflows via workflow_call to e2e-live.yml against prod — one container render per engine, share fixture, docs example, and one Style round-trip after health/bundle curls.

Authoring deploy turns off cancel-in-progress so mid-smoke cancellation cannot strand live-preview sessions in the global Sandbox pool.

e2e-live.yml expands into the full live layer: weekly Monday 05:00 UTC prod canary (with AI), workflow_call smoke mode, dispatch inputs ai and pkg_pr_new_ref, split steps (parallel Sandpack vs --workers=1 container suites), previously orphaned specs (row-striping, import-live, etc.), broker-token-gated authed share with trace scrubbing, and concurrency keyed by mode/smoke run id.

e2e-starter-matrix.yml adds monthly cron (1st at 03:00 UTC) with prod fallback when schedule has no inputs, staggered from the weekly canary to avoid competing for five global container slots.

runner/AGENTS.md CI/CD table documents the new triggers and deploy smoke behavior.

Reviewed by Cursor Bugbot for commit b62ed73. Bugbot is set up for automated code reviews on this repo. Configure here.

…hly matrix

e2e-live.yml grows three ways in: the manual dispatch it always had (now
with ai and pkg_pr_new_ref inputs), a weekly Monday prod canary for
external drift (the hosted bundler, npm, the broker, docs-bucket rot —
deploy breakage is caught at deploy time, so daily would only buy
triage), and a workflow_call smoke mode both deploy workflows now invoke
after their curl checks: the @smoke subset, one container, minutes.

The full deployed run folds in the specs nothing ever ran (row-striping,
preview-recovery's live half, import-live) plus the DEV-2203 additions,
split by what they hold: the sandpack/API-read step runs parallel, the
container step strictly --workers=1 against the global pool of five, the
authed share round-trip treats a rotten E2E_BROKER_TOKEN as a notice
rather than a red run, and the AI step runs on the canary or by asking.

e2e-starter-matrix.yml gains the monthly sweep (1st, 05:00 UTC) — the
only coverage of the full starters × majors surface, at a cadence
matched to npm-release timescales. Concurrency keeps deployed runs
uncancellable so no run ever strands container sessions.
Comment thread .github/workflows/e2e-live.yml
Comment thread .github/workflows/deploy-runner-authoring.yml
Dan Zyto added 2 commits August 17, 2026 08:37
The post-deploy smoke boots a live-preview container; an unconditional
cancel-in-progress killed the whole run — smoke included — on the next
push to master, stranding that session in the global Sandbox pool
(cleanup lives in a finally a cancellation never reaches). Deploys now
queue, matching deploy-runner-api.yml.

The other #189 finding — e2e-live listing style-panel.spec.ts before it
existed on this branch — is resolved by linearising the stack: the spec
now sits in this branch's ancestry (185 → 186 → 187 → here).

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Agentic security review of the DEV-2203 workflow wiring: one high-severity finding on broker JWT exposure via Playwright artifacts.

Open in Web View Automation 

Sent by Cursor Security Agent: Security Reviewer

Comment thread .github/workflows/e2e-live.yml

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9e958c1. Configure here.

Comment thread .github/workflows/e2e-live.yml
Dan Zyto added 2 commits August 17, 2026 08:45
…ugbot #189)

The authed-share step puts a live session JWT in sessionStorage and an
Authorization header; a Playwright trace records both, GitHub does not
redact secrets inside artifact zips, and this repo is public — so a
failed canary would have published a valid @handsontable.com session
for seven days. That step now runs with --trace off --retries=0 and
scrubs test-results/ and playwright-report/ before the on-failure
upload can see them (|| status capture: Actions runs bash -e, so a
plain $? after a failing command is unreachable).

The monthly starter matrix moves to 03:00 on the 1st: at 05:00 a
Monday-the-1st would start it alongside the weekly canary, and matrix
--workers=2 plus the canary's serialized container suite against the
same global pool of five is exactly the contention both workflow
headers warn about.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant