-
Notifications
You must be signed in to change notification settings - Fork 1
Stabilize workflow failure event wait #1050
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
dennisonbertram
merged 1 commit into
main
from
codex/issue-1049-workflow-failure-timeout
Jul 30, 2026
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
66 changes: 66 additions & 0 deletions
66
docs/plans/2026-07-30-issue-1049-workflow-failure-timeout-impact-map.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # Cross-Surface Impact Map: Issue #1049 Workflow Failure-Event Timeout | ||
|
|
||
| ## Task | ||
|
|
||
| - Task / issue: stabilize the live workflow failure-event test, #1049. | ||
| - Plan: `2026-07-30-issue-1049-workflow-failure-timeout-plan.md`. | ||
| - Owner: Codex. | ||
| - Status: implemented and fully verified locally; hosted checks pending. | ||
|
|
||
| ## Current Ownership, Callers, and Data Flow | ||
|
|
||
| - Entry: `TestEngineDefinitionSubscribeAndFailureEvents`. | ||
| - Production flow: Engine persists failed workflow state and publishes | ||
| `workflow.failed` to its subscribers. | ||
| - Test flow: subscribe, wait for stored terminal state, then drain the live | ||
| stream until the failure event. | ||
| - Search conclusion: the two-second timer is test-local; production API, | ||
| workflow, and provider deadlines are separate. | ||
|
|
||
| ## Config, API, CLI, and Tools | ||
|
|
||
| - Config/env/defaults: none. | ||
| - API/CLI/wire/tool behavior: unchanged. | ||
| - Event contract: continue requiring exact `workflow.failed` delivery. | ||
|
|
||
| ## Persistence and Compatibility | ||
|
|
||
| - Stores/schemas/migrations/caches: unchanged. | ||
| - Compatibility: no runtime change. | ||
|
|
||
| ## Lifecycle, Security, and Reliability | ||
|
|
||
| - Concurrency: allow race-instrumented shared CI enough scheduling time. | ||
| - Cleanup: stop the timer and cancel the subscription. | ||
| - Auth/privacy/secrets: none. | ||
| - Failure diagnostics: retain the complete observed event list. | ||
|
|
||
| ## Product and Integration Surfaces | ||
|
|
||
| - Harness/workflow runtime, API, TUI, web, macOS GUI, providers: no code change. | ||
| - Automation: full race and hosted gates become host-speed tolerant. | ||
| - UX/accessibility: none. | ||
|
|
||
| ## Deployment and Operations | ||
|
|
||
| - Deployment/migration/flags: none. | ||
| - Rollback: revert if a genuinely absent failure event no longer fails. | ||
| - Observability/operator docs: none. | ||
|
|
||
| ## Regression Tests | ||
|
|
||
| - Red: full race gate timed out after two seconds with the first two events. | ||
| - Green: focused normal/race `-count=100`. | ||
| - Controls: stored status and error remain failed; live stream must still | ||
| contain `workflow.failed`. | ||
| - Full: workflows normal/race and `./scripts/test-regression.sh`. | ||
|
|
||
| ## Documentation and Handoff | ||
|
|
||
| - Update engineering log, long-term log, plan, impact map, and plans index. | ||
| - No public docs. | ||
|
|
||
| ## Warning Check | ||
|
|
||
| - All runtime/product surfaces are explicitly unaffected because the repair is | ||
| confined to one test-owned timer. |
59 changes: 59 additions & 0 deletions
59
docs/plans/2026-07-30-issue-1049-workflow-failure-timeout-plan.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| # Plan: Make the Workflow Failure-Event Test Contention-Tolerant | ||
|
|
||
| ## Context | ||
|
|
||
| - Governing GitHub issue: #1049. | ||
| - Problem: `TestEngineDefinitionSubscribeAndFailureEvents` allows only two | ||
| wall-clock seconds for live `workflow.failed` delivery after the stored run | ||
| reaches failed. | ||
| - Red evidence: the full repository race gate timed out with | ||
| `workflow.started` and `workflow.step.started` already observed. | ||
| - Constraint: preserve the live-event assertion and production ordering. | ||
|
|
||
| ## Scope | ||
|
|
||
| - In scope: a leak-safe, ten-second test deadline for this event assertion. | ||
| - Out of scope: workflow runtime ordering, event fanout, storage, production | ||
| timeouts, and unrelated timing tests. | ||
|
|
||
| ## Documentation Contract | ||
|
|
||
| - Feature status: test-only bug repair. | ||
| - Public docs affected: none. | ||
| - Evidence: engineering and long-term logs, plan, impact map, plans index. | ||
|
|
||
| ## Test Plan | ||
|
|
||
| - Red: retain the exact full-race timeout as regression evidence. | ||
| - Green: focused normal and race tests at `-count=100`. | ||
| - Package: `internal/workflows` normal and race. | ||
| - Full: `./scripts/test-regression.sh` and GitHub required checks. | ||
|
|
||
| ## Cross-Surface Impact Map | ||
|
|
||
| - See `2026-07-30-issue-1049-workflow-failure-timeout-impact-map.md`. | ||
|
|
||
| ## Implementation Checklist | ||
|
|
||
| - [x] Create contract-complete bug #1049. | ||
| - [x] Capture the full-race failure and architecture search evidence. | ||
| - [x] Write the plan and impact map before code. | ||
| - [x] Replace the short one-shot deadline with a stopped ten-second timer. | ||
| - [x] Run focused stress, package, and full local gates. | ||
| - [ ] Pass hosted required checks. | ||
| - [ ] Merge through a closing PR. | ||
|
|
||
| ## Verification | ||
|
|
||
| - Focused normal and race tests passed at `-count=100`. | ||
| - The complete `internal/workflows` package passed normal and race tests. | ||
| - `./scripts/test-regression.sh` passed normal, race, and the 85.6% coverage | ||
| gate with zero uncovered functions. | ||
|
|
||
| ## Risks and Mitigations | ||
|
|
||
| - Risk: a longer wait could conceal a missing event. | ||
| - Mitigation: the test still fails with the complete event history after a | ||
| bounded deadline and still requires `workflow.failed`. | ||
| - Risk: `time.After` retains its timer until firing on the fast path. | ||
| - Mitigation: use `time.NewTimer` with `defer Stop`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When
executeemitsworkflow.failedafterSubscribetakes itsGetEventssnapshot but before it registersliveundere.mu(engine.go:137-147), the event is absent from both returned history and the channel forever. Since this test starts execution before subscribing, that race can still produce the reported failed stored state with no terminal event; changing the deadline to ten seconds only makes the same failure eight seconds slower. Make history retrieval and subscriber registration atomic relative toemit, or otherwise close that gap, rather than relying on a longer timer.AGENTS.md reference: AGENTS.md:L31-L35
Useful? React with 👍 / 👎.