[Bug]: Workflow failure event test times out under race contention

[dennisonbertram]

Summary

TestEngineDefinitionSubscribeAndFailureEvents waits only two wall-clock seconds for workflow.failed after the workflow store already reports the run failed. Under the repository-wide race/coverage load, event fanout can be delayed beyond that deadline even though the event is emitted correctly. The mandatory full regression gate failed with only workflow.started and workflow.step.started observed before timeout.

Acceptance criteria

• The test continues to require the live subscription to deliver workflow.failed after the stored run reaches failed.
• The wait budget tolerates shared CI race-detector contention without weakening the event-type or error assertions.
• The subscriber is always cancelled and no timer/goroutine leaks.
• Focused normal/race stress, internal/workflows normal/race, full regression, and hosted checks pass.

In scope

• This workflow failure-event test wait and durable regression documentation.
• Engineering log, plan, impact map, plans index.

Out of scope

• Workflow engine ordering, event fanout implementation, stored status semantics, other timeout tests, or production defaults.

Current architecture and search evidence

• Engine marks the workflow run failed and then publishes workflow.failed to subscribers.
• waitForWorkflowRun polls the stored terminal status; the subsequent live-event loop uses time.After(2 seconds).
• Hosted full race verification reached failed storage state but did not schedule fanout consumption before the two-second deadline.
• Issue Race suite flakes under CPU contention: goroutine and event timing tests #958 already documents this class of shared-CI wall-clock flakes and recommends a 10-second-or-condition-driven test budget.
• Adjacent production paths and API timeouts do not use this test-local timer.

Impact analysis

• Harness/workflow runtime/API/TUI/macOS GUI: no production behavior change.
• Config/schema/persistence/providers/tools/security: none after repository search.
• Concurrency/reliability: test retains the live event assertion with a generous bounded deadline suitable for race instrumentation.
• Compatibility/deployment/observability: no deployment change; hosted gate becomes representative of correctness rather than host speed. Rollback if the test stops failing when the event is genuinely absent.
• Documentation: engineering/long-term logs, plan, impact map, plans index only.

Test-first plan

1. Red: ./scripts/test-regression.sh race phase timed out after two seconds with workflow.started and workflow.step.started only.
2. Green: replace the two-second one-shot wait with a leak-safe ten-second timer while preserving the exact workflow.failed assertion.
3. Stress: focused normal/race at count=100 and complete internal/workflows normal/race.
4. Full: ./scripts/test-regression.sh and hosted required checks.

Rollout and rollback

Test-only closing PR with exact-head automated review and required checks. Revert if missing workflow.failed no longer fails within the bounded budget.

Dependencies

Child repair related to #958 and epic #1000. Blocks #1048, #1047, #1045, #1043, #1041, #1033 and the callback/GUI merge chain.