Skip to content

fix(test, frontend): raise Vitest timeouts for CI stalls - #7623

Open
aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:fix/frontend-vitest-timeouts
Open

fix(test, frontend): raise Vitest timeouts for CI stalls#7623
aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:fix/frontend-vitest-timeouts

Conversation

@aglinxinyuan

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The macOS leg of build / frontend goes red on a different unit test every few days — always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days:

Run Test Error
31665399757 UserDatasetVersionCreatorComponent > onClickCreate creates a dataset with a sanitized name … Test timed out in 5000ms
31630884042 AdminUserComponent > sortByAffiliation compares affiliations … Hook timed out in 10000ms
31411656559 WorkflowRuntimeStatisticsComponent > should create Test timed out in 5000ms

Root cause: the runner stalls, not the test. The stall lands on whichever test happens to be executing. From run 31665399757 — one commit, one matrix, two OSes:

Measure ubuntu-latest macos-latest
the spec file that failed (10 tests) 240 ms 11 727 ms
suite wall clock 89.85 s 252.88 s
cumulative test time 182.34 s 307.69 s
runner size 4 cores / 16 GB 3 cores / 7 GB

That file is not systematically slow — it took 219 ms on macOS in an earlier run, and 443 ms locally. The 11.7 s is a stall. jsdom + v8-coverage workers on 3 cores / 7 GB run under real memory pressure, which is where multi-second pauses come from.

Change File
testTimeout 5s → 30s, hookTimeout 10s → 30s frontend/vitest.config.ts, frontend/vitest.browser.config.ts
fail-fast: false on the frontend matrix .github/workflows/build.yml
Timeouts row in the runner-setup table frontend/TESTING.md

Bumping the one test's timeout would be whack-a-mole — the next stall picks a different spec. 30 s absorbs a stall an order of magnitude worse than any observed; a spec that legitimately needs 30 s is broken, and the job's own timeout still bounds a true hang.

The fail-fast opt-out matches every other multi-leg matrix in build.yml (platform, platform-integration, agent-service, infra). Today one flaky OS cancels the other two legs, so the run no longer says whether the failure reproduces off that OS — exactly the evidence needed to tell a runner flake from a real break.

Before:  macOS stalls 5s -> that test fails -> ubuntu + windows cancelled
After:   macOS stalls 5s -> absorbed; a real break still fails all legs

If macOS keeps flaking after this, the next lever is capping maxWorkers on that leg to cut memory pressure. Left out here: it trades wall clock for stability and can't be measured from a non-macOS box.

Any related issues, documentation, discussions?

Related to #6073 — the same failure mode on the Windows leg, which already proposes both of these remedies. Left open and assigned; the three new macOS occurrences are recorded there.

How was this PR tested?

No production code is touched; the change is to the test harness and CI config.

Check Result
yarn test:ci (full jsdom suite, new config) 200 files, 4433 passed / 1 skipped — same counts as CI's ubuntu leg
config actually wired a throwaway spec with a 6 s beforeEach + 8 s body passes (14 026 ms); the 8 s body fails on the old 5 s default. Removed before commit
build.yml parses js-yaml load → jobs.frontend.strategy['fail-fast'] === false
formatting prettier-eslint --list-different clean

The flake itself can't be reproduced on demand — that's the nature of a runner stall. What this PR asserts is verifiable: the ceiling that the stalls blow past is now 6× higher, and the surviving matrix legs still report their own results.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

The macOS leg of `build / frontend` goes red on a different unit test
every few days -- always a timeout, never the same spec, always green
on rerun. Three occurrences in the last four days:

| Run | Test | Error |
| --- | --- | --- |
| 31665399757 | UserDatasetVersionCreatorComponent > onClickCreate ... | Test timed out in 5000ms |
| 31630884042 | AdminUserComponent > sortByAffiliation ... | Hook timed out in 10000ms |
| 31411656559 | WorkflowRuntimeStatisticsComponent > should create | Test timed out in 5000ms |

The tests are not the problem: the runner stalls, and the stall lands
on whichever test is executing. In run 31665399757 the offending spec
file took 11727ms on macos-latest and 240ms on ubuntu-latest for the
same commit; in an earlier run the same file took 219ms on macOS.
Suite totals from that run show the same picture -- 252.88s wall on
macOS vs 89.85s on ubuntu, with a cumulative test time of 307.69s vs
182.34s. macos-latest gives 3 cores and 7 GB against ubuntu's 4 and
16, so the jsdom + v8-coverage workers run under real memory pressure
there.

Raise testTimeout and hookTimeout to 30s in both Vitest configs, which
absorbs a stall an order of magnitude worse than any observed so far.
A spec that legitimately needs 30s is broken, and the job's own
timeout still bounds a true hang. Per-test timeouts would be
whack-a-mole: the next stall picks a different test.

Also opt the frontend matrix out of fail-fast, as every other
multi-leg matrix in build.yml already does. Today one flaky OS
cancels the other two legs, which destroys exactly the evidence
needed to tell a runner flake from a real break.

Before:  macOS stalls 5s -> that test fails -> ubuntu + windows cancelled
After:   macOS stalls 5s -> absorbed; a real break still fails all legs
Copilot AI lite review requested due to automatic review settings August 13, 2026 05:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Aug 13, 2026
@github-actions github-actions Bot added fix frontend Changes related to the frontend GUI ci changes related to CI docs Changes related to documentations and removed release/v1.2 back porting to release/v1.2 labels Aug 13, 2026
@github-actions
github-actions Bot requested a review from xuang7 August 13, 2026 05:03
@github-actions

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
release/v1.2 Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7.

Auto-label run.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @parshimers
    You can request their reviews formally with /request-review @parshimers.

  • Contributors with relevant context: @renovate-bot, @Yicong-Huang
    You can notify them by mentioning @renovate-bot, @Yicong-Huang in a comment.

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.00%. Comparing base (dd7d813) to head (b3772da).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7623      +/-   ##
============================================
+ Coverage     88.96%   89.00%   +0.03%     
- Complexity     4338     4350      +12     
============================================
  Files          1178     1178              
  Lines         46835    46836       +1     
  Branches       5226     5226              
============================================
+ Hits          41667    41685      +18     
+ Misses         3422     3413       -9     
+ Partials       1746     1738       -8     
Flag Coverage Δ
access-control-service 70.00% <ø> (ø)
agent-service 98.62% <ø> (ø)
amber 84.98% <ø> (+0.09%) ⬆️
computing-unit-managing-service 60.38% <ø> (ø)
config-service 77.31% <ø> (ø)
file-service 68.90% <ø> (ø)
frontend 90.57% <ø> (-0.01%) ⬇️
notebook-migration-service 78.89% <ø> (ø)
pyamber 97.56% <ø> (+0.04%) ⬆️
workflow-compiling-service 57.89% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 4 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 310ab88 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 434 0.265 21,648/32,127/32,127 us 🔴 -5.0% / 🔴 +97.0%
🔴 bs=100 sw=10 sl=64 939 0.573 104,692/159,560/159,560 us 🔴 +17.3% / 🔴 +47.5%
bs=1000 sw=10 sl=64 1,084 0.662 925,470/1,003,573/1,003,573 us ⚪ within ±5% / 🟢 -7.1%
Baseline details

Latest main 310ab88 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 434 tuples/sec 457 tuples/sec 775.33 tuples/sec -5.0% -44.0%
bs=10 sw=10 sl=64 MB/s 0.265 MB/s 0.279 MB/s 0.473 MB/s -5.0% -44.0%
bs=10 sw=10 sl=64 p50 21,648 us 22,627 us 12,743 us -4.3% +69.9%
bs=10 sw=10 sl=64 p95 32,127 us 31,747 us 16,310 us +1.2% +97.0%
bs=10 sw=10 sl=64 p99 32,127 us 31,747 us 18,926 us +1.2% +69.7%
bs=100 sw=10 sl=64 throughput 939 tuples/sec 973 tuples/sec 1,001 tuples/sec -3.5% -6.2%
bs=100 sw=10 sl=64 MB/s 0.573 MB/s 0.594 MB/s 0.611 MB/s -3.5% -6.2%
bs=100 sw=10 sl=64 p50 104,692 us 101,825 us 101,399 us +2.8% +3.2%
bs=100 sw=10 sl=64 p95 159,560 us 136,023 us 108,206 us +17.3% +47.5%
bs=100 sw=10 sl=64 p99 159,560 us 136,023 us 118,195 us +17.3% +35.0%
bs=1000 sw=10 sl=64 throughput 1,084 tuples/sec 1,110 tuples/sec 1,026 tuples/sec -2.3% +5.6%
bs=1000 sw=10 sl=64 MB/s 0.662 MB/s 0.677 MB/s 0.626 MB/s -2.2% +5.7%
bs=1000 sw=10 sl=64 p50 925,470 us 901,558 us 996,304 us +2.7% -7.1%
bs=1000 sw=10 sl=64 p95 1,003,573 us 995,220 us 1,042,531 us +0.8% -3.7%
bs=1000 sw=10 sl=64 p99 1,003,573 us 995,220 us 1,074,934 us +0.8% -6.6%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,461.15,200,128000,434,0.265,21647.74,32127.13,32127.13
1,100,10,64,20,2131.01,2000,1280000,939,0.573,104691.64,159559.51,159559.51
2,1000,10,64,20,18447.26,20000,12800000,1084,0.662,925470.16,1003572.54,1003572.54

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci changes related to CI docs Changes related to documentations fix frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants