Remove the firestore flake probe workflow once #776 is resolved

[tyler-reitz]

.github/workflows/flake-probe.yaml was added in #780 as a temporary measurement tool for #776. It is workflow_dispatch only, so it costs nothing until someone runs it, but it is not meant to live in the repo indefinitely. This issue is its removal ticket, filed at merge time so the intent does not survive only in a PR body.

Remove it when

#776 is resolved, not merely measured. Measuring the flake rate is not fixing it: between the measurement and the fix sits the actual @grpc/grpc-js override change, and the probe is the thing that verifies that change landed. So it outlives the first run by design.

Concretely, it can go once all of these hold:

• The probe has run in CI and produced a rate for both arms.
• A decision has been taken on the @grpc/grpc-js override, either way.
• If the override lands, the probe has been re-run to confirm the rate moved.
• Removal PR deletes .github/workflows/flake-probe.yaml. Nothing else references it.

Two reasons this is unlikely to be same-day

1. The default run is underpowered. #776 already found 30 runs per arm inadequate at p = 0.12, and 30 is the default here. At that rate you would expect roughly 4 flakes in 30 baseline runs against 0 in the override arm, which does not reach significance. Reaching it needs closer to 50 per arm. At an estimated 20-25s per iteration (one emulator boot plus one test file; for calibration, the full five-emulator suite runs in ~57s in CI), 50 per arm is roughly 47 minutes of wall clock against a 60 minute job timeout. Plan on more than one run.

2. The probe may not be able to see the CI bug at all. It runs one emulator and one test file; CI runs five emulators and the whole suite in parallel. RESOURCE_EXHAUSTED: Received message larger than max has never appeared in a firestore-only run, and that signal is the entire reason to suspect the local repro and the CI failure are different bugs. If the first run comes back with a clean RESOURCE_EXHAUSTED column, that is not evidence the CI bug is absent. It is evidence this configuration cannot reach it, and the response is to widen the probe toward the full suite, which is a second change to this workflow rather than a deletion.

The job summary states this caveat on every run so a clean table is not read as a verdict.

Suggested first run

Start at the default 30 per arm (~30 minutes, comfortably inside the timeout) and read the RESOURCE_EXHAUSTED and overlap columns before spending 50 per arm on statistical power. The overlap column, added in review, is what says whether the gRPC desync and the #776 assertion actually co-occur.

Refs #776. Added in #780.

[tyler-reitz]

First run is done, and it changes what removal depends on

Run 31204989659, defaults (30 iterations per arm), both Node versions. 120 of 120 runs passed.

Node	Arm	grpc-js	Pass	Flake	Hang	Infra	RESOURCE_EXHAUSTED
22	baseline	1.9.16	30	0	0	0	0
22	override	1.14.4	30	0	0	0	0
24	baseline	1.9.16	30	0	0	0	0
24	override	1.14.4	30	0	0	0	0

No unmatched failures, so the classifier is not hiding anything behind a reworded vitest message. Each job took about 10 minutes, roughly 8s per iteration, so the 20-25s estimate in the issue body above was pessimistic and Armando's 7s local figure was the accurate one.

This is the "cannot reach the CI bug" outcome the issue anticipated, and it should not be read as good news.

1. The 12% local rate does not hold in CI for this configuration. Zero flakes in 60 baseline runs. If p were 0.12 that outcome has probability 0.88^60, about 0.05%. By the rule of three, the firestore-only CI flake rate is under 5% at 95% confidence.
2. Zero RESOURCE_EXHAUSTED across all 120 runs, which confirms the caveat directly: that signal has never appeared in a firestore-only run and still has not. It is the entire reason to suspect the local repro and the CI failure are different bugs, and this probe cannot see it.
3. The override arm is uninformative. It cannot improve on a baseline that never failed, so there is still no evidence either way on the @grpc/grpc-js bump. The comparison the probe was built to make has not actually been made yet.

Next step: widen the probe to the full suite

The probe runs vitest run firestore against one emulator. CI runs npm run test: five emulators, the whole suite, parallel workers. Given the above, the firestore-only configuration is measuring something that mostly does not fail, so the next change is to add a mode that runs the real CI workload.

⚠️ This collides with timeout-minutes and the resolution matters. A full-suite iteration is around 57s rather than 8s (measured from Test Node.js 22 on recent PRs). At 30 per arm with both arms on one runner that is roughly 57 minutes against the current 60 minute timeout, and 50 per arm would be about 95 minutes.

Raise timeout-minutes to 120-180. Do not solve this by putting the arms back on separate runners. Splitting them was the original defect #780 was reviewed for: the premise of the probe is that the machine matters, so separate runners reintroduce the variable being measured. The longer job is the cost of a valid comparison. The always() summary step means a run that does hit the wall still reports the arms that finished.

Removal conditions, revised

The checklist in the issue body still stands, with one item added ahead of it:

• Add a full-suite mode and re-run, since the firestore-only configuration has now been shown not to reproduce either failure.
• Decide on the @grpc/grpc-js override on the strength of that data.
• If the override lands, re-run to confirm the rate moved.
• Delete .github/workflows/flake-probe.yaml.

A useful negative result for #776 in the meantime: whatever is failing in CI is not reproducible by running the firestore suite alone on CI hardware, at either grpc-js version. That narrows the search toward emulator contention, parallel workers, or total resource pressure, and away from the Firestore client library in isolation.

[tyler-reitz]

Correction: the 57s per-iteration figure above is wrong, and the timeout collision mostly is not real

In the comment above I said a full-suite iteration costs about 57s, and that 30 per arm would be roughly 57 minutes against a 60 minute timeout, so the timeout had to rise.

57s was the duration of the whole Test Node.js 22 job, including checkout, npm ci, Java setup, functions install and artifact download. Reading the step timings, the Run tests step itself is 23s.

Mode	Per iteration	30/arm (60 runs)	50/arm (100 runs)	200/arm (400 runs)
firestore-only	~8s (measured)	~10 min	~16 min	~55 min
full-suite	~23s (measured)	~28 min	~43 min	~153 min

So full-suite at the default 30 per arm fits inside the existing 60 minute timeout with room to spare, and 50 per arm fits too. Only the 200 cap can hit the wall. The raise to 180 minutes in the PR below is headroom for that case, not a requirement for ordinary runs.

The rest of that comment stands, including the reason the arms must not go back on separate runners.

The next step is now open as a PR

#785 adds the full-suite workload mode, defaulting to it, and keeps firestore-only as the cheaper isolated option.

Two things it turned up that were not visible from the issue:

• The probe was missing the functions deps install. test.yaml runs npm install in ./functions before npm run test, and the functions emulator does not come up without it. Every full-suite iteration would have failed.
• A failing double check - emulator is running was being counted as a hang. Those health checks fail by timing out, so they matched the hang pattern; auth, firestore and database each open with one. In full-suite mode this would have fired regularly and inflated the bucket meant for CI: test/firestore.test.tsx flakes intermittently on both Node 22 and Node 24 #776's 120s hang. Found by running the real suite locally and feeding the actual log through the classifier, not by inspection.