SERVER-131624: Add --batch-test-discovery and cache suiteconfig results#116
Draft
lavigneer wants to merge 2 commits into
Draft
SERVER-131624: Add --batch-test-discovery and cache suiteconfig results#116lavigneer wants to merge 2 commits into
lavigneer wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR reduces resmoke.py subprocess startup overhead during config generation by (1) optionally batching resmoke test discovery across many suites and pre-seeding the in-process discovery cache, and (2) memoizing suiteconfig results per suite to avoid repeated resmoke startups across generated tasks.
Changes:
- Add
--batch-test-discoveryCLI flag and plumbing to optionally prewarm the discovery cache before generation. - Implement batched
test-discovery(--suiterepeated) inResmokeProxy::prewarmand seed the per-suite discovery cache from the multi-document YAML stream. - Add a per-suite memoization cache for
get_suite_configto avoid redundantresmoke.py suiteconfiginvocations.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/resmoke/resmoke_proxy.rs | Adds prewarm batching, suiteconfig memoization, and shared command construction for discovery. |
| src/main.rs | Introduces --batch-test-discovery flag and passes it into the execution configuration. |
| src/lib.rs | Wires the flag through dependencies and triggers prewarming prior to task generation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lavigneer
force-pushed
the
lavigneer/batch-test-discovery
branch
from
July 15, 2026 18:37
ae8c8fa to
5639808
Compare
lavigneer
force-pushed
the
lavigneer/memoize-test-discovery
branch
from
July 16, 2026 18:47
c3a9e21 to
f522577
Compare
lavigneer
force-pushed
the
lavigneer/batch-test-discovery
branch
from
July 20, 2026 15:13
5639808 to
6ea5db4
Compare
lavigneer
changed the title
…discovery With this flag, all resmoke-generated suites are discovered up front via batched 'resmoke.py test-discovery --suite A --suite B ...' invocations (chunks of 100) that seed the per-suite discovery cache, instead of paying resmoke's ~2s python startup once per suite. Requires a resmoke version whose test-discovery accepts repeated --suite arguments; older branches simply don't pass the flag and keep per-suite discovery. Fuzzer tasks, multiversion-generate tasks, and burn_in tasks are excluded from the prewarm list: their 'suite' is not a plain resmoke suite and a single SuiteNotFound fails the whole batch. Multiversion per-old-version suites are still discovered lazily. A failed batch falls back to lazy per-suite discovery rather than failing generation. On mongodb-mongo-master this replaces ~323 sequential ~2s resmoke calls with 4 batched invocations totalling ~30s. Cache resmoke suiteconfig results per suite write_standard_suite creates a fresh ResmokeConfigCache per generated task, so 'resmoke.py suiteconfig' ran once per task (~800 python startups, ~160s of a ~200s generation run). Cache the parsed config in ResmokeProxy keyed by suite name, same per-entry locking scheme as the discovery cache, and log suiteconfig invocation durations. Address review: propagate subprocess errors, verify batch output count Replace run_command unwraps with error propagation in run_test_discovery and get_suite_config, log the requested suite name rather than a misleading suite_config field, discard a batch whose document count does not match the request instead of zipping by order blindly, and use poison-safe lock recovery for the caches. Fix clippy type_complexity and nonminimal_bool in batch discovery code Add a SuiteConfigCache type alias for the suite_config_cache field and simplify the boolean in prewarm_test_discovery, both flagged by clippy on the Rust 1.95 toolchain.
lavigneer
force-pushed
the
lavigneer/batch-test-discovery
branch
from
July 20, 2026 15:32
6ea5db4 to
947b45a
Compare
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on #115. Two changes:
--batch-test-discoveryflag (src/main.rs,src/lib.rs,src/resmoke/resmoke_proxy.rs): before generation, collect the suite of every resmoke-generated task (bazel targets included; fuzzer, multiversion-generate, and burn_in tasks excluded since their "suite" is not a plain resmoke suite), dedupe, and discover them via batchedresmoke.py test-discovery --suite A --suite B ...calls in chunks of 100, seeding the discovery cache from the multi-document YAML output. A failed batch logs and falls back to lazy per-suite discovery — generation cannot be broken by a bad suite. Multiversion per-old-version suites remain lazily discovered.src/resmoke/resmoke_proxy.rs):get_suite_configresults are now memoized per suite with the same per-entry-lock scheme.write_standard_suitebuilds a freshResmokeConfigCacheper generated task, soresmoke.py suiteconfigpreviously ran once per task (~800 python startups, 666s CPU measured).Why
Resmoke subprocess startup (~2s) dominated version_gen generation time. Measured on mongodb-mongo-master patches: 323 suites batch-discovered in ~30s (vs ~11 CPU-minutes), and generation dropped 201s → 142s with the suiteconfig cache. Combined with #115, the
generate version shstep went 7m26s → 4m05s on patch builds (~357s → ~142s of generator time at ~2,160 runs/week ≈ 129 machine-hours/week).Compatibility
The flag requires resmoke
test-discoveryto accept repeated--suiteargs (10gen/mongo#58681). Branches that don't pass the flag keep exactly the old behavior; release branches pin older generator versions anyway.Testing
Unit tests for prewarm seeding and suiteconfig caching; full
cargo testpasses; verified end-to-end on mongodb-mongo-master patch builds (all 4 batches seeded, generated config identical task set).