Skip to content

Do not merge fix: stop the FDv2 synchronizer rotation from spinning (fixes testDebugUnitTest OOM) - #384

Draft
abelonogov-ld wants to merge 2 commits into
mainfrom
andrey/fix-fdv2-recovery-test-oom
Draft

Do not merge fix: stop the FDv2 synchronizer rotation from spinning (fixes testDebugUnitTest OOM)#384
abelonogov-ld wants to merge 2 commits into
mainfrom
andrey/fix-fdv2-recovery-test-oom

Conversation

@abelonogov-ld

@abelonogov-ld abelonogov-ld commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

FDv2DataSourceTest.recoveryResetsToFirstAvailableSynchronizer failed CI with java.lang.OutOfMemoryError, breaking testDebugUnitTest on the 5.14.0 release run. There were two problems: the tests misused the mocks, and the data source turned that misuse into a busy loop instead of a plain failure.

Tests. Three recovery tests shared a single MockQueuedSynchronizer instance across every call to their synchronizer factory. FDv2DataSource closes the previously active synchronizer when it switches sources, so when the recovery condition reset the source index and rebuilt the first synchronizer, the factory handed back an already-closed mock, which reports SHUTDOWN from next() immediately. Each factory now builds a fresh mock, which is what real synchronizer factories do.

Data source. The rotation loop advanced to the next source as soon as a session ended, with no lower bound on how long a session had to last. Two closed mocks therefore rotated as fast as the CPU allowed, writing about three log messages per iteration into the unbounded list in LogCapture until the test JVM exhausted its heap. Rotation now pauses when a session ends sooner than it plausibly could have connected, growing the pause up to 30 seconds while sessions keep ending immediately and resetting once a session lasts. The pause waits on shutdownCause rather than sleeping, so a stop() during it is acted on right away.

The same hazard existed without any mocks: a single configured synchronizer that reports SHUTDOWN as soon as it is built would have spun this loop forever in a real app.

Verification

  • New test synchronizersThatShutDownImmediatelyDoNotSpin builds two synchronizers that shut down immediately and asserts the rotation stays rate limited. Without the production fix it reports 1,243,438 synchronizers built in 1.5 seconds; with it, three.
  • On unmodified main the two recovery tests each ran for the full 10s awaitApplyCount timeout, since a closed mock can never deliver the third expected apply, and that 10s of busy-looping is what exhausted the heap under CI's slower conditions. They now finish in ~3.0s, matching the intended 1s fallback plus 2s recovery.
  • :launchdarkly-android-client-sdk:testDebugUnitTest passes locally: 729 tests, 0 failures. :launchdarkly-android-client-sdk:lint is clean.

Note for reviewers

The inner loop that consumes results from a single synchronizer is still paced entirely by that synchronizer's next() future, so a source that resolves next() instantly and forever (for example one that reports INTERRUPTED with no internal backoff) can still spin. That felt like the source's contract rather than the orchestrator's, so I left it alone; happy to add a bound there too if you'd rather the data source defend against it.

Test plan

  • CI build-and-publish / testDebugUnitTest passes

Note

Overview
FDv2 synchronizer rotation no longer advances to the next source the instant a session ends. If a synchronizer session lasts less than 500ms (e.g. immediate SHUTDOWN), FDv2DataSource waits before building the next one, with a pause that doubles up to 30s while short sessions keep happening and resets once a session runs longer. The wait uses shutdownCause with a timeout so stop() can unblock immediately instead of sleeping.

Tests were updated so synchronizer factories return a new mock on each build (reusing one instance after close() only yields SHUTDOWN and triggered the spin). A new test synchronizersThatShutDownImmediatelyDoNotSpin asserts rotation stays bounded when every synchronizer shuts down on first next().

Reviewed by Cursor Bugbot for commit 69f18b0. Bugbot is set up for automated code reviews on this repo. Configure here.

abelonogov-ld and others added 2 commits August 12, 2026 15:43
The recovery tests shared one MockQueuedSynchronizer instance across every
factory call. Because FDv2DataSource closes the previous synchronizer when it
switches, recovery rebuilt an already-closed mock, which reported SHUTDOWN
immediately and made the synchronizer loop spin between two dead sources. The
spin filled the log capture until the test JVM ran out of memory, which is how
testDebugUnitTest failed in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
The synchronizer loop advanced to the next source as soon as a session ended,
with no lower bound on how long a session had to last. A source that reports
SHUTDOWN as soon as it is built therefore let the loop rotate at CPU speed:
over a million synchronizers were built in 1.5 seconds in a test that
reproduces it.

Rotation now pauses when a session ends sooner than it plausibly could have
connected, growing the pause up to 30 seconds while sessions keep ending
immediately and resetting once one of them lasts. The pause waits on
shutdownCause instead of sleeping, so a stop() during it is acted on at once.

Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld requested a review from a team as a code owner August 12, 2026 22:58
@abelonogov-ld
abelonogov-ld marked this pull request as draft August 12, 2026 23:18
@abelonogov-ld abelonogov-ld changed the title fix: stop the FDv2 synchronizer rotation from spinning (fixes testDebugUnitTest OOM) Do not merge fix: stop the FDv2 synchronizer rotation from spinning (fixes testDebugUnitTest OOM) Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant