[BigQueryIO] Parallelize schema update integration tests - #39622
Open
bvolpato wants to merge 1 commit into
Open
Conversation
bvolpato
marked this pull request as ready for review
August 5, 2026 22:53
Contributor
|
Assigning reviewers: R: @ahmedabu98 for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
ahmedabu98
approved these changes
Aug 5, 2026
ahmedabu98
left a comment
Contributor
There was a problem hiding this comment.
LGTM, nice optimization
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
StorageApiSinkSchemaUpdateITcurrently runs all 32 parameterized cases serially inside one test class. Gradle already gives GCP integration tests four forks, but it can only distribute work between classes.This splits the matrix by
useInputSchemainto two concrete test classes backed by the same test implementation. Each class owns a separate BigQuery dataset, so parallel setup and cleanup cannot interfere. Early-rollout coverage includes both classes explicitly.No test cases, assertions, row counts, stream counts, schema triggers, polling intervals, or soak delays change.
Live result
Java_GCP_IO_Directpassed on this PR. ItsintegrationTesttask completed in 1h24m52s.Four nearby successful runs without this split took 1h43m07s to 1h47m26s, with a 1h44m37s median: 31012265163, 31014377562, 31028738723, and 31042299715. This PR reduced that median by 19m45s, or 19%.
The full Gradle command completed in 1h46m53s versus a 2h10m41s baseline median, an 18% reduction. Full-command timing also includes compilation and cache variance, so
integrationTestis the cleaner comparison.Historical timing model
Scheduled master run 30486556613 recorded all 32 schema-update cases passing in 81m49s. Grouping those measured cases by the new class boundary gives:
This predicts a 29m39s, or 36%, reduction for this critical shard when both classes receive existing Gradle workers. It does not increase
maxParallelForksor add another workflow job.Two classes are intentional. The longest individual parameter took 49m11s, only 2m59s below the two-class estimate. Splitting all four parameters would therefore add more concurrent Storage Write API tests for little additional critical-path gain.
Testing
./gradlew :sdks:java:io:google-cloud-platform:compileTestJava -PdisableSpotlessCheck=true./gradlew :sdks:java:io:google-cloud-platform:spotlessJavaCheckintegrationTest --test-dry-run: 32 cases discovered across two classes, no base-class testsbigQueryEarlyRolloutIntegrationTest --test-dry-run: same 32 cases discoveredJava_GCP_IO_Direct: passed against live GCP servicesCHANGES.mdentry; test scheduling only.