e2e: Port the chat test harness improvements: retry visibility, video on retries only#1746
Merged
andremion merged 1 commit intoJul 13, 2026
Conversation
… on retries only - RetryRule: each failed attempt that is retried is written as its own Allure result sharing the real test's historyId, with that attempt's steps, error and artifacts, so TestOps groups attempts as retries and can flag flaky tests - RetryRule: screen recording runs only on retry attempts; previously every test recorded an 8 Mbit/s video that was discarded on pass - RetryRule: the recording file uses methodName instead of displayName (the display name's parentheses break when the recording commands go through the device shell), and a recording stop failure no longer replaces the test result - RetryRule: move to io.getstream.video.android.rules (the file declared a chat package), remove the unused Retry annotation, simplify DatabaseOperations - Wait: waitForText and waitForCount poll at 50ms instead of spinning the CPU until timeout; remove the unused waitForTextToChange - Allurefile: name launches 'Cron checks' only for real scheduled events, so manual dispatches stay distinguishable in TestOps - Fastfile: batch_tests splits round-robin, always yielding exactly batch_count groups; the previous rounded-up slicing could produce fewer groups and crash on nil for the last batch Ported from GetStream/stream-chat-android#6568.
Contributor
PR checklist ✅All required conditions are satisfied:
🎉 Great job! This PR is ready for review. |
Contributor
Author
|
@CodeRabbit review |
Contributor
✅ Action performedReview finished.
|
Contributor
SDK Size Comparison 📏
|
Contributor
WalkthroughRetry infrastructure was moved and rewritten, UI waits gained polling delays, Allure cron detection was restricted to scheduled events, and Fastlane batching now validates round-robin groups. ChangesRetry infrastructure
UI automation polling
Fastlane test orchestration
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant TestStatement
participant RetryRule
participant ScreenRecorder
participant AllureLifecycle
TestStatement->>RetryRule: evaluate test attempt
RetryRule->>ScreenRecorder: start recording for retry attempts
TestStatement-->>RetryRule: throw failure
RetryRule->>ScreenRecorder: stop recording safely
RetryRule->>AllureLifecycle: schedule finished failed-attempt result
RetryRule->>TestStatement: retry or rethrow last failure
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@fastlane/Fastfile`:
- Around line 101-106: Validate batch_count is greater than zero and batch is
within the valid range before the current_batch selection uses modulo. Raise a
UI.user_error! with the existing batching context for invalid values, then
preserve the round-robin selection and empty-batch guard for valid inputs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: e4a0175c-c9da-4b48-8001-ce6e57b35434
📒 Files selected for processing (5)
demo-app/src/androidTestE2etestingDebug/kotlin/io/getstream/video/android/rules/RetryRule.ktdemo-app/src/androidTestE2etestingDebug/kotlin/io/getstream/video/android/tests/StreamTestCase.ktdemo-app/src/androidTestE2etestingDebug/kotlin/io/getstream/video/android/uiautomator/Wait.ktfastlane/Allurefilefastlane/Fastfile
|
testableapple
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Port the E2E harness improvements from the chat repo (GetStream/stream-chat-android#6568): make retry attempts visible in Allure TestOps, record video only on retries, and fix the CI batch split.
Resolves AND-1308
Implementation
RetryRule: each failed attempt that is retried becomes its own Allure result (same historyId), so TestOps shows retries and can flag flaky tests. Before, a pass-on-retry looked identical to a stable pass.RetryRule: screen recording runs only on retry attempts. Before, every test recorded an 8 Mbit/s video and discarded it on pass.RetryRule: the recording file usesmethodNameinstead ofdisplayName(the display name's parentheses break when the recording commands go through the device shell on this uiautomator version), and a recording stop failure logs instead of replacing the test result. Both were caught by a live probe run during the port.RetryRulemoved toio.getstream.video.android.rules: the file declared a chat package inside this repo.Wait.kt:waitForTextandwaitForCountpoll at 50ms instead of spinning a CPU core until timeout; unusedwaitForTextToChangeremoved.Allurefile: launches are named "Cron checks" only for real scheduled runs, so manual dispatches of the nightly stay out of the flakiness analytics (TestOps project 69).Fastfile:batch_testssplits round-robin (the old rounded-up slicing could produce fewer groups than batches and crash onnilfor the last shard).Metrics
E2E CI duration (this repo), sum of the 3 test batch jobs:
That overlaps with the best baseline, so on duration this is within single-run variance. This suite is smaller and more network-bound than chat's, so the recording overhead it removes was a smaller share of its runtime. The nightly trend will tell whether a duration gain shows on top.
Retry visibility and recordings (the substance of this PR):
For reference, chat measured a large duration gain from the same change (PR gate from about 40 to about 25 minutes wall clock; see GetStream/stream-chat-android#6568), but its suite is bigger and CPU-bound, so those numbers do not transfer here.
Testing
Verified live on an emulator with a temporary probe test (fails attempt 1, passes attempt 2), pulling
/sdcard/googletest/test_outputfiles/allure-resultsafterwards:historyId; the failed attempt carries its own steps, error,logcat_1/screenshot_1/hierarchy_1, and no video; the passed result has the lateststart, so TestOps keeps it as the current one.connectedE2etestingDebugAndroidTest.No UI changes.
Summary by CodeRabbit