Reduce smoke test flakiness#2720
Merged
Merged
Conversation
…peAgent into dev/robgruen/flakey
# Conflicts: # ts/packages/dispatcher/dispatcher/README.AUTOGEN.md
…actions
The playerSchema.agr renamed two actions in a prior commit but the variant
fixture grammars and benchmark JSONL were left with the old names:
- `resume` → `resumePlayback`
- `playTrack` (flat params) → `playMusic` (target: { kind, ... } params)
This caused `generateAllDeltas()` to produce 466 spurious deltas (vs the 25
committed rows) because every resume and play-by-artist corpus entry created
a delta between HEAD and each variant.
Changes:
- v1-v9 variant .agr files: Resume rules updated to `resumePlayback`;
PlaySpecificTrack rules updated to `playMusic` with target-nested params
- player.regression-benchmark.jsonl: 3 v1-lost-transport rows updated
(actionA.actionName `resume` → `resumePlayback`)
- player.utterances.jsonl: corpus expectedAction fields updated to new names
(resume → resumePlayback, playTrack → playMusic with new parameter structure)
…peAgent into dev/robgruen/flakey
…tructionCacheResolver test
…peAgent into dev/robgruen/flakey
- Revert test back to \"play despacito\" (no artist required) - Add \"isOptional\": true to the artist sub-phrase in basic.json - Add \"optional\": true to M:artist name_7 in constructions.json so track-only utterances (no artist) match the play construction
…le action interpretation
…nd doc Opt-in reliability accounting in translateTestCommon, gated on TRANSLATION_RELIABILITY_DIR (no behavior change when unset): per suite it tallies attempts, assertion failures, per-request outcome variations, and LLM translation tokens, writing one JSON tally per suite in afterAll. reliability/report.ts (compiled to dist/test/reliability/report.js) aggregates the tallies into reliability/README.md - the mean translation tokens between flaky failures across all suites - appends each run to reliability/history.json, and renders the trend as inline mermaid xychart charts. The generated README.md is prettier-ignored to avoid table-alignment churn.
Mirror UnknownAction.reason on ReasoningAction so the model must state why a request needs multi-step reasoning instead of a single known action. Optional, so it never blocks translation, and it surfaces in traces to diagnose deferrals. Observed effect: in a repeat=20 full translate run this eliminated the mcpfilesystem move-file reasoning deferral (14 -> 0 escapes) and took the suite from 15 failures to 0, with no test tolerances added.
Append the recent runs to the trend, including the repeat=20 stress runs before (15 failures) and after (0 failures across 6.79M tokens) the clarify + reasoning schema fixes.
robgruen
marked this pull request as ready for review
July 22, 2026 04:31
robgruen
enabled auto-merge
July 22, 2026 04:31
TalZaccai
approved these changes
Jul 22, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a suite-wide reliability reporting system for translation tests, adds new test tolerance logic for duplicate actions, and improves error handling and test data for translation reliability. The most important changes are summarized below.
Translation reliability reporting
reliability/report.ts, which aggregates per-suite translation test results and generates a comprehensive markdown report (README.md) with metrics, trend charts, and a history log (history.json). This enables tracking the mean number of translation tokens between flaky failures (MTTF) across all translation test suites. [1] [2] [3].prettierignoreto exclude the auto-generated reliability report from formatting.Test tolerance improvements
duplicateOfPrevioussentinel forextraActionsin translation tests, allowing tests to tolerate cases where the model repeats its final action without accepting arbitrary extra actions. Updated type definitions and helper functions to support this. [1] [2] [3]Error handling improvements
fetchWithPoolinrestClient.tsto guard against undefined responses, preventing hard failures and better handling intermittent network issues seen in translation reliability tests.Test data and schema updates
actionsfield toHistoryContextinrequestAction.tsto provide structured action history for translation prompts.playerManifest.jsonschema description for clarity and completeness.