Skip to content

[stacked on #2189] Publish the parity replay harness through the plugin SDK testing kit - #2193

Closed
SawyerHood wants to merge 1 commit into
bb/third-party-canary-echo-provider-exercises-every-thr_u89sy372gvfrom
provider-gaps-parity-public
Closed

[stacked on #2189] Publish the parity replay harness through the plugin SDK testing kit#2193
SawyerHood wants to merge 1 commit into
bb/third-party-canary-echo-provider-exercises-every-thr_u89sy372gvfrom
provider-gaps-parity-public

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #2189 (the echo-provider canary). Layer 1 of 3 of the provider-gaps stack (#2193#2194#2195). Closes gap G-A from #2189.

What was wrong

The recorded-replay oracle — the harness every first-party bridge regression-tests with — was private and first-party-keyed. replayRecording, compareParity, assembleRecordedEvents, replayRecordedCells, the replay child and rerecord lived in @bb/provider-parity and the protocol package's testing/parity.ts, none of it re-exported from @get-bb/plugin-sdk/provider-bridge/testing. Worse, resolveReplayProfile threw UnreplayableProviderError for any provider id outside FIRST_PARTY_BRIDGE_MODULES and resolveBridgeLaunch only knew the four first-party module paths inside a bb checkout. Record mode was already public, so a marketplace plugin could record a session and then do nothing with it.

What changed

packages/provider-bridge-protocol/src/testing/parity.ts is now provider-agnostic. The caller names the recording, the provider id (keys the assembler's minted ids), the bridge process, and — for a bridge with a provider child — a ReplayProviderProfile:

  • resolveProviderBridgeLaunch({ modulePath, pluginId, cwd?, dataDir?, bootstrapPath?, nodeArgs? }) builds the bridge process exactly as the runtime spawns it: the bootstrap (bridge-worker-entry), the module, the plugin scope. The bootstrap resolves from beside the harness — the source entry in a checkout, the bundled provider-bridge-worker-entry.mjs in the published SDK.
  • ReplayRecordingOptions takes providerId, bridge: ProviderBridgeLaunch, profile?: ReplayProviderProfile (DEFAULT_REPLAY_PROFILE for a bridge with no provider child). ReplayProviderProfile lost its bridgeFamily key; dialect, env, rewriteRuntimeLine, prepareState stay.
  • replayRecordedCells takes recordingsRoot and a bridge(cell) callback instead of assuming the committed fixtures and a checkout.
  • replayRecording restores the recorded cwd in every line the bridge emits (the replay runs in a fresh temp workspace; a bridge that derives a command's cwd or a read path from cwd would otherwise never compare with its recording, and a re-recorded lane would carry the temp path).

testing/first-party-replay.ts (new, private): FIRST_PARTY_BRIDGE_MODULES, resolveReplayProfile (codex / claude-code / acp-* / pi), UnreplayableProviderError, resolveBridgeLaunch(checkoutRoot, providerId), firstPartyReplayBridge, and replayFirstPartyRecordedCells — the checkout-keyed wrapper the three first-party bridge.recorded-conformance.test.ts suites now call.

testing/rerecord.ts (new): rerecordCurrentBridgeLane — the lane writer behind pnpm rerecord (anchoring, fractional seq, recorded request-id restoration), minus the first-party launch and the redaction pass. @bb/provider-parity's rerecord.ts and index.ts become thin consumers (launch, per-cell loop, redact.mjs).

@get-bb/plugin-sdk/provider-bridge/testing re-exports the harness with the experimental_ prefix (experimental_replayRecording, experimental_compareParity, experimental_assembleRecordedEvents, experimental_replayRecordedCells, experimental_checkRecordedCellReplay, experimental_rerecordCurrentBridgeLane, experimental_resolveProviderBridgeLaunch, experimental_resolveProviderBridgeBootstrapPath, the normalizers and the recording readers) plus the types. scripts/build-runtime.mjs bundles the bootstrap (with the daemon's ESM require banner — it pulls cross-spawn through @bb/process-utils) and copies replay-provider-child.mjs into dist/, where the harness resolves both by import.meta.url.

examples/plugins/echo-provider:

  • recordings/echo-agent/turn-tools/: a real recording. A dev bb's host daemon ran the plugin's built artifact with BB_PROVIDER_BRIDGE_RECORD_DIR in its env, bb plugin install <path> + bb plugin config echo-provider set shout true, bb thread spawn --provider echo-agent, then scripts/provider-recordings/package-cells.mjs (redacted; 0 survivors). 3 runtime lines, 7 bridge lines; the echo bridge has no provider child, so the provider lanes are empty.
  • provider-bridge.parity.test.ts (the acceptance criterion): imports only the public kit; replays the cell through src/provider-bridge.ts via experimental_resolveProviderBridgeLaunch, diffs against the recording's own assembled events, runs experimental_checkRecordedCellReplay, and round-trips experimental_rerecordCurrentBridgeLane on a copy of the recording (new lane keeps the recorded cwd, names the recorded echo-req-1, assembles to the same events).
  • src/provider-bridge.ts: the delegation's childRef is keyed on the bb thread id instead of the minted provider thread id. childRef is a value the normalizer does not intern (it interns item and turn ids), so a ref carrying per-process entropy could never reproduce across a replay. Found by the new test.
  • README: the parity test in the proof list and a "Re-record" section.

Docs: docs/provider-bridge-protocol.md (the harness is published, how a profile points a child at the replay script), docs/api_to_audit.md (the testing kit's entry: the new members and three audit items — profile/dialect shape, the two shipped programs, the workspace-path substitution), the bb-plugin-authoring skill (a "Recorded replay" paragraph).

No wire change; HOST_DAEMON_PROTOCOL_VERSION untouched.

How you verified

  • pnpm exec turbo run typecheck test --filter=bb-plugin-echo-provider: 5 files, 21 tests pass. The parity self-run: 32 recorded events → 32 replayed, 0 diffs, 0 grammar drops, 0 stalls, exit 0; recorded/turn-tools/{replays, events-schema-valid, grammar, turn-lifecycle, not-empty} all pass. Before the childRef fix the same test failed with the delegation rows only-in-old/only-in-new (the ref carried the recording process's nonce) — the test finds real non-determinism.
  • The published path, not just the workspace: a script from /tmp importing packages/plugin-sdk/dist/provider-bridge-testing.js, launching the built artifact (~/.bb-dev/.../plugin-host-artifacts/echo-provider/<digest>/host.mjs) through the bundled bootstrap with no tsx: stalls [] exit 0 events 32 -> 32 passed true diffs 0, five recorded-cell rules pass.
  • pnpm exec turbo run test --filter=@bb/provider-parity: 43/43 cells reproduce their recordings (unchanged).
  • pnpm exec turbo run test --filter=@bb/provider-bridge-protocol: 217 pass. --filter=@get-bb/plugin-sdk: 127 pass (typecheck, bundled-types, package-exports, doc tests).
  • The three first-party bridge.recorded-conformance.test.ts suites (codex, claude-code, acp) pass through replayFirstPartyRecordedCells.
  • apps/server echo-provider-canary.test.ts passes (the childRef change does not affect its assertions).
  • G1 ratchet (scripts/check-provider-literal-ratchet.mjs): not on this base (it lives on Add the provider-literal ratchet guardrail (G1) #2120); skipped.

What a third party still cannot do

  • Replay a provider child that speaks neither json-rpc nor the Claude CLI control protocol: ReplayDialect is those two (audit item 4).
  • Project rows: the row projector (@bb/thread-view) is still private, so compareParity from a plugin compares events and grammar drops with rows: []. ParityRowProjector is injected, but nothing public implements it.

AGENT GENERATED: by Claude Opus 5

The recorded-replay oracle (replayRecording, compareParity, the replay
child, rerecord) was private to @bb/provider-parity and keyed on a
first-party map: resolveReplayProfile threw UnreplayableProviderError for
any provider not in FIRST_PARTY_BRIDGE_MODULES, so a marketplace plugin
could record (record mode is public) but never replay or compare.

parity.ts is now provider-agnostic: the caller names the recording, the
bridge process (resolveProviderBridgeLaunch builds one from a module path
through the bootstrap, as the runtime spawns it) and, for a bridge with a
provider child, a ReplayProviderProfile. The first-party profiles, module
paths and the checkout-keyed wrappers move to testing/first-party-replay.ts
(private). The lane writer behind `pnpm rerecord` becomes
rerecordCurrentBridgeLane in testing/rerecord.ts; @bb/provider-parity is a
thin consumer (launch, per-cell loop, redaction). replayRecording restores
the recorded workspace path in the bridge's output, so a bridge that
derives paths from cwd compares with its recording.

@get-bb/plugin-sdk/provider-bridge/testing re-exports the harness with the
experimental_ prefix and ships the bootstrap bundle and the replay child
beside its bundle (the harness resolves both from import.meta.url).

The echo canary records a real cell (recordings/echo-agent/turn-tools,
captured on a dev bb with BB_PROVIDER_BRIDGE_RECORD_DIR and packaged with
the recordings scripts) and gains provider-bridge.parity.test.ts: replay
through the bridge with zero diffs and every recorded-cell rule green, plus
the re-record round trip, importing only the public kit. The bridge's
delegation childRef is keyed on the bb thread id instead of the minted
provider thread id so a replay can reproduce it.

Co-Authored-By: Claude <noreply@anthropic.com>
@SawyerHood SawyerHood changed the title Publish the parity replay harness through the plugin SDK testing kit [stacked on #2189] Publish the parity replay harness through the plugin SDK testing kit Aug 21, 2026
@SawyerHood

Copy link
Copy Markdown
Collaborator Author

Coordinator review of stack #2196 (#2193#2194#2195) — APPROVE pending CI. Correctly stacked on the canary #2189 (G-A's acceptance test lives there); gh stack sync re-targets when it lands.

G-A — the big one, verified: provider-bridge.parity.test.ts imports only @get-bb/plugin-sdk/provider-bridge/testing; the plugin still has zero @bb/* imports; the public kit exports experimental_replayRecording / experimental_compareParity / experimental_rerecordCurrentBridgeLane (prefixed per AGENTS.md), and I confirmed no first-party provider literals remain in the public parity path — it's genuinely keyed by providerId. The self-run replays a real recorded cell through the bridge as the runtime spawns it (32→32 events, 0 diffs, 0 grammar drops) and works from /tmp against the published dist with no tsx — that's the real third-party experience. First-party gates held (parity 43/43, protocol 217, SDK 127). Bonus: it caught a real non-determinism (delegation childRef carried per-process entropy) — exactly what a replay oracle is for.

G-B: canonical types exported by name, pinned in the bundled-.d.ts test. G-C: icon {glyph} projected onto ProviderInfo and drawn (svg[data-icon="Zap"] instead of "E"); the eight first-party projections are pinned against the plugins' own declarations; correctly no HOST_DAEMON bump (server→app only). A4 307/307 zero diffs against the main-minted baseline.

Process note (not on this stack): you were right to gate against the .bak — WS3 had rewritten the shared A4 baseline in place. I've restored snapshots/rows/ to the main-minted snapshot, reset the shared allowlist, and written the rule (snapshots/README-BASELINE.md): rows/ is always main; workstreams snapshot into a shadow dir with their own allowlist. Your gating was the correct instinct.

Honest residuals noted: ReplayDialect covers json-rpc + the Claude CLI dialect only; row-level parity stays private because @bb/thread-view is private (events + grammar drops for plugins). Both are audit items, not blockers.

Sawyer merges; I do not.

AGENT GENERATED: by Claude Opus 4.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant