fix(providers): normalize GitHub Copilot Responses streams

[Simon-Opopeee]

Summary

• Normalize GitHub Copilot /v1/responses SSE streams before they reach Codex clients: remove provider-only encrypted/obfuscation fields, pin client-visible response and item ids, and emit canonical function/custom-tool completion events.
• Preserve the raw Copilot wire shape for inspection and continuation paths while applying the repair only to the GitHub Copilot client-facing relay.
• Keep the Copilot repair scoped to provider compatibility. The separate native Codex history-inspection bound requested during review is tracked in #1115.
• Closes [Provider] Normalize GitHub Copilot Responses SSE for Codex clients #1110. Related closed reports: GitHub Copilot streamed tool-call arguments are obfuscated, stalling agentic turns in Codex clients #989 and GitHub Copilot /responses wire incompatible with Responses clients (obfuscated deltas, encrypted reasoning, re-encrypted ids) #995.

Verification

• bun run test — 9,236 pass, 8 skip, 0 fail (9,244 tests across 583 files).
• bun test tests/github-copilot-sse-rewrite.test.ts tests/github-copilot-stream-contract.test.ts tests/sse-payload-rewrite.test.ts tests/sse-inspector-bounds.test.ts tests/passthrough-abort.test.ts — 47 pass, 0 fail.
• bun run typecheck — pass.
• bun run privacy:scan — pass.
• git diff --check — pass.
• Global package validation: ocx --version reports opencodex 2.10.2; ocx start and ocx sync stay alive/exit 0 with Codex integration enabled and openai_base_url at http://127.0.0.1:10101/v1.
• Live authenticated Copilot probes through 10101 using github-copilot/gpt-5.6-luna: reasoning, function-tool, and custom-tool streams all returned HTTP 200, canonical event names, response.completed, [DONE], zero encrypted_content, zero obfuscation, and no event-name mismatches.

Checklist

• Scope stays focused and avoids unrelated cleanup.
• Docs or release notes were updated when needed.
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

• All CI tests are green on my local testing.

• I pushed my PR to the latest dev commit.

• I resolved all correct Codex and CodeRabbit findings.

• My PR is ready for review.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change adds bounded rollout inspection and a stateful GitHub Copilot Responses SSE rewriter. It normalizes identifiers, sequences, reasoning fields, and tool-call data. The response pipeline composes rewrites, pins completed IDs, and handles dropped partial blocks.

Changes

Copilot Responses compatibility

Layer / File(s)	Summary
Bounded rollout inspection src/codex/native-residue.ts, tests/codex-native-residue.test.ts	Referenced rollout files are limited to 64 MiB, read through resolved paths, checked for post-read changes, and classified as indeterminate when inspection fails.
Copilot SSE block repair src/server/github-copilot-responses-repair.ts, tests/github-copilot-sse-rewrite.test.ts, tests/github-copilot-stream-contract.test.ts	Copilot SSE blocks normalize response and item IDs, remove encrypted and obfuscation fields, preserve readable summaries, reconstruct tool-call input from .done values, normalize sequences, and pass through malformed blocks.
Rewrite pipeline and response-ID pinning src/server/relay.ts, src/server/responses/core.ts, tests/passthrough-abort.test.ts, tests/sse-inspector-bounds.test.ts	Payload, Copilot, and snapshot rewrites compose independently. Inspectors can pin completed IDs to the first observed ID. The Copilot option is forwarded to eager and tee-based inspection.
Resilient block processing src/server/sse-payload-rewrite.ts, tests/sse-payload-rewrite.test.ts, tests/ci-workflows.test.ts	Pull processing continues after dropped blocks and releases translator budget state. The translation workflow test expects copilot-requests: write permission.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CopilotProvider
  participant BlockRewriter
  participant SSEInspector
  participant ResponsesRelay
  participant CodexClient
  CopilotProvider->>BlockRewriter: Send provider SSE blocks
  BlockRewriter->>BlockRewriter: Normalize IDs, sequences, reasoning, and tool calls
  BlockRewriter->>SSEInspector: Forward rewritten response records
  SSEInspector->>ResponsesRelay: Pin completed response ID when enabled
  ResponsesRelay->>CodexClient: Send canonical SSE blocks and [DONE]

Loading

Possibly related PRs

• lidge-jun/opencodex#990: Directly related to the GitHub Copilot SSE obfuscation rewrite expanded by this change.
• lidge-jun/opencodex#996: Directly related to the earlier Copilot Responses rewrite refactored by this change.
• lidge-jun/opencodex#350: Related to completed-response ID handling in src/server/relay.ts.

Suggested labels: chore

Suggested reviewers: ingwannu, lidge-jun

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The native rollout inspection limit in src/codex/native-residue.ts addresses a separate history-size concern rather than Copilot stream normalization.	Move the native rollout inspection changes and related test into a separate pull request, unless issue #1110 is expanded to include that scope.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The Copilot SSE repair meets issue #1110 requirements for stable IDs, canonical events, readable tool data, field removal, terminal completion, and raw-frame preservation.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the primary change: normalizing GitHub Copilot Responses streams.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Deterministic PR hygiene checks passed.

[github-actions]

⏳ DRAFT

• review readiness checklist open (3/4 boxes ticked).

What to do

• Tick all four boxes in the PR description once you're done (currently 3/4).

Review readiness checklist

• ✅ All CI tests are green on my local testing.
• ✅ I pushed my PR to the latest dev commit.
• ✅ I resolved all correct Codex and CodeRabbit findings.
• ⬜ My PR is ready for review.

3/4 boxes ticked.

This PR stays in draft until every box above is ticked.

[Simon-Opopeee]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/server/github-copilot-responses-repair.ts`:
- Around line 178-195: Update rememberItem to avoid throwing
TranslatorBudgetExceededError when MAX_TRACKED_ITEMS is reached; instead stop
retaining additional output indices and mark the repair as tainted, following
the degraded behavior used by relay handling. Ensure emitProcessedBlocks can
continue processing and streaming output beyond the retained-item limit while
preserving existing byte-budget enforcement.

In `@tests/github-copilot-sse-rewrite.test.ts`:
- Around line 89-93: Replace the hard-coded 137 split in the
relaySseWithBlockRewrite test with an offset derived from the upstream fixture
content, ensuring it falls within the intended frame rather than at a fixed byte
position. Keep the two-chunk stream split and existing assertions unchanged so
edits to the frame fixtures preserve mid-frame boundary coverage.

In `@tests/sse-inspector-bounds.test.ts`:
- Around line 61-78: Add a default-off test alongside the existing
pinCompletedResponseIdToFirstSeen test in the SSE inspector suite. Create the
inspector without setting the flag, feed the same client-visible creation and
upstream terminal events, and assert the completed response retains the upstream
terminal id rather than rewriting it.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c2776e7c-f08f-41bc-89e9-ac89bdeefbd5

📥 Commits

Reviewing files that changed from the base of the PR and between ef13178 and 560f346.

📒 Files selected for processing (12)

• src/codex/native-residue.ts
• src/server/github-copilot-responses-repair.ts
• src/server/relay.ts
• src/server/responses/core.ts
• src/server/sse-payload-rewrite.ts
• tests/ci-workflows.test.ts
• tests/codex-native-residue.test.ts
• tests/github-copilot-sse-rewrite.test.ts
• tests/github-copilot-stream-contract.test.ts
• tests/passthrough-abort.test.ts
• tests/sse-inspector-bounds.test.ts
• tests/sse-payload-rewrite.test.ts

[Simon-Opopeee]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[lidge-jun]

Reviewed in a sweep of the bug-labelled backlog. The Copilot work is sound and I found no defect in the repair path.

The gap is real: dev's passthrough builds rewrites only for image/item/snapshot repair and then relays the original Copilot bytes (src/server/responses/core.ts:2053,2074,2221), with no Copilot-specific normalization. Pinning the client-visible ids, dropping encrypted_content/obfuscation, and replacing untrustworthy tool deltas with the authoritative .done value is the right shape — and keeping the raw upstream wire for inspection and continuation means the repair does not destroy the evidence you would need to debug it later. The end-to-end test drives handleResponses rather than the helper alone, which is what makes it a real regression test.

Please split the native-history bound into its own PR. The 64 MiB cap on rollout inspection is a genuinely useful fix — dev currently reads and splits the entire file in memory (src/codex/native-residue.ts:357-404), and a multi-gigabyte rollout taking down ocx start is a real failure. But it has nothing to do with Copilot stream normalization. Two unrelated changes in one PR means the reviewer with Copilot context and the reviewer with startup-path context are the same person, and whichever half raises a question blocks the other. Split, and both land faster.

Second blocker: the verification claim. Your description says the full suite was interrupted, which I appreciate you stating rather than rounding up. The readiness gate needs a green ci on the exact head before it will accept a completion, so the suite does need to finish.

Your branch is on the latest dev commit, which is unusual in this queue and makes both of the above cheap to do now.

Good instinct on preserving the raw wire shape while repairing the client-facing view. That is the distinction most stream-normalization patches get wrong.

[Simon-Opopeee]

Thanks for the review. I addressed the requested scope and verification changes.

• The native 64 MiB rollout-inspection change is removed from this PR and is isolated in #1115.
• PR fix(providers): normalize GitHub Copilot Responses streams #1111 now contains only the GitHub Copilot Responses compatibility work; no native-history files or tests remain in this diff.
• The exact PR head is 840a57c334094e47e6cdfd27107ddcab826d9200, rebased onto dev at a5b3782716eaa142b94fdffd62817d2a5be0a02c.
• The complete local suite finished on that exact HEAD: 9,236 pass, 8 skip, 0 fail across 9,244 tests and 583 files.
• The focused Copilot/SSE suite finished with 47 pass, 0 fail.
• bun run typecheck, bun run privacy:scan, and git diff --check all pass.
• The raw Copilot frames remain available for inspection and continuation; the normalization is applied only to the client-facing Copilot relay.

I also removed the now-obsolete native-history sentence from issue #1110 and updated this PR description to match the split and the exact verification results. The three previous CodeRabbit findings are resolved and their review threads are marked resolved.

The remaining repository CI workflows (Cross-platform CI and React Doctor) are currently action_required for the fork PR, rather than failing. I have left the PR in draft and the final readiness box unchecked until the repository-side approval allows the ci check to run and finish successfully on this exact HEAD.