Skip to content

bench: evaluate agent continuation with --json-envelope - #55

Merged
yusukebe merged 1 commit into
yusukebe:mainfrom
yearth:codex/agent-continuation-bench-pr
Aug 1, 2026
Merged

bench: evaluate agent continuation with --json-envelope#55
yusukebe merged 1 commit into
yusukebe:mainfrom
yearth:codex/agent-continuation-bench-pr

Conversation

@yearth

@yearth yearth commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a deterministic 120-record HTML fixture with ground truth kept outside the agent workspace
  • add conformance and guided/ablated adoption prompts with a fixed multi-page budget
  • normalize Claude Code stream-json, correlate standalone ax calls through a benchmark-local shim, and grade answer, continuation protocol, access isolation, and run metrics
  • add a guarded 3+3+3 matrix runner and publish sanitized Sonnet 5 results in bench/RESULTS.md

Results

Condition Exact Protocol pass Envelope adoption
Conformance 2/3 3/3
Adoption guided 0/3 2/3 2/3
Adoption ablated 0/3 0/3 0/3

The report keeps answer correctness, protocol correctness, and active envelope adoption separate. It also discloses three earlier stopped matrix attempts and the resulting methodology changes, so these results are presented as adaptive and exploratory rather than confirmatory.

Raw session records, telemetry, generated fixtures, and ground truth remain untracked. No src/ files are changed.

Validation

  • bun test — 255 pass, 0 fail
  • bunx tsc --noEmit
  • bun run format
  • bun run build
  • replay grading of all nine retained runs remained gradable with access checks passing and published scores unchanged
  • independent final code review: ready to merge

Closes #53

@yearth

yearth commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

@yusukebe When you have a moment, could you take a look and let me know if you would prefer any changes to the benchmark design, results reporting, or scope before review? The current CI is green, and the PR remains limited to bench/ with no product-code changes. Thank you!

@yusukebe

Copy link
Copy Markdown
Owner

@yearth

Okay. I was feeling bad these days, but I'm back. I'll check it!

@yearth

yearth commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Glad to hear you're feeling better. Please take care of yourself, and there's no rush at all from my side. Thank you for taking a look when you have time!

@yusukebe yusukebe left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@yusukebe

yusukebe commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Hey @yearth

Super interesting! This is an ideaful and well-implemented. Looks good. Thanks!

@yusukebe
yusukebe merged commit 314cfd0 into yusukebe:main Aug 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench: evaluate agent continuation with --json-envelope

2 participants