docs: roadmap — evals first (v0.4), then Speed: fewer agent turns (v0.5)#26
Merged
Conversation
Insert a new phase above evals: speed measured in agent turns and tokens-per-turn, not wire latency. Pull-shaped only — broad push pre-analysis across tabs is explicitly rejected (debugger conflicts, wasted work, permission-tier violations), as is a REPL mode. Ship-now list in win-per-effort order: error responses that embed the next observation, guarded chains (reins run with per-step assertions), SKILL.md turn-frugal patterns, snapshot epochs + deltas + structural compression, reins observe + anticipatory act payloads, declarative fill-form, sticky tab / digests / screenshot economics. Phase-later: paginate, self-healing refs, per-site recipe cache, closed-vocabulary semantic hints. Renumber subsequent phases (evals v0.5, depth v0.6, reach v0.7+) and move the old Phase 4 batch-mode bullet into Phase 2 as guarded chains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Daemon-side sticky tab breaks under concurrency: several agents drive several tabs through one daemon, and there is no reliable session identity to scope a pin by (fresh process per command, no persistent shell state across harness invocations) — agent A pins, agent B re-pins, A acts on B's tab. Fix the underlying hazard statelessly instead: acting commands stop defaulting to the active tab and error with the roster when more than one tab is driven, mirroring the existing multiple-browser rule. Read commands keep the convenience default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build the ruler before optimizing: the speed phase rewrites the agent-facing contract (error shapes, snapshot format, the skill's core loop), and without a baseline there is no way to prove a turn cut helped or didn't regress correctness. The eval fixture site is also the testbed the speed primitives are developed against. Eval phase gains a Baselines bullet (tool-calls / tokens / wall-clock per task recorded on every run) and a note that trajectory assertions track whatever SKILL.md currently teaches. The speed phase's shipping bar becomes: baselines move, execution evals stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
karngyan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Roadmap restructure: adds a Speed phase and puts evals before it.
Why evals first
The speed phase rewrites the agent-facing contract (error shapes, snapshot format, the skill's core loop). Without a baseline there's no way to prove a turn cut helped — or didn't regress correctness — except user bug reports. Build the ruler, then optimize. The eval fixture site (login wall, paginated table, form wizard) doubles as the testbed the speed primitives are developed against.
New in the eval phase: a Baselines bullet — every run records tool-calls / tokens / wall-clock per task; those numbers are what every speed item is measured against, and the regression tripwire after it lands.
The speed phase
Speed is measured in agent turns and tokens-per-turn, not wire latency — IPC is milliseconds, LLM inference between commands is seconds. Routine flow today: 5–6 turns.
Rejected: broad push/pre-analysis across tabs (debugger conflicts, wasted work on untouched tabs, reads sites before directed — violates the v0.3 permission tiers); REPL mode (harnesses are one-command-per-turn; the daemon already holds session state). Concurrency is a standing constraint: several agents drive several tabs through one daemon and there's no reliable session identity — so no daemon-side "current tab" state; acting commands error with the roster when more than one tab is driven, mirroring the multi-browser rule.
Ship-now, win-per-effort order:
reins runwith per-stepexpect:assertions, halt-with-delta; deterministic, not a mini-agent&&batching, API-before-DOM, don't re-snapshot unless told)50× listitem {…})reins observe+ anticipatory payloads on acting commandsreins fill-form— deterministic matching, ambiguity = report not guessscreenshot --if-changed/--refcropsPhase-later:
paginate, self-healing refs (audit-logged), per-site recipe cache (origin-exact, selectors-only, tier enforcement extension-side), closed-vocabulary semantic hints (enums the page can trigger but never author).Shipping bar: Phase 2 baselines move in the right direction and execution evals stay green — a speed item that buys turns by regressing correctness doesn't ship.
🤖 Generated with Claude Code