feat(studio): give an agent eyes with studio_frame

[miguel-heygen]

What

studio_frame renders the composition to a PNG at a given time and returns its URL.

Stacked on #3515.

Why

This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust.

No agent can judge motion from source. "What does this look like at 2.4 seconds" is not a question a file answers, and it is the question that matters when the thing you are building is a video.

How

Reuses Studio's existing capture endpoint through buildFrameCaptureUrl rather than inventing a second one. The server renders with Puppeteer, so the frame reflects the file on disk.

Two things the tool refuses to fake:

It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion.

It waits before capturing, 150ms by default. The render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the pre-edit composition. That exact staleness was a real bug in this repo once, and the fix is a watcher, not a synchronous invalidation, so the window still exists. An agent reading a stale frame as "my edit failed" would thrash. The wait is on by default, settleMs makes it tunable, and the tool description names the failure mode rather than leaving it to be rediscovered.

It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render.

Test plan

• Unit tests added/updated
• Manual testing performed
• Documentation updated (if applicable)

12 tests in frameTools.test.ts:

• URL shape, including the time actually captured.
• Seeks first when given a time; captures where the playhead landed, not what was asked (asserted with a clamping player).
• Asserts the wait happens BEFORE the probe, by recording call order. Without that this test would pass even if the settle were a no-op.
• Settle time is honoured, clamped at 5s, defaulted on nonsense input, and skipped entirely at zero.
• A renderer 500 becomes a failure with a hint, not a dead URL.
• No project open fails before touching the renderer.
• Negative and non-finite times are rejected without seeking.

Full package suite 4555 passing across 410 files. bunx tsc --noEmit clean. bunx fallow audit --fail-on-issues clean.

Not yet exercised against a real renderer; that is part of the end-to-end capture in the final unit.

[vanceingalls]

Approve — the design does the work here.

• Return-time = landed-time is the right contract. url and time are built from the same readPlayhead() after the seek/settle, so they can never disagree. The tool description names the failure mode ("attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion"), and the clamping test in frameTools.test.ts pins it. This is the kind of invariant that only survives if it is asserted; here it is.
• Settle-before-probe has a call-order test. expect(order).toEqual(["wait:150", "probe"]) would fail if the settle silently became a no-op — most codebases test that the settle happens, not that it happens first. Given the whole point is masking the file-watcher window, that ordering is the load-bearing bit.
• { readOnlyHint: true, untrustedContentHint: true } on the tool annotations. The return is a URL a model will dereference into pixels. Flagging the payload as untrusted content is the right defensive shape and matches the trust boundary the tool actually crosses.

One small observation, not a blocker: unlike studio_seek in #3515, studio_frame does not verify the seek landed. A silently-failed seek (no adapter mounted) would return time: <pre-seek playhead> and the agent would likely read that as "player clamped" rather than "seek never fired." Cheap to add — compare readPlayhead() before and after wait() when input.time !== undefined, and fold into the failure surface — but happy to defer to a follow-up if you would rather keep this PR focused.

Reviewed head 57c9bb0

Review by Via

The base branch was changed.

[james-russo-rames-d-jusso]

🟢 hyperframes#3516 — feat(studio): give an agent eyes with studio_frame — at 0fffa662 — leaving as COMMENT (LGTM from my side, no blockers).

CI (green on latest attempt, 0fffa662)
27 SUCCESS / 10 SKIPPED / 0 FAILURE on the attempt that completed 2026-08-30 05:23–05:38. The 8 ruleset-required contexts all landed green on this attempt: Lint, Format, Typecheck, Build, Producer: unit tests, Producer: integration tests, Test, Studio: load smoke. Also green: Studio: timeline viewport gate, SDK: unit + contract + smoke, Test: runtime contract, Render on windows-latest, Tests on windows-latest, Preview parity, preview-regression, regression, player-perf, CodeQL (javascript-typescript / python / actions), File size check, Fallow audit, Smoke: global install, CLI smoke (required), Semantic PR title, CodeQL, WIP. Standard qualifier for this repo: dismiss_stale_reviews_on_push=false + require_last_push_approval=true, so a subsequent push needs a fresh pass; per-name status can hide a stale cancelled attempt.

Peer signals at HEAD
None at 0fffa662. Via's approval at 57c9bb0d was auto-dismissed when the branch rebased after #3515 merged — its non-blocker (seek-landing not verified) carries over (see note 2).

Traced claims (verified at source)

• Composition-boundary reuse is real, not surface-level. studioFrame in frameTools.ts calls buildFrameCaptureUrl in utils/frameCapture.ts, which builds the same /api/projects/:id/thumbnail/* URL that CompositionThumbnail.tsx in player/components/ and studioSelectionSnapshot.ts in utils/ build. Server-side, api.get("/projects/:id/thumbnail/*", ...) in packages/studio-server/src/routes/thumbnail.ts handles both, gated by a single-slot thumbnailGenerationCoordinator (concurrency=1) shared with the human-UI thumbnail path. Same worker, same mutex, same cache dir — the reuse claim is not aspirational.
• "Read-only" is honest from the client-visible surface. annotations: { readOnlyHint: true, untrustedContentHint: true } on the tool in useStudioAgentTools.ts. No client state is mutated — no forced re-mount, no selection change, no playhead motion unless input.time is passed (in which case requestSeek is the agent's own edit, not a hidden side effect). The server writes into .thumbnails/ on cache miss, but that's a disk cache keyed on content-hash of the composition HTML — not composition state — and it's the same cache the human-UI populates. Untrusted-content annotation is the right defensive shape for a URL a model will dereference into pixels.
• HEAD probe works. Hono's #dispatch in hono/dist/hono-base.js explicitly re-dispatches HEAD through the registered GET handler and strips the body (new Response(null, await GET(...))), so probeFrame with method: "HEAD" populates the .thumbnails/ cache on miss and the agent's follow-up fetch(url) hits cache — the cache-buster v=Date.now() is the same in both calls (same url string). Not double-render, contra a possible reading.
• Return-time = landed-time. URL and time are both built from the same readPlayhead() after the settle in frameTools.ts, so they can never disagree. Clamping test in frameTools.test.ts pins it explicitly (t=10.000 when player clamped time: 999).
• Settle-before-probe ordering pinned by call-order test. The expect(order).toEqual(["wait:150", "probe"]) assertion in frameTools.test.ts would fail if the settle silently became a no-op — most repos assert the settle happens, not that it happens first. Given the whole point is masking the file-watcher's write-stability window, ordering is the load-bearing bit and it's asserted.
• studio_inspect has no TOCTOU on getCurrentSelection. studioInspect in inspectTools.ts captures deps.getCurrentSelection() once at the top of the function and threads it through — the reference-equality check current?.element === element at the bottom uses the same captured value. Race between "human clicks another element" and "tool reads selection" resolves cleanly to whatever the tool saw at entry.

Non-blocking notes

1. PR body scope: studio_inspect isn't mentioned. The description is entirely about studio_frame, but the diff also adds studioInspect in inspectTools.ts (208 lines) + its test file (197 lines) + wiring in StudioAgentTools.tsx and useStudioAgentTools.ts — a second undisclosed agent-tool. Reviewers who read the body will miss it; log/analytics readers keying on studio_frame in the tool list will see studio_inspect land unannounced. Worth naming in the description, or splitting; not worth reworking merge on.

2. Seek-landing still not verified (carried from Via's dismissed R1). In studioFrame — deps.requestSeek(input.time) → wait(settleMs) → readPlayhead(). The tool is honest about "return the time actually captured," so a silently-failed seek (no adapter mounted) or a mid-settle human seek produces time: <playhead's-actual-value> and an agent reading motion draws the wrong conclusion. The failure mode "seek never fired" is indistinguishable from "player clamped to end" through this surface. Cheap defense: capture readPlayhead().currentTime before requestSeek, compare after wait; if input.time !== undefined and the delta is neither ≈ input.time nor plausibly-clamped-to-duration, surface as failed with a hint. studio_seek in #3515 has a landing-verify pattern worth mirroring here. Worth-fixing on the same beat as #3515, defer-to-followup fine.

3. Cache-buster v=Date.now() fights the server-side content-hash cache. buildFrameCaptureUrl in frameCapture.ts always sets v=<Date.now()>, and the server's cacheKey in thumbnail.ts folds urlVersionKey into the cache key. But the same cacheKey already includes sourceKey (SHA-1 of the composition HTML read via readFileSync), manualEditsKey, and motionKey — all read from the actual file on each request. So v adds nothing to invalidation correctness (server always sees the fresh file bytes) but guarantees a cache miss on every repeated call. Two studio_frame calls on unchanged content = two full Puppeteer renders, gated only by the concurrency=1 mutex. Not a blocker (mutex bounds the damage), but a resource smell — worth dropping v on the studio_frame path, or trusting the content hash across the board. Nit-plus.

4. studio_inspect empty-animations ambiguity. In describe() in inspectTools.ts, when isCurrentSelection === false, animations: [] and animationEditingBlocked: "animations are only readable for the current selection". That combination is correct, but the empty array reads as "this element has no animations" to a superficial parse — same shape as a selection with genuinely zero animations. An agent that keys on animations.length === 0 first will miss the block reason. Consider animations: null when unavailable so a null check disambiguates authoritative-empty from unknown. Design nit.

5. What's NOT in the diff. No test for studio_frame under a mid-settle human seek (the readPlayhead TOCTOU class); no integration test at the studio-server boundary asserting probeFrame's HEAD hits the same cache key the subsequent GET reads (protects against a future framework swap breaking Hono's HEAD-through-GET dispatch — see note 3 of traced claims); no distinguishing failure surface between "Chrome browser missing" (server logs it, returns 500) and "renderer crashed" (also 500) — the tool's reason is uniform "the renderer returned 500". None blocking, but each would sharpen the failure story an agent has to reason from.

Summary. The design does the work here: return-time = landed-time is the right contract, settle-before-probe is asserted at the order-of-operations level, readOnlyHint + untrustedContentHint correctly frame the trust boundary, and the endpoint-reuse claim holds up under a trace (same Puppeteer path, same coordinator, same cache dir). Cache-buster and seek-landing-verify are the two threads I'd pull on a follow-up; the PR-body scope gap and animations: null are cheap paperwork. Ready to ship from where I sit.

— Review by Rames D Jusso