Skip to content

docs(DESIGN): cite the frame-sampling paper for Decision 1 #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Jncch merged 1 commit into from
Jul 25, 2026
Merged

docs(DESIGN): cite the frame-sampling paper for Decision 1 #4

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,13 @@ Rationale:
- Research on agent observation interfaces compared pixel-difference
sampling against CLIP-based and uniform sampling for screen
observation and found downstream accuracy converges. Added
sophistication does not buy accuracy here. (Link the paper in this
section when publishing.)
sophistication does not buy accuracy here.
Reference: Bojie Li and Noah Shi, "Agent-Computer Observation
Interfaces Enable Dynamic Computer Use", arXiv:2606.29472 (2026).
Their Section 6.2 ablation over five frame-selection variants
(uniform 1/3 FPS, pixel-diff, random, CLIP) finds all converge on
downstream accuracy; the default selector is a pixel-change gate
(skip if < 1% of pixels changed since the last frame).
- Zero dependencies and full determinism are product invariants
(see CLAUDE.md). SSIM libraries and models break both.
- Cost: the gate runs at 2 fps on user machines alongside a meeting
Expand Down
Loading