From fd181a6c30b6d2586cfe08b13203add02b45b7fc Mon Sep 17 00:00:00 2001 From: Jncch <84889782+Jncch@users.noreply.github.com> Date: Sat, 25 Jul 2026 09:45:13 +0900 Subject: [PATCH] docs(DESIGN): cite the frame-sampling paper for Decision 1 Fill the "(Link the paper in this section when publishing.)" placeholder with the verified reference: Li & Shi, "Agent-Computer Observation Interfaces Enable Dynamic Computer Use", arXiv:2606.29472 (2026). Their Section 6.2 ablation over uniform/pixel-diff/random/CLIP frame selection finds downstream accuracy converges, with a <1% pixel-change gate as the default - the basis for the "simple pixel math, no learned models" decision. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WyitJEKkrbE4VYXnk2buwA --- DESIGN.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index ed6e3a7..a5067d3 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -14,8 +14,13 @@ Rationale: - Research on agent observation interfaces compared pixel-difference sampling against CLIP-based and uniform sampling for screen observation and found downstream accuracy converges. Added - sophistication does not buy accuracy here. (Link the paper in this - section when publishing.) + sophistication does not buy accuracy here. + Reference: Bojie Li and Noah Shi, "Agent-Computer Observation + Interfaces Enable Dynamic Computer Use", arXiv:2606.29472 (2026). + Their Section 6.2 ablation over five frame-selection variants + (uniform 1/3 FPS, pixel-diff, random, CLIP) finds all converge on + downstream accuracy; the default selector is a pixel-change gate + (skip if < 1% of pixels changed since the last frame). - Zero dependencies and full determinism are product invariants (see CLAUDE.md). SSIM libraries and models break both. - Cost: the gate runs at 2 fps on user machines alongside a meeting