fix(agent/media): 四项工具契约修复——链接音频/插入默认时长/whisper 标记过滤/静音分析跳过文本#204
Merged
Conversation
added 4 commits
July 6, 2026 21:08
…-audio (#196) The MCP add_clips and insert_clips tool bodies hardcoded add_linked_audio: false, so importing a video clip with an audio track never produced the linked audio partner clip, even though the underlying place_clip engine (opentake-ops) already implements the full linking logic (gated on target-video-track + source-video + has_audio) with existing unit test coverage. Flip both call sites to add_linked_audio: true. place_clip's own gate means this is a no-op for audio-only sources, image/text clips, or placements onto an audio track, so no new false-positive linking is introduced.
) insert_clips defaulted an omitted entries[].durationFrames to 0, which command.rs's insert_clips validator immediately rejects ("durationFrames must be >= 1"), so the tool could never actually use its documented "omit to use the full source length" behavior. Derive it instead from the referenced MediaManifestEntry's duration (source seconds, truncated to project frames via `(duration * fps) as i32`, matching the existing conversion idiom in this file's analysis_target), then subtract any requested trimStartFrame/trimEndFrame (already project-frame source offsets). Error clearly when the source has no known duration, or when the trims consume the entire source, instead of falling through to the generic "durationFrames must be >= 1" message.
whisper.cpp emits ordinary decoded text like "[BLANK_AUDIO]" over
silent/non-speech audio gaps (learned from its training captions, not
a special control token). Left unfiltered, this became a real 9s
"[BLANK_AUDIO]" caption clip via add_captions, and split into bogus
words via get_transcript's word-level path.
Add is_non_speech_marker (opentake-media/transcribe/mod.rs): matches
a segment/word whose ENTIRE trimmed content is one bracketed/
parenthesized filler marker ([BLANK_AUDIO], (inaudible), [MUSIC],
{BLANK_AUDIO}, [ Pause ], *music*, case-insensitive), covering the
bracket/case variants whisper.cpp is known to produce. The match is
anchored start-to-end and restricted to letters/underscores/spaces,
so real speech that merely contains a parenthetical (e.g. "he said
(hello) to her") or a bracketed number/word ("[42]") is left alone.
Wire it into WhisperTranscriber::transcribe_pcm at both the segment
and per-token level — the earliest point after whisper's own output,
shared by every consumer (add_captions, get_transcript, search_media's
spoken index) since they all read the same TranscriptionResult.
tighten_silences walked every clip on the selected track/clipIds (including a text/caption clip, whose media_ref is always "" — it has no source file), and unconditionally tried to extract analysis PCM for each one. For a text clip this always fails, surfacing a "media path not found for mediaRef: " warning on every call whenever a text clip happened to share the track (or be passed explicitly). Filter silence_targets to only clips whose media_type is Video or Audio — a whitelist, so it also covers any future non-audio/video overlay type (e.g. Lottie), not just Text — across all three selection modes (explicit clipIds, explicit trackIndex, and the audio-track auto-detect default).
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
四个独立缺陷修复,每项一个 commit,均带单测。
#196 链接音频
MCP add_clips/insert_clips 的 add_linked_audio 从硬编码 false 改为 true。引擎侧 place_clip 的条件链(video 轨 && video 源 && has_audio)自动跳过无音轨素材;命令层音频轨选择/新建链路经单测确认已接通。UI 路径(opentake-core/dto.rs)按任务范围未动,另行讨论。
Fixes #196
#197 insert_clips 默认时长
省略 durationFrames 时从 manifest 的素材 duration × 项目 fps 截断换算(移植铁律),再减 trim 偏移;结果 <1 时报清晰错误,不再无脑报 'got 0'。
Fixes #197
#198 whisper 非语音标记过滤
在紧贴 whisper 输出的最早公共层过滤整段仅为 [BLANK_AUDIO]/[MUSIC]/(inaudible) 形式的标记 token,字幕生成与 get_transcript 不再产出垃圾字幕(真机曾复现 9 秒 '[BLANK_AUDIO]' 字幕)。
Fixes #198
#200 tighten_silences 跳过无源媒体 clip
静音分析遍历跳过 text 等无 mediaRef 的 clip,不再冒 'media path not found' warning。
Fixes #200
验证
cargo fmt --check ✓ / cargo clippy --workspace -D warnings ✓ / cargo test -p opentake-agent -p opentake-media(600+ 项)✓