Skip to content

perf(desktop): stream Pocket TTS decoder chunks - #3257

Draft
johnmatthewtennant wants to merge 5 commits into
jtennant/pocket-tts-settings-v1from
jtennant/pocket-tts-streaming
Draft

perf(desktop): stream Pocket TTS decoder chunks#3257
johnmatthewtennant wants to merge 5 commits into
jtennant/pocket-tts-settings-v1from
jtennant/pocket-tts-streaming

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Context

Pocket TTS currently waits for an entire synthesized text chunk before it queues any audio. For longer agent replies, that adds avoidable latency even though sherpa-onnx exposes each decoded Pocket audio block synchronously while generation is still running.

Summary

This change streams each independent Pocket Mimi decoder block into Buzz's persistent playback queue as soon as it is available. It retains only a short final tail so fade-out is applied exactly once, preserves the existing sentence and cross-message lookahead pipeline, and keeps local and remote barge-in responsive while synthesis is still running.

Changes

  • Exposes Pocket synthesis with a progress callback through the reusable buzz-voice primitive.
  • Adds a callback-to-playback assembler that copies independent decoder blocks, clips only out-of-range samples, retains the final fade tail, and validates the callback/final-audio contract.
  • Starts playback before synthesis completes while continuing to synthesize and queue later blocks and messages asynchronously.
  • Separates audible playback activity from synthesis-in-flight state so silent decoder gaps reopen microphone input without disarming barge-in.
  • Preserves the real playback-tail cooldown, voice-preview completion, quiet first-word onsets, and voice transitions across streaming gaps.
  • Stops callback queueing immediately when cancellation, voice changes, shutdown, or playback errors occur.

Related issue

N/A. No related issue was supplied.

Testing

Passed locally:

  • cargo fmt --manifest-path desktop/src-tauri/Cargo.toml --all -- --check
  • cargo clippy --manifest-path desktop/src-tauri/Cargo.toml --lib -- -D warnings
  • Focused TTS, STT gap, voice settings, and agent enqueue/truncation tests listed below
  • Normal pre-push checks: desktop check, 3,734 desktop tests, 1,835 Tauri tests with 14 hardware/keychain tests ignored, 824 mobile tests with 1 intentional skip, and all configured Rust unit suites
  • Final NCMB reconciliation against the current Settings parent
  • Final timeless-comments, dead-code, and Stacksmith reviews with no findings

Manual audio validation remains required because time-to-first-audio, decoder-boundary quality, and physical microphone barge-in depend on local audio hardware and cannot be exercised by this headless task.

Screenshots

Not applicable. This changes the native audio pipeline and has no visual UI change.

Reviewer-reproducible examples

From a fresh checkout:

. ./bin/activate-hermit

cargo test \
  --manifest-path desktop/src-tauri/Cargo.toml \
  huddle::tts:: \
  --lib

cargo test \
  --manifest-path desktop/src-tauri/Cargo.toml \
  huddle::stt::tests \
  --lib

cargo test \
  --manifest-path desktop/src-tauri/Cargo.toml \
  huddle::tts_settings::tests \
  --lib

cargo test \
  --manifest-path desktop/src-tauri/Cargo.toml \
  agent_tts_routing_tests \
  --lib

Observed result:

huddle::tts: 56 passed, 0 failed
huddle::stt::tests: 2 passed, 0 failed
huddle::tts_settings::tests: 17 passed, 0 failed
huddle::agent_tts_routing_tests: 2 passed, 0 failed

The focused suites prove that playback receives PCM before generation finishes, callback blocks retain their order without duplication, quiet speech is preserved, microphone gating reopens during silent decoder gaps, local and remote barge-in remain armed, playback cooldown survives subsequent synthesis, voice preview waits across gaps, cancellation stops queueing, and the Settings-owned enqueue/truncation boundary remains intact.

Manual hardware check:

  1. Run just dev and select a Pocket voice in Agent text to speech settings.
  2. Join a huddle and trigger a multi-block agent reply.
  3. Confirm the first audio block starts before the reply finishes synthesizing and block boundaries have no duplication, omission, or clipping.
  4. Speak during a silent decoder gap and confirm microphone input is retained while the remaining TTS is cancelled.
  5. Preview a voice and confirm the preview plays through decoder gaps to completion.

@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/pocket-tts-settings-v1 branch from 4891a92 to c118f5a Compare July 28, 2026 04:48
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant