Skip to content

feat(mobile): add on-device Pocket TTS - #3290

Draft
johnmatthewtennant wants to merge 5 commits into
jt/buzz-voice-refactorfrom
jtennant/mobile-pocket-tts
Draft

feat(mobile): add on-device Pocket TTS#3290
johnmatthewtennant wants to merge 5 commits into
jt/buzz-voice-refactorfrom
jtennant/mobile-pocket-tts

Conversation

@johnmatthewtennant

Copy link
Copy Markdown
Contributor

Context

Buzz Desktop can speak submitted assistant responses with the local Pocket TTS engine. Mobile had no equivalent, so voice playback required a separate product path or a network service.

Summary

This PR adds private, on-device Pocket TTS to Buzz Mobile. It reuses the shared buzz-voice engine and desktop text preparation, keeps the engine warm in a Dart worker isolate, downloads the 473 MB FP32 model after install, and integrates native audio behavior on iOS and Android.

Changes

  • Adds a minimal buzz-voice-mobile C ABI for persistent engine create/destroy, cancellable synthesis, PCM ownership, and shared chunk preparation.
  • Moves desktop prompt preparation and PCM shaping into buzz-voice so desktop and mobile use the same text cleanup, sentence grouping, padding, clamp, and fade behavior.
  • Builds against the official sherpa-onnx 1.13.4 iOS and Android artifacts with pinned archive checksums.
  • Downloads the pinned Pocket TTS FP32 ONNX model into Application Support/files with per-file checksums, progress, cancellation, timeout handling, disk-space checks, atomic replacement, recovery, retry, and stale-download cleanup.
  • Keeps the model out of Flutter assets and app bundles. The downloaded runtime payload is 473,251,634 bytes.
  • Runs synthesis in a persistent Dart worker isolate and queues PCM chunks in order without blocking the UI.
  • Adds sticky cancellation with explicit reset, warm-engine startup, stable failure handling, and interruption-safe lifecycle behavior.
  • Configures iOS AVAudioSession for spoken playback and Android audio focus, routes, interruptions, and bounded AudioTrack draining.
  • Speaks submitted assistant plain text and uses submitted user text to steer/cancel the current response.

Related issue

N/A. No matching issue or PR was found.

Testing

  • cargo test -p buzz-voice -p buzz-voice-mobile: 38 shared voice tests and 2 ABI tests passed.
  • cargo clippy -p buzz-voice -p buzz-voice-mobile --all-targets -- -D warnings: passed.
  • flutter test: 844 tests passed, 1 expected skip.
  • ./gradlew :app:testDebugUnitTest: passed, 303 tasks completed.
  • ./scripts/mobile-voice-native.sh build-ios: passed for iOS device arm64 and simulator arm64/x86_64.
  • ./scripts/mobile-voice-native.sh build-android: passed for Android arm64-v8a and x86_64.
  • Full iOS Simulator app build with xcodebuild: passed.
  • Android emulator end-to-end download, engine load, PCM synthesis, warm synthesis, playback, and cancellation: passed.
  • App artifact inspection confirmed the 473 MB Pocket model is not present in the Android APK or iOS app bundle.

Screenshots

The Pocket Voice settings state is attached in the screenshot comment generated by the repository screenshot workflow.

Reviewer-reproducible examples

Android end to end

Use an Android emulator or device with at least 1 GB of free application storage:

cd mobile
export JAVA_HOME=$(/usr/libexec/java_home -v 17)
flutter test integration_test/pocket_voice_integration_test.dart \
  -d <android-device-id> \
  --dart-define=BUZZ_POCKET_DOWNLOAD=true

Observed on the isolated arm64 emulator:

first_pcm_ms=827
synthesis_ms=825
audio_s=2.560
rtf=0.322
playback_ms=2632
warm_pcm_ms=726
cancel_ms=814
download_ms=80653
rss_bytes=1038196736
peak_rss_bytes=1041985536

The test downloads and verifies every model asset, creates the resident engine, asserts 24 kHz mono signed-16-bit PCM, performs a warm synthesis, exercises native playback, and verifies prompt cancellation.

Native packaging

./scripts/mobile-voice-native.sh build-ios
ANDROID_NDK_HOME="$ANDROID_SDK_ROOT/ndk/28.2.13676358" \
  ./scripts/mobile-voice-native.sh build-android

Observed output:

iOS: release static libraries built for iphoneos-arm64,
     iphonesimulator-arm64, and iphonesimulator-x86_64
Android: release cdylibs built for arm64-v8a and x86_64

Shared engine contract

cargo test -p buzz-voice -p buzz-voice-mobile
cargo clippy -p buzz-voice -p buzz-voice-mobile --all-targets -- -D warnings

Observed output:

buzz-voice: 38 passed
buzz-voice-mobile: 2 passed
clippy: passed with warnings denied

Notes

  • Pocket synthesis remains stochastic because the sherpa Pocket API does not expose a reproducible seed. This PR preserves the desktop quality semantics.
  • The pinned model is the full-precision January 2026 Pocket ONNX export. Moving to Kyutai's April 2026 model requires a compatible sherpa/ONNX export and a separate quality and performance evaluation.
  • Lowest-memory physical iPhone and Android validation remains a rollout gate because no safe dedicated low-memory device was available for this run.

Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
@johnmatthewtennant

Copy link
Copy Markdown
Contributor Author

🤖 ### Pocket Voice settings

The post-install model download exposes size, progress, cancellation, retry, and the on-device privacy boundary.

mobile-pocket-voice-settings

johnmatthewtennant added a commit that referenced this pull request Jul 28, 2026
Signed-off-by: John Tennant <jtennant@squareup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant