Skip to content

Add on-device Whisper dictation for the chat composer - #2

Open
OurBigAdventure wants to merge 2 commits into
buildwithparallel:masterfrom
OurBigAdventure:feat/on-device-dictation
Open

Add on-device Whisper dictation for the chat composer#2
OurBigAdventure wants to merge 2 commits into
buildwithparallel:masterfrom
OurBigAdventure:feat/on-device-dictation

Conversation

@OurBigAdventure

Copy link
Copy Markdown

Summary

  • Adds a Dictate control that fills the chat composer from the microphone using Whisper Tiny in a local Web Worker.
  • Speech stays on the device: env.allowRemoteModels is disabled, and weights are staged at build time under /assets/whisper/models/ instead of calling a cloud speech API.
  • Uses fp32 ONNX graphs (encoder_model.onnx, decoder_model_merged.onnx). Quantized (q8) Whisper graphs fail under the current Transformers.js / ONNX Runtime combination (MatMulNBits missing scale).

This is separate from Add Voice, which attaches Codec2 or Opus audio to an LXMF message.

Why

Field use needs speech-to-text without sending audio off the device.

Test plan

  • Open a conversation, click Dictate, speak, stop, and confirm text lands in the composer.
  • Confirm the first use may take a few seconds to load the local model, and later uses reuse it.
  • Confirm dictation still works after a frontend rebuild (npm run download-whisper-assets then Vite).
  • Confirm Add Voice is unchanged.

OurBigAdventure and others added 2 commits August 14, 2026 11:29
Keep speech-to-text local so microphone audio never leaves the device for processing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Quantized q8 graphs fail session creation under Transformers.js 4.2 and the current ONNX Runtime MatMulNBits path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant