Fix degraded AirPods/Bluetooth dictation accuracy#593
Conversation
rohithgoud30
force-pushed
the
fix/airpods-bluetooth-capture-format
branch
from
ecfe0d6 to
e309b8f
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ecfe0d664b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Greptile SummaryThis PR fixes three distinct capture-path defects that degraded dictation accuracy when AirPods or other Bluetooth headsets were the input device. The changes are tightly scoped and correctly implement the described fixes with no unrelated UI or behavioral changes.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
Addressed in a8fdd52: the gate is now installed as the first statement in the do block, before startPreferredAudioCapture(), so the first-audio event can never be reported before the gate exists. The start-failure catch path already cancels the pending gate, and the 2.5 s timeout task cannot fire mid-start since the start sequence is synchronous on the main actor. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8fdd52a7c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4068cf4cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Thanks for the PR! Seems like a good one but since this could overall break the build if I ship it without testing it for a while, how do you suggest we go with this? since it touches the bottleneck part of the code, little worried haha. I do not use headphones much, so I am not a reliable candidate to test this unfortunately. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd1740c8cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@altic-dev Totally fair! I suggest merging it into a beta or pre-release first and asking the folks from #452 to try it. I use AirPods Pro and Max daily, so I can test it heavily during the first week. |
|
down for that! if you want to tag em and see if you can bring the bros together on this, that would give me more confidence to send this in :) Thanks for understanding!! |
|
Hi @NACC96 and @kyrichen55, you both reported AirPods issues in #452. This PR changes how FluidVoice handles Bluetooth microphone format changes. Would either of you be willing to test a beta build once it is available? |
The direct capture backend cached the input stream's virtual format once at creation. AirPods switch A2DP->HFP when input IO starts, which can change the stream format after creation, so packets were decoded and labeled with a stale sample rate and byte layout. Capture start now re-reads and validates the format (and rebinds the stream listener if the stream object changed) before AudioDeviceStart. Block-based listeners on a private serial queue watch the stream's virtual format and the device's nominal sample rate; on a change the IOProc stops publishing misdecoded packets and the consumer surfaces an onFormatChange callback so the owner can rebuild capture. Teardown drains the listener queue before freeing to avoid a use-after-free.
The capture pipeline downsampled to 16 kHz with linear interpolation and no anti-alias filter, folding content above 8 kHz into the speech band. StreamingResampler wraps AVAudioConverter (max SRC quality, no priming) behind a stateful streaming API with reused buffers, a flush() that drains the converter's FIR tail at end of recording, and a self-reset when the source rate changes across passthrough interludes. Tests cover 16 kHz passthrough, 48->16 kHz length, alias suppression (10 kHz at 24 kHz source must not fold to 6 kHz), in-band power preservation, chunked-vs-one-shot continuity, and flush behavior.
The capture pipeline now converts to 16 kHz through StreamingResampler, flushes the converter tail when recording stops, and appends to the audio buffer under the pipeline lock so a flushed tail cannot land before in-flight packet samples during route recovery. A direct-capture format change (AirPods A2DP->HFP) schedules the existing audio route recovery, which rebuilds capture with a fresh format; a prepared instance whose format changed is never reused. onCaptureStarted now fires on the first accepted audio packet (2.5 s timeout fallback) instead of right after AudioDeviceStart, so the UI stops inviting the user to speak into a Bluetooth mic that is not yet delivering audio.
Arming the gate first guarantees the pipeline's first-audio event can never be reported before the gate exists, regardless of how the start sequence evolves. The start-failure catch already cancels the gate.
Use an acquire load for formatChanged in the IOProc, drain the listener queue in start() before clearing the flag so a stale queued invocation cannot re-trigger recovery, reset the output buffer frameLength in process() to match flush(), and reset a stale converter when the stream returns to native 16 kHz so flush() cannot drain a previous rate's tail.
A notification landing between the format read and the flag clear was drained and then wiped, starting capture with a stale format and no recovery trigger. The drain and clear now happen first, so any change after the clear leaves the flag set, and the stream-rebind path re-reads the format after the new stream's listener is armed.
captureRecordingContext, private AI prewarm, and the dictation prompt configuration now run before capture starts, so a quick stop that cancels the capture-started gate can no longer skip target-app context or process the transcript with a stale provider configuration. The gated callback now only shows the recording overlay.
rohithgoud30
force-pushed
the
fix/airpods-bluetooth-capture-format
branch
from
e0c88cd to
c01bcbb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c01bcbb77a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2 participants
Description
With AirPods as the input device, dictation was noticeably worse than with the built-in microphone: garbled or time-stretched transcriptions, missing first words, and lower accuracy overall. Three capture-path defects caused this, all tied to the Bluetooth A2DP to HFP route transition that happens when the AirPods microphone activates.
AudioDeviceStart. Block-based listeners (on a private serial queue, drained before teardown) watchkAudioStreamPropertyVirtualFormatandkAudioDevicePropertyNominalSampleRate, the IOProc stops publishing misdecoded packets the moment a change is flagged, and the existing audio-route recovery rebuilds capture with a fresh format.onCaptureStartedfired right afterAudioDeviceStartreturned, 1 to 2 seconds before a Bluetooth mic actually delivers audio, and pre-start samples are timestamp-trimmed, so the UI invited users to speak into a dead mic. It now fires on the first accepted audio packet, with a 2.5 s timeout fallback.StreamingResampler(AVAudioConverter, max SRC quality, reused buffers, stateful across hardware callbacks) replaces it, flushes the converter's FIR tail when recording stops so the end of the final word is preserved, and appends to the audio buffer under the pipeline lock so ordering is deterministic during route recovery.Built-in microphone behavior is unchanged (its format never changes and its first packet arrives within tens of milliseconds).
Type of Change
Related Issue or Discussion
Closes #592
Testing
swiftlint --strict --config .swiftlint.yml Sources(0 violations)swiftformat --config .swiftformaton the files this PR adds/changes (pre-existing violations elsewhere in ASRService.swift left untouched to avoid drive-by churn)xcodebuild test -project Fluid.xcodeproj -scheme Fluidpasses 29 tests, including 6 new StreamingResampler tests (alias suppression, in-band power preservation, chunked continuity vs one-shot, flush tail)Screenshots / Video
Notes
This does not eliminate the brief playback dip reported in #452 (the A2DP to HFP profile switch itself is macOS behavior), but it fixes the capture-side fallout of that transition. Bluetooth HFP microphone quality also remains physically below the built-in mic; the fix removes the app-side corruption (wrong-rate decode, lost first words, aliasing) so AirPods accuracy lands close to the route's actual capability. The one behavior change reviewers may notice: with Bluetooth input, the recording overlay appears when the mic is genuinely live (up to 1 to 2 seconds later) rather than immediately.