Skip to content

feat(desktop): auto-enable huddle transcription for agents - #3180

Draft
johnmatthewtennant wants to merge 9 commits into
mainfrom
jtennant/auto-transcribe-agent-huddles
Draft

feat(desktop): auto-enable huddle transcription for agents#3180
johnmatthewtennant wants to merge 9 commits into
mainfrom
jtennant/auto-transcribe-agent-huddles

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Context

Agents in Buzz huddles need voice transcription so they can receive and act on spoken conversation. Transcription previously depended on a user enabling it manually, including when an agent was already present when the huddle started.

The huddle participant role is authoritative. A participant is treated as an agent only when membership identifies it with the bot role or the existing agent identity model identifies the DM participant as an agent.

Live staging validation also exposed two enrollment defects. Starting a huddle from an agent DM did not enroll the agent, and manually adding an agent who was already an active parent-channel member produced a role-mutation warning after ephemeral enrollment had succeeded. Dictation and transcription worked after selecting the correct input device, so this PR does not change the speech-to-text model path.

Summary

Automatically enable Buzz voice transcription once when the first agent is present in a huddle, including initial membership hydration and reconnects.

Explicit user control remains authoritative. If a user enables or disables transcription, reconnects and UI remounts preserve that choice. When the last agent leaves, Buzz leaves the current transcription state unchanged.

Starting a huddle from an agent DM now enrolls that agent automatically. Adding an agent who is already active in the parent channel preserves the existing role and completes without a warning.

Changes

  • Detect agent presence from authoritative bot-role huddle membership.
  • Auto-enable transcription during huddle creation, join, post-connect membership hydration, pipeline hot start, and local agent addition.
  • Notify the React huddle state when the backend auto-enables transcription.
  • Preserve explicit user enable/disable state and avoid repeated automatic toggles.
  • Guard asynchronous create, join, membership, audio, and speech-to-text setup work with a huddle lifetime generation so superseded work cannot alter a replacement huddle.
  • Prevent a delayed bootstrap snapshot from overwriting a newer huddle state event.
  • Enroll known agent participants when a huddle starts from an agent DM or owned-agent profile.
  • Preserve any active parent-channel membership instead of rewriting its role. A failed parent add is treated as idempotent only when a follow-up authoritative membership read confirms the agent is active.
  • Add focused Rust and deterministic desktop E2E coverage for transcription state, hydration ordering, agent-DM enrollment, owned-agent profile enrollment, and the idempotent add-agent dialog result.

Related issue

N/A.

Screenshots

Not applicable. This changes huddle behavior without adding a new visual surface.

Reviewer-reproducible examples

Live red reproduction on the published Desktop settings and streaming build, PR #3257 at 93602d9e013684d716d80d14ed8d1ee87a986b6f:

  1. Open a direct message with a running agent.
  2. Start a huddle from the DM.
  3. Observe that the agent is not enrolled in the ephemeral huddle.
  4. Add the agent with the robot control.
  5. Observe successful ephemeral enrollment followed by Added to huddle, but parent channel failed: relay returned 400 Bad Request: invalid: only owners/admins may change an active member's role.

Deterministic green reproduction from a fresh checkout of this branch:

pnpm --dir desktop build:e2e
pnpm --dir desktop exec playwright test tests/e2e/huddle-transcription.spec.ts --project=smoke
pnpm --dir desktop exec playwright test tests/e2e/mentions.spec.ts --project=smoke --grep "system agent profile exposes owned agent actions|system agent avatar exposes owned agent actions|owned bot profile exposes message and huddle actions|owned agent mention profile exposes message and huddle actions"

Observed output:

4 passed
4 passed

The huddle spec verifies authoritative initial agent hydration, one explicit OFF command, remount persistence, unchanged state after last-agent removal, a newer event winning over delayed hydration, agent-DM start payload enrollment, and an idempotent parent-membership result closing the add-agent dialog without a warning. The profile scenarios verify owned agents expose the huddle action and the start payload includes the agent pubkey.

Remaining live branch smoke:

  1. Launch this branch as the real Tauri app on a Mac with microphone permission, a cached speech-to-text model, and a live relay.
  2. Select the intended physical microphone or BlackHole 2ch input.
  3. Open a direct message with a running agent and start a huddle.
  4. Verify the agent is enrolled without using the robot control and the transcript control turns on.
  5. Speak or inject a distinctive phrase and verify its transcript reaches the ephemeral huddle.
  6. Turn transcription off, reconnect or remount the UI, and verify it remains off.
  7. In a huddle whose agent is already an active parent-channel member under another role, add the agent and verify the dialog closes without the parent-role warning.

The staging reproduction confirmed that dictation works with the correct input device. The remaining smoke validates this branch's native app, live relay membership, and audio-device integration together.

Testing

Run against head 2ff30f3d87af50624a718595037d4a5fd909fc2b:

cd desktop
pnpm exec biome check src/features/profile/ui/UserProfilePopover.tsx src/features/channels/ui/ChannelMembersBar.tsx src/features/huddle/components/HuddleBar.tsx src/testing/e2eBridge.ts tests/e2e/huddle-transcription.spec.ts tests/e2e/mentions.spec.ts tests/helpers/bridge.ts
pnpm exec tsc --noEmit
pnpm check:file-sizes
pnpm build:e2e
pnpm exec playwright test tests/e2e/persona-sync.spec.ts --project=integration
pnpm exec playwright test tests/e2e/huddle-transcription.spec.ts --project=smoke
pnpm exec playwright test tests/e2e/mentions.spec.ts --project=smoke --grep "system agent profile exposes owned agent actions|system agent avatar exposes owned agent actions|owned bot profile exposes message and huddle actions|owned agent mention profile exposes message and huddle actions"
cd ..
cargo fmt --manifest-path desktop/src-tauri/Cargo.toml --all -- --check
cargo test --manifest-path desktop/src-tauri/Cargo.toml huddle::agents::tests::existing_parent_membership_is_preserved_regardless_of_role
git diff --check

All commands passed. After rebasing onto current main, the persona-sync integration spec passed 2/2, the huddle smoke spec passed 4/4, and the four focused profile/mention scenarios passed 4/4. The huddle E2E spec had also passed three consecutive runs after its mocked event channel was made deterministic.

just desktop-tauri-test used the repository-owned sidecar setup and ran 1,838 tests. It reported 1,812 passed, 14 ignored, and 12 failed because the restricted local environment denied local socket binds. None touched huddle code.

The existing Builderbot, Kalvin, and minimize-diff reviews found no actionable issues at 481a73dbc88708b3af6e76ac0cb9df3027c8cf04. Dead Code found no issues on the rebased implementation head. Timeless Comments identified the membership-refresh comment as under-documenting its auto-enable side effect; the comment was corrected, and the final selective rerun found zero issues on 2ff30f3d87af50624a718595037d4a5fd909fc2b.

@johnmatthewtennant
johnmatthewtennant marked this pull request as ready for review July 27, 2026 19:18
@johnmatthewtennant
johnmatthewtennant requested a review from a team as a code owner July 27, 2026 19:18
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/auto-transcribe-agent-huddles branch from 67b4842 to 55a674c Compare July 27, 2026 19:51
@johnmatthewtennant
johnmatthewtennant marked this pull request as draft July 27, 2026 19:53
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/auto-transcribe-agent-huddles branch 4 times, most recently from 9e14c31 to 7b152da Compare July 28, 2026 14:45
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/auto-transcribe-agent-huddles branch from 7b152da to 2ff30f3 Compare July 28, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant