fix(client): surface onAudioAlignment on WebRTC transport

[JoanDuarte]

Problem

VoiceConversation.handleAudio() fires the onAudioAlignment callback when an audio event with alignment metadata arrives, but on the WebRTC transport this callback never fires.

In WebRTCConnection.ts, the RoomEvent.DataReceived handler drops the entire JSON message when message.type === "audio", with the comment "Filter out audio messages for WebRTC - they're handled via audio tracks." That comment is true for the audio bytes (audio_base_64) — LiveKit's audio track plays them — but the alignment metadata (chars, char_start_times_ms, char_durations_ms) lives in the same message and gets dropped with it.

Net effect: consumers using onAudioAlignment on @elevenlabs/react-native or any WebRTC-transport @elevenlabs/client user receive nothing. The Swift SDK exposes alignment correctly; the WebSocket transport exposes alignment correctly; only WebRTC drops it.

Fix

Route audio messages through handleMessage() so VoiceConversation.handleAudio can fire onAudioAlignment, then return without further processing (since LiveKit audio tracks already handle playback).

 if (message.type === "audio") {
+  if (isValidSocketEvent(message)) {
+    this.handleMessage(message);
+  }
   return;
 }

Verification

Tested end-to-end against a live @elevenlabs/react-native@1.0.1 session on iOS. Prior to the fix onAudioAlignment never fired on WebRTC; after the fix it fires per audio chunk with the expected { chars, char_start_times_ms, char_durations_ms } payload, matching the WebSocket transport's behavior. Per-word karaoke transcript stays synced to agent speech end-to-end across a multi-turn conversation.

Side effects audited

VoiceConversation.handleAudio does the following after onAudioAlignment:

1. Fires onAudio(audio_base_64) if subscribed. Harmless on WebRTC for consumers who don't subscribe; if they do, raw bytes are what they opted into.
2. Updates currentEventId and canSendFeedback. Previously these didn't advance on WebRTC audio events, which was arguably a pre-existing bug (sendFeedback couldn't reference WebRTC event IDs).
3. Calls updateMode("speaking"). Already fires elsewhere on WebRTC via mode events; idempotent here.

No behavior regression observed.

Changeset

Patch-level bump to @elevenlabs/client. Changeset included.

---

Note

Low Risk
Low risk: a small change to WebRTC data-channel message routing to ensure audio events reach existing handlers; main risk is unintended side effects from processing additional audio messages (e.g., duplicate onAudio callbacks) if downstream consumers subscribe.

Overview
Fixes WebRTC conversations dropping type: "audio" data-channel messages by routing them through handleMessage (when valid) before returning, so VoiceConversation.handleAudio can emit onAudioAlignment from the alignment metadata.

Adds a patch changeset for @elevenlabs/client documenting the alignment callback fix on the WebRTC transport.

^{Reviewed by Cursor Bugbot for commit 10b65aa. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 10b65aa. Configure here.}

[cursor]

Duplicate onAudio callback from two audio paths

Medium Severity

In browser WebRTC environments, handleAudio now fires twice per audio chunk — once from the server's data-channel message (newly routed through handleMessage), and again from the existing setupAudioCapture worklet which re-captures LiveKit track audio and also calls handleMessage with a synthetic audio event. Both carry audio_base_64, so onAudio subscribers receive duplicate audio data. Additionally, currentEventId ping-pongs between the server's event_id and the local sequential audioEventId, potentially breaking sendFeedback tracking.

Additional Locations (1)

• packages/client/src/utils/WebRTCConnection.ts#L655-L677

 

^{Reviewed by Cursor Bugbot for commit 10b65aa. Configure here.}

[kraenhansen]

What do you think @JoanDuarte? Is this a valid concern which needs to be addressed?

[kraenhansen]

My initial reaction is that it's hard to align the timestamps in these events with the audio streaming over WebRTC. What's your experience in that regard?

[Dev43]

Confirming this on @elevenlabs/client@1.8.1. End-to-end repro with a Speech Engine WebRTC session: alignment is sent on the LiveKit _reliable data channel inside type: "audio" JSON messages.
I would love to see this merged.