fix(stt): stop Whisper handlers finalizing turns on progressive audio - #451
Merged
andimarafioti merged 1 commit intoAug 12, 2026
Conversation
In --mode realtime the VAD emits progressive chunks while the user is still speaking, then a final chunk once speech ends. BaseSTTHandler.before_emit_output marks a turn revision complete as soon as it sees a Transcription, and should_process_input then rejects every later chunk of that revision as "input-after-final". The four Whisper-family handlers never read vad_audio.mode and always yielded Transcription, so the first progressive chunk closed the turn and the rest of the utterance was discarded. A full sentence reached the LLM as its first fragment. ParakeetTDTSTTHandler already branches on mode, which is why this only surfaced on languages Parakeet does not cover. Branch on the mode in whisper_stt_handler, faster_whisper_handler, lightning_whisper_mlx_handler and mlx_audio_whisper_handler, yielding PartialTranscription for progressive audio and leaving the final path untouched. No enable_live_transcription flag is needed in these handlers. VAD only emits progressive chunks when live transcription is on, since s2s_pipeline sets enable_realtime_transcription from --enable_live_transcription and vad_handler gates the progressive yield on it, so the mode check alone is sufficient. The regression test drives the real BaseSTTHandler hooks in run() order and leaves the input queue empty between chunks. Enqueuing the whole utterance up front trips the separate progressive-before-final guard, which discards the progressive chunks and lets the test pass against unfixed code. Verified against real model weights on all four backends, driven by the real VADHandler: whisper (openai/whisper-small), faster-whisper (small.en), whisper-mlx (small) and mlx-audio-whisper (whisper-small-mlx). Each truncated a 4s utterance to "I think" before the change and delivered it in full after. Fixes huggingface#412
andimarafioti
self-requested a review
August 12, 2026 09:18
This was referenced Aug 12, 2026
andimarafioti
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #412
Summary
vad_audio.modein the four Whisper-family STT handlers so progressiveaudio yields
PartialTranscriptionrather thanTranscriptiontests/test_whisper_progressive_transcription.py, covering all four backendsThe diagnosis and the shape of the fix are from @whitepepper355 in #412, including the
follow-up comment identifying
mlx_audio_whisper_handler.pyas a fourth affected file.Problem
In
--mode realtimethe VAD emits a progressive chunk while the user is still speaking,then a final chunk once speech ends.
BaseSTTHandlercloses a turn on the firstTranscriptionit observes:should_process_inputsubsequently rejects every later chunk carrying that(turn_id, turn_revision)asinput-after-final.ParakeetTDTSTTHandleravoids this by inspecting the mode atparakeet_tdt_handler.py:244. The four Whisper-family handlers never readvad_audio.modeand yieldedTranscriptionunconditionally, so the progressive chunkclosed the turn and the remainder of the utterance was discarded:
STT/whisper_stt_handler.pySTT/faster_whisper_handler.pySTT/lightning_whisper_mlx_handler.pySTT/mlx_audio_whisper_handler.pyParakeet is the default STT and covers roughly 25 languages. The defect is therefore
reached only after switching to a Whisper backend, which any language outside that set
requires.
Why the mode check alone is sufficient
ParakeetTDTSTTHandlergates its progressive path behindenable_live_transcription.An equivalent flag is not required in these handlers.
s2s_pipeline.py:500setsvad_kw.enable_realtime_transcriptionfrom--enable_live_transcription, andvad_handler.py:623gates the progressive yield on that value. Progressive chunkstherefore exist only when live transcription is enabled, and a handler that receives one
is already operating in that mode.
Verification
Each backend was exercised with real model weights, driven by the real
VADHandlerrather than synthetic chunk boundaries. The VAD emitted
['progressive', 'final']inevery case.
Reference transcription of the complete utterance:
whisperopenai/whisper-smallI thinkfaster-whispersmall.en, int8I think.whisper-mlxsmallI thinkmlx-audio-whisperwhisper-small-mlxI thinkBefore, on each backend:
After:
The
dropping staleline falls from one occurrence per turn to zero, which is thesignal #412 predicted.
On this branch: 923 passed, 1 skipped.
ruff check,ruff format --checkandmypy src/report no issues.Test design
The regression test drives the
BaseSTTHandlerhooks inrun()order and leaves theinput queue empty between chunks. Enqueuing the whole utterance up front instead trips
the separate
progressive-before-finalguard inshould_process_input, which discardsthe progressive chunks and allows the test to pass against unfixed code. Reverting
src/while retaining the test produces 8 failures and 8 passes.The optional backends are stubbed at import time so all four are covered on Linux CI,
where
faster-whisperandlightning-whisper-mlxare not installed.Out of scope
Exercising the transformers handler against real weights also surfaces an
IndexErrorat
whisper_stt_handler.py:120, wherepred_ids[0, 1]reads a text token rather than alanguage tag under transformers 5.6.2. That behaviour is #375 and is already addressed
by #378, so it is left untouched here.
This change does not attempt to quantify the added latency of running STT twice per
utterance, which #412 raises. Measuring it requires the full LLM and TTS chain.