Skip to content

fix(stt): stop Whisper handlers finalizing turns on progressive audio - #451

Merged
andimarafioti merged 1 commit into
huggingface:mainfrom
Mayankpratapsingh022:fix/whisper-progressive-partial-transcription
Aug 12, 2026
Merged

fix(stt): stop Whisper handlers finalizing turns on progressive audio#451
andimarafioti merged 1 commit into
huggingface:mainfrom
Mayankpratapsingh022:fix/whisper-progressive-partial-transcription

Conversation

@Mayankpratapsingh022

Copy link
Copy Markdown
Contributor

Fixes #412

Summary

  • branch on vad_audio.mode in the four Whisper-family STT handlers so progressive
    audio yields PartialTranscription rather than Transcription
  • add tests/test_whisper_progressive_transcription.py, covering all four backends

The diagnosis and the shape of the fix are from @whitepepper355 in #412, including the
follow-up comment identifying mlx_audio_whisper_handler.py as a fourth affected file.

Problem

In --mode realtime the VAD emits a progressive chunk while the user is still speaking,
then a final chunk once speech ends. BaseSTTHandler closes a turn on the first
Transcription it observes:

def before_emit_output(self, output: STTOut) -> None:
    if isinstance(output, Transcription):
        self._mark_completed_final_revision(output)

should_process_input subsequently rejects every later chunk carrying that
(turn_id, turn_revision) as input-after-final.

ParakeetTDTSTTHandler avoids this by inspecting the mode at
parakeet_tdt_handler.py:244. The four Whisper-family handlers never read
vad_audio.mode and yielded Transcription unconditionally, so the progressive chunk
closed the turn and the remainder of the utterance was discarded:

  • STT/whisper_stt_handler.py
  • STT/faster_whisper_handler.py
  • STT/lightning_whisper_mlx_handler.py
  • STT/mlx_audio_whisper_handler.py

Parakeet is the default STT and covers roughly 25 languages. The defect is therefore
reached only after switching to a Whisper backend, which any language outside that set
requires.

Why the mode check alone is sufficient

ParakeetTDTSTTHandler gates its progressive path behind enable_live_transcription.
An equivalent flag is not required in these handlers. s2s_pipeline.py:500 sets
vad_kw.enable_realtime_transcription from --enable_live_transcription, and
vad_handler.py:623 gates the progressive yield on that value. Progressive chunks
therefore exist only when live transcription is enabled, and a handler that receives one
is already operating in that mode.

Verification

Each backend was exercised with real model weights, driven by the real VADHandler
rather than synthetic chunk boundaries. The VAD emitted ['progressive', 'final'] in
every case.

Reference transcription of the complete utterance:

I think the weather today is really quite nice and I would like to go outside.
Backend Model Delivered to the LLM before After
whisper openai/whisper-small I think matches reference
faster-whisper small.en, int8 I think. matches reference
whisper-mlx small I think matches reference
mlx-audio-whisper whisper-small-mlx I think matches reference

Before, on each backend:

[progressive] -> LLM Transcription 'I think'
dropping stale STT input-after-final for turn=turn_1 rev=0
[final] REJECTED by should_process_input

After:

[progressive]  draft PartialTranscription 'I think'
[final] -> LLM Transcription 'I think the weather today is really quite nice and I would like to go outside.'

The dropping stale line falls from one occurrence per turn to zero, which is the
signal #412 predicted.

On this branch: 923 passed, 1 skipped. ruff check, ruff format --check and
mypy src/ report no issues.

Test design

The regression test drives the BaseSTTHandler hooks in run() order and leaves the
input queue empty between chunks. Enqueuing the whole utterance up front instead trips
the separate progressive-before-final guard in should_process_input, which discards
the progressive chunks and allows the test to pass against unfixed code. Reverting
src/ while retaining the test produces 8 failures and 8 passes.

The optional backends are stubbed at import time so all four are covered on Linux CI,
where faster-whisper and lightning-whisper-mlx are not installed.

Out of scope

Exercising the transformers handler against real weights also surfaces an IndexError
at whisper_stt_handler.py:120, where pred_ids[0, 1] reads a text token rather than a
language tag under transformers 5.6.2. That behaviour is #375 and is already addressed
by #378, so it is left untouched here.

This change does not attempt to quantify the added latency of running STT twice per
utterance, which #412 raises. Measuring it requires the full LLM and TTS chain.

In --mode realtime the VAD emits progressive chunks while the user is still
speaking, then a final chunk once speech ends. BaseSTTHandler.before_emit_output
marks a turn revision complete as soon as it sees a Transcription, and
should_process_input then rejects every later chunk of that revision as
"input-after-final".

The four Whisper-family handlers never read vad_audio.mode and always yielded
Transcription, so the first progressive chunk closed the turn and the rest of the
utterance was discarded. A full sentence reached the LLM as its first fragment.
ParakeetTDTSTTHandler already branches on mode, which is why this only surfaced on
languages Parakeet does not cover.

Branch on the mode in whisper_stt_handler, faster_whisper_handler,
lightning_whisper_mlx_handler and mlx_audio_whisper_handler, yielding
PartialTranscription for progressive audio and leaving the final path untouched.

No enable_live_transcription flag is needed in these handlers. VAD only emits
progressive chunks when live transcription is on, since s2s_pipeline sets
enable_realtime_transcription from --enable_live_transcription and vad_handler
gates the progressive yield on it, so the mode check alone is sufficient.

The regression test drives the real BaseSTTHandler hooks in run() order and leaves
the input queue empty between chunks. Enqueuing the whole utterance up front trips
the separate progressive-before-final guard, which discards the progressive chunks
and lets the test pass against unfixed code.

Verified against real model weights on all four backends, driven by the real
VADHandler: whisper (openai/whisper-small), faster-whisper (small.en),
whisper-mlx (small) and mlx-audio-whisper (whisper-small-mlx). Each truncated a
4s utterance to "I think" before the change and delivered it in full after.

Fixes huggingface#412
@andimarafioti
andimarafioti self-requested a review August 12, 2026 09:18

@andimarafioti andimarafioti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This is an important fix. I haven't used whisper in a while, which is why I missed it. Codex found a few issues, but I want to merge this quickly, so I'll merge this to a branch before targetting main. I'll keep your commits in the history!

@andimarafioti
andimarafioti merged commit 0dfe107 into huggingface:main Aug 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Whisper-family STT handlers ignore mode, causing premature turn finalization mid-utterance in --mode realtime

2 participants