[Bug] Whisper Base Small model with accuracy issue and service not stable issue

[MindyCMM]

Describe the issue

One ISV team is developing local AI features based on FL Whisper model, they want to know whether we can help them on the fixing and improvement.
Model : Whisper Base /Small

1. We often observe word segmentation issues in the transcription results, where unnecessary spaces are inserted. For example, "2006年" may be transcribed as "200 6年". Do you have any recommended optimization approaches for improving tokenization and text formatting accuracy?
2. During testing, we occasionally encounter a situation where the model service becomes unresponsive (typically after repeatedly starting and stopping the feature). Once this occurs, the service can no longer be accessed successfully until the PC is rebooted.
   o Is there an API available to forcibly restart the LLM service?
   o Is there a recommended minimum interval between service startup and shutdown operations to avoid this issue?

To reproduce

1.We often observe word segmentation issues in the transcription results, where unnecessary spaces are inserted. For example, "2006年" may be transcribed as "200 6年". Do you have any recommended optimization approaches for improving tokenization and text formatting accuracy?
2. During testing, we occasionally encounter a situation where the model service becomes unresponsive (typically after repeatedly starting and stopping the feature). Once this occurs, the service can no longer be accessed successfully until the PC is rebooted.

Urgency

No response

Platform and architecture

Windows X64

OS Version

Windows 11 25H2 x64 - 26200.9168

Installation type

Released package/binary

Foundry Local version

SDK1.2.3

API or surface area

C# SDK

Hardware acceleration/backend

CPU

Backend/runtime version

No response

[Wayne-Ch]

Thanks for reporting these issues MindyCMM.

Transcription accuracy

Since you are transcribing Chinese audio, please try whisper-large-v3-turbo with the same input to establish the upper limit of Whisper quality. Base and Small models are more prone to hallucinations and other quality limitations. Comparing the results will help us determine whether the observed behavior is a model-capacity limitation or an issue requiring further investigation.

The current Whisper models can process up to 30 seconds of audio per request. Audio longer than 30 seconds must be segmented before transcription.

I can also test your workload with the Nemotron ASR model if you can share the testing script. Alternatively, you can test it directly—the Nemotron CPU model is available in the Foundry Local catalog, so the comparison should be straightforward. Foundry Local currently provides both CPU and CUDA catalog variants for Whisper and Nemotron.

Service unresponsiveness

To investigate the service becoming unresponsive after repeated start/stop operations, could you share:

• A minimal reproduction or your testing script
• The exact start/stop sequence, timing, and approximate number of cycles before the issue occurs
• Relevant SDK or service logs captured around the failure
• Whether terminating and restarting the process recovers the service without rebooting the PC

This information will help us reproduce and investigate the stability issue without assuming a specific root cause.