Blog post: When Does Constrained Decoding Actually Help a Small VLM?

[Arjun-Avadhanam]

A 4-cell ablation study testing how LoRA fine-tuning and Outlines constrained decoding interact on SmolVLM-256M for structured receipt extraction (SROIE dataset).

Three findings:

1. Constrained decoding is enormously valuable in the zero-shot regime (0% → 97% schema validity)
2. Once LoRA-trained, constrained decoding adds no measurable benefit (49.6% vs 49.8%)
3. The repetition_penalty hyperparameter that fixes degenerate loops in the untrained model causes silent failures in the trained model via an FSM-mask interaction

Repo: https://github.com/Arjun-Avadhanam/SmolVLM-CD
Gradio demo included with a rep_penalty slider to reproduce Finding 3 live.