torchvision.io to torchcodec - #5
Open
ariG23498 wants to merge 27 commits into
Open
Conversation
…uggingface#47940) Fix Gemma sliding_window halving on every config save/reload
![serge[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…t_mismatch` (tensor values differ (2)) (huggingface#47566) Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com> Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
) The Secret Leaks job fails on any push that CREATES a branch, and passes on every later push to it. The action derives its scan base from github.event.before, which is all-zeros for a new branch, so trufflehog walks the entire history instead of the pushed changes: new branch: scanning repo {"head": "..."} 415676 chunks, 230 MB later push: scanning repo {"base": "...", "head": ...} 11 chunks, 8 KB The 230 MB scan reports 397 "verified" secrets, all from Lob's detector, all false. That detector matches `(live|test)_<35 hex>` and verifies a hit by POSTing to api.lob.com/v1/us_verifications, where 422 is taken to mean "key is valid, body is bad" — the endpoint 422s on a bad body regardless of the key, so every candidate string is "verified" and the job exits 183. Pass an explicit base (falling back to the default branch when there is no before-SHA) so a new branch is scanned against main, and exclude the lob detector so a future full scan cannot resurrect the same 397 hits. Every push is now a diff scan, which also drops ~2 min of CI per new branch.
…(other (2)) (huggingface#47909) Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com>
* Declare sdpa support in TimmWrapper * Declare sdpa support in TimmBackbone
…ggingface#47968) [tests] Fix expected output for qwen2_5_vl batch_wo_image on CUDA Update the (None, None) expected string for `test_small_model_integration_test_batch_wo_image` to match actual model output on CUDA A10G: "a large language model" / "a wide range of tasks" instead of "an AI language model" / "various tasks". Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
huggingface#47960) Fix Jinja block endings in CHAT WITH MODELS' Writing a chat template docs
…her` (other (2)) (huggingface#47937) * [serge] Fix 2 integration tests for model `got_ocr2` failing with `other` (other (2)) * Add comment explaining root cause of got_ocr2 format test change The dtype change in 6217adc (Default auto, huggingface#42805) shifts model logits enough to flip the first generated token from \title{ to R\&D. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…ggingface#47944) * Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init Sample A_log in float32 (instead of the module's default dtype) when initializing Qwen3_5, Qwen3_5Moe, and Qwen3Next GatedDeltaNet modules. Under bfloat16, torch.empty(...).uniform_(0, 16) can round a value to 0, and log(0) = -inf, permanently zeroing that head's decay gradient for the rest of training. Adds a regression test for each of the three affected models asserting A_log stays float32 and finite when the model is constructed with dtype=torch.bfloat16. Fixes huggingface#47831 * Simplify A_log init fix: bump uniform_ lower bound instead of float32 cast Per review feedback, avoid the -inf-on-log(0) bug by sampling A from uniform_(0.01, 16) instead of casting to float32. Drops the added regression test and dtype casting in favor of a one-line comment.
…ggingface#47949) Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…uggingface#47921) * Fix _init_weights on CLIP models whose children carry quantized weights `CLIPPreTrainedModel._init_weights` initializes the parameters of child modules by name (`module.q_proj.weight`, `module.fc1.weight`, ...). A quantized checkpoint replaces `weight` on those children with packed tensors -- `weight_packed` for compressed-tensors, `qweight` for GPTQ/AWQ -- so loading one raises `AttributeError: 'Linear' object has no attribute 'weight'` from the init pass that `from_pretrained` runs for the keys that are missing from the checkpoint. Route every child-weight initialization through a helper that skips a child that has no `weight`, mirroring the guard the generic `PreTrainedModel._init_weights` already applies to `nn.Linear`. Initialization of a non-quantized model is unchanged. MetaCLIP-2 inlines this method through modular, so its modeling file is regenerated. * Update src/transformers/models/clip/modeling_clip.py * Update src/transformers/models/metaclip_2/modeling_metaclip_2.py * Apply ruff format to the edited docstring and regenerate MetaCLIP-2 --------- Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
…lutions test (huggingface#47972) Remove stale (None, None) fallback with garbage token in qwen2_5_vl batch_different_resolutions test The (None, None) fallback had ' addCriterion\n' prefixing the second batch item's expected output — a stale value from an older commit (c61ca64) that no longer matches any current hardware's output. Hardware-specific keys (cuda/rocm/xpu) all have clean expected values. Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…ggingface#37033 (huggingface#47948) * [emu3] Fix image generation broken since huggingface#37033 `Emu3ForConditionalGeneration.forward()` received `pixel_values` and `image_sizes` but never forwarded them to `self.model()`, so images were silently ignored and generation produced garbage output. Two fixes: - Pass `pixel_values` and `image_sizes` to `self.model()` in `Emu3ForConditionalGeneration.forward()` (modular + modeling files) - Cast `pixel_values` to the VQVAE's dtype in `Emu3VQVAE.encode()` to handle float16 inputs with float32 conv weights (PyTorch 2.13+ is stricter about dtype mismatches than earlier versions) Also update the expected integration test string to match current model output (minor generation difference due to PyTorch version). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [emu3] Add dtype cast to modular file to match modeling file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [emu3] Fix expected strings for test_model_generation_batched on A10G (cuda:8) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [emu3] Fix OOM in test_model_generation_multi_image and update expected strings for A10G - Reduce image processor resolution to 32x32 tokens (256x256 pixels) in test_model_generation_multi_image to avoid SDPA OOM on A10G (two 64x64 images produce ~8192 tokens, requiring 8+ GiB for attention alongside the 4-bit quantized model) - Update ("cuda", 8) expected strings for both test_model_generation_batched and test_model_generation_multi_image to match actual A10G outputs - Use 200x200 image downloads consistently across all integration tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [emu3] Expose vocabulary_mapping as property on Emu3ForConditionalGeneration After huggingface#37033 split the monolithic class into Emu3Model + head, vocabulary_mapping moved into Emu3Model. Add a property on Emu3ForConditionalGeneration to delegate to self.model.vocabulary_mapping, restoring the public API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [emu3] Add setUp/tearDown cleanup to Emu3IntegrationTest to free GPU memory between tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
* fix gemma4 video_features to device * align gemma4 unified with gemma4
* docs * fix * feedback
* [Gemma] Update expected values for A10G after PyTorch 2.9.0 upgrade
Update 7 failing Gemma integration tests to reflect output changes on
CUDA compute capability 8.x (A10G) after the PyTorch 2.8.0→2.9.0 upgrade
in the CI Docker image (`10d557123b`):
- test_model_2b_4bit: add ("cuda", 8) key with new brush output
- test_model_7b_fp16: add ("cuda", 8) key with "1995 4.0L 4x4" output
- test_model_7b_bf16: update ("cuda", 8) key[0] to "small game" output
- test_model_7b_fp16_static_cache: update ("cuda", 8) to "1999 4.0L 4x4"
- test_model_7b_4bit: update ("cuda", 8) with new quantization outputs
- test_compile_static_cache: add ("cuda", 8) key with "110bhp" variant
- test_export_static_cache: remove stale ("cuda", 8) key (A10G now matches (None, None))
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [Gemma] Fix test_model_7b_fp16_static_cache ("cuda", 8) expected value
Update second element to "DIY mini-f" which is the output produced on a
fresh CUDA context (as in CI). "DIY 3D" is produced on warm CUDA contexts
(repeated SSH runs) due to different memory allocation and SDPA kernel
selection on first vs subsequent process invocations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "[Gemma] Fix test_model_7b_fp16_static_cache ("cuda", 8) expected value"
This reverts commit cf17b84.
* [Gemma] Truncate flaky suffix in test_model_7b_fp16_static_cache
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [Gemma] Fix test_compile_static_cache: compile __call__ instead of forward
Compiling model.forward causes a CUDA graph tree RuntimeError on
PyTorch >= 2.9 when multiple generate() calls are made on the same
model instance. Using model.__call__ (matching the Mistral test pattern)
avoids the CUDA graph lifecycle conflict.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
[AXK1] Fix expected logits for CUDA A10G in test_model_logits_batched
The expected values for ("cuda", (8, 6)) were incorrect since the model
was added. Measured and updated on A10G (CUDA compute capability 8.6).
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…ned (OLMo garbage generation) (huggingface#47988) [GPTNeoX] Fix post_processor not being overridden when loading from pretrained `_from_pretrained` strips `add_bos_token`/`add_eos_token` from init_kwargs when a tokenizer.json is present, so `TokenizersBackend.__init__` sees no explicit bos/eos kwargs and skips calling `update_post_processor()`. For GPTNeoX (which rebuilds its Rust tokenizer from scratch rather than loading it directly from tokenizer.json), the post_processor from the JSON file is applied as-is and is never overridden — causing OLMo-7B to receive an unexpected EOS token appended to every input, producing garbage generation. Restore the explicit `update_post_processor()` call at the end of `GPTNeoXTokenizer.__init__` so the Rust post_processor always reflects the `add_bos_token`/`add_eos_token` settings. Regression introduced by huggingface#42563 (commit 73a13f8). Fixes OlmoIntegrationTest::test_model_7b_greedy_generation producing `.1.1.1.1.1...` garbage instead of coherent text. Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…#47986) * [OLMo] Fix OOM in logits tests by adding torch.no_grad() test_model_7b_logits and test_model_7b_twin_2t_logits call model() directly without torch.no_grad(), causing PyTorch to retain intermediate activations for potential backward passes. On A10G (22 GB), this extra memory pressure causes OOM when accelerate tries to move CPU-offloaded layers to GPU during the forward pass. Adding torch.no_grad() resolves the issue, consistent with how generate() handles inference internally. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Also fix test_model_1b_logits device mismatch and missing no_grad OLMo-1B fits entirely on GPU with device_map="auto", so the input tensor must be moved to model.device. Also wrap with torch.no_grad() and move output to CPU for comparison, consistent with the 7B tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…ingface#47989) * [OLMoE] Update expected logits for A10G and add torch.no_grad() Expected values in test_model_7b_logits were stale (3/8 mean elements mismatched). Updated to values captured on A10G. Also added torch.no_grad() consistent with other integration tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix OLMo2/OLMo3 slow test failures (OOM + stale logits) - Add `tearDownClass` to `Olmo3InternalIntegrationTest` and `Olmo3IntegrationTest` so each class's model is freed before the next class loads, preventing VRAM exhaustion (~14 GB per 7B model) - Add `with torch.no_grad():` to all forward-pass calls in OLMo2 and OLMo3 integration tests - Update stale expected logit values for OLMo2-1B and OLMo3-Internal (shanearora/2025-sep-a-base-model) to match current model outputs - Fix `Olmo3IntegrationTest::test_real_model_7b_greedy_generation_batched` expected value: batched generation with unequal-length prompts produces an empty response for the shorter (padded) item - Fix `test_generate_beyond_sliding_window` expected: model outputs ASCII apostrophe in "you've", not the curly U+2019 that was hardcoded Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [OLMo3] Fix batched generation test: use left padding + update expected output The tokenizer defaults to right padding, which causes the shorter padded sequence to generate EOS immediately after "assistant\n", producing an empty response. Setting padding_side="left" (required for decoder-only batched generation) produces proper output for both sequences. Update item[1] expected value accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…f 8-bit quantization (huggingface#47991) * [DeepSeekV2] Fix integration tests: use device_map=auto, remove 8-bit quantization The model (DeepSeek-V2-Lite) exceeds A10G VRAM even with 8-bit quantization. Switch to device_map="auto" (bfloat16, no quantization) which offloads excess weights to CPU. Update expected logit values accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [DeepSeekV2] Update expected values for device_map=auto bfloat16 loading Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [DeepSeekV2] Fix curly apostrophe in batch_fa2 expected text, add tearDown cleanup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [DeepSeekV2] Round expected logit values to 4 decimal places Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
… API changes) (huggingface#47995) * [Whisper] Fix float32/float16 dtype mismatch in integration tests whisper-large-v3 and distil-large-v3 are now saved as float16 on the Hub. Tests loaded the model without explicit dtype (loaded as float16) but cast input_features only to device (stayed float32), causing: RuntimeError: Input type (float) and bias type (c10::Half) should be the same Fix: use input_features.to(torch_device, dtype=model.dtype) in 5 tests: - test_large_generation - test_large_generation_multilingual - test_large_batched_generation - test_distil_token_timestamp_generation - test_large_timestamp_generation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Update test_large_timestamp_generation expected transcript batch_decode(output_offsets=True) now returns a plain dict instead of a list, and offsets is empty when return_timestamps was not set at generation time. Update EXPECTED_TRANSCRIPT to match current output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Update test_whisper_shortform_single_batch_prev_cond expected output CUDA A10G now produces comma-separated transcription ("leather seating, so soft", "folks, I lurched") vs previous ("seating so soft", "folks. I lurched"). Update cuda_expectation and cuda_expectation2 to match current output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Fix remaining integration test failures (value mismatches, API changes, static cache) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Clarify static cache reset comment in test_tiny_static_generation_long_form Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Restore truncated eager vs static comparison in test_tiny_static_generation_long_form torch.compile introduces tiny fp differences that diverge after ~85-138 generated tokens; compare only the first 40 generated tokens ([:41]) for basic correctness, and keep the two-static-run comparison to verify cache-reset correctness. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Add PR reference to static generation test comment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Remove unnecessary min_len in static cache reorder assertion Both static runs produce equal-length outputs so min_len is not needed; use a direct full-sequence comparison instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * format --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…() (huggingface#47997) [Whisper] Fix batch decode_with_timestamps handling in WhisperTokenizer.decode() PR huggingface#40936 (05c0e1d) made batch_decode delegate to decode() natively, passing a 2D list of token ids. The decode_with_timestamps branch was not updated to handle batched input, causing TypeError when >= compared a list to an int. Add the same isinstance check already present in the else branch. Fixes test_whisper_longform_single_batch (first failing in CI on 2025-11-28, introduced by PR huggingface#40936 on 2025-11-27). Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
* mini model original * mini model and modular * update modular * update moe logic to match standard * updates debug scripts * original code, debug scripts, mini model, etc. * fix sparseblockmoe and decoderlayer to match standards, by converting config * slim down moe modular, fix config conversion * rm Step3p7CausalLMOutputWithPast, unused functions, return BaseModelOutputWithPast * standard renaming * tests and siglip vision encoder * build mini checkpoint in right format + conversion script * new demo and compare outputs workflow * major cut of modular, no image_embeds, etc. * fix tests * remove debug files * address feedback - add processor and config to modular * apply more feedback, reduce image processing * apply round 2 of feedback! * more feedback and weight converter * feedback * gemma3attention and rm SharedExperts class * Untrack local debug scripts * Untrack local debug scripts p2 * Untrack local debug scripts p3 * Revert core model loading changes * standard rope * clean up * add tok * tests integration * docs and tests unskip * tests * laguana parents * fix step3p5_vision conversion after rebase onto main VisionUnfuseAndPermuteForRope was deprecated upstream in favor of composing Chunk()+PermuteForRope() directly; migrate the two step3p5_vision entries accordingly. Also regenerate modeling_step3p7.py for main's updated RotaryEmbedding converter template. * add checkpoint links to step3p7 config docstrings check_config_docstrings.py requires every registered config class to link a real checkpoint via @auto_docstring(checkpoint=...); add the stepfun-ai/Step-3.7-Flash link to all three step3p7 config classes. * allowlist step3p7's unused legacy config attributes, add contribution date check_config_attributes.py flags config fields never read in the modeling file; add Step3p7TextConfig/Step3p7VisionConfig entries for the legacy hub-config kwargs and inherited MoE-aux-loss fields this architecture doesn't use (each already documented as such in the config docstrings). Also add the missing contribution date to step3p7.md via utils/add_dates.py. * more feedback / cleanup! * modular conversion missed * modualr * mini checkpoint * feedback * modeling from converter * revert core model loading changes from rebase * config attrs * date md * Drop redundant num_heads arg from Step3p7Attention Step3p7Attention.__init__ took config and num_heads separately even though the config passed in already carries num_attention_heads (mlinter TRF029). Derive it internally instead. * add date * rebase and date --------- Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-171-249.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-175-19.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-161-178.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-163-220.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-163-236.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-168-95.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-175-132.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-174-196.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-169-247.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-165-59.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-165-131.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-163-58.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-175-241.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-172-73.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-160-192.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-175-165.ec2.internal> Co-authored-by: ita.zaporozhets@huggingface.co <ita_zaporozhets@ip-26-0-166-68.ec2.internal>
…on, and speed regression (huggingface#48000) * [Whisper] Fix cache type and layer count for speculative decoding with WhisperForCausalLM Fix 1 — is_updated UnboundLocalError (WhisperAttention.forward) PR huggingface#39956 wrapped the `is_updated` assignment in an isinstance guard but left it consumed unconditionally below. The bug was dormant while Whisper always received EncoderDecoderCache. PR huggingface#43679 changed generate() to pre-initialize a plain DynamicCache for is_encoder_decoder=False models, activating the latent bug. Fix: initialize is_updated=False before the guard, identical to PR huggingface#40517 which fixed 55 other models. Fix 2a — Wrong cache type: DynamicCache instead of EncoderDecoderCache (WhisperDecoder.forward) generate() uses config.is_encoder_decoder to pick the cache type. WhisperForCausalLM sets is_encoder_decoder=False, so PR huggingface#43679 made generate() pre-initialize a plain DynamicCache. But WhisperForCausalLM uses cross-attention: self- and cross-attention share the same layer_idx slots in DynamicCache and overwrite each other. Fix: when generate() passes an empty DynamicCache and encoder_hidden_states are present, convert it to EncoderDecoderCache before the first forward pass. Fix 2b — Wrong layer count: 32 slots pre-allocated for a 2-layer decoder (WhisperDecoder.forward) DynamicCache(config=self.config) calls config.get_text_config(decoder=True), which for WhisperConfig returns self unchanged (flat config, no nested decoder sub-config). It then reads config.num_hidden_layers, which WhisperConfig maps via attribute_map to encoder_layers=32. For distil-whisper-large-v2, encoder_layers=32 but decoder_layers=2, so 32 cache slots are pre-allocated for a 2-layer decoder. Slots 2-31 are uninitialized and crash on crop(-n). Fix: deepcopy the config and override num_hidden_layers with decoder_layers before passing to DynamicCache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Restore absolute cache crop in _update_past_and_masks for speculative decoding Fix 3 — Cache size accumulation (candidate_generator.py) Commit 5ec7b73 ("crop with negative always") changed _update_past_and_masks to always remove a fixed number of tokens (crop(-1)) instead of cropping to an absolute target. When multiple speculative candidates are rejected in a round, the cache grows by several tokens but only 1 is trimmed. Over successive rounds, cache.get_seq_length() exceeds input_ids.shape[1]. _prefill then computes next_sequence_length = input_ids.shape[1] - cache.get_seq_length() which goes negative, causing input_ids[:, -n:] to produce a [1, 0] empty tensor that crashes in WhisperAttention.forward() at the reshape. Verified by bisect: at the parent of 5ec7b73 (68075f8), fixes 1+2a+2b alone are sufficient (speed assertion only). At 5ec7b73 with only fixes 1+2a+2b the reshape crash reproduces. Adding this fix restores the test to speed-assertion-only. Fix: compute the absolute target size from input_ids and only crop if the cache exceeds it, using a negative delta to comply with the current Cache.crop(negative) API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Generate] Sync adaptive confidence threshold to generation_config for ConfidenceCriteria PR huggingface#42702 introduced `self.assistant_generation_config` (a deepcopy with defaults applied) to hold the assistant model's config, but the adaptive threshold update (`update_candidate_strategy`) was only syncing the new best threshold to `self.assistant_generation_config.assistant_confidence_threshold`, not to `self.generation_config.assistant_confidence_threshold`. `ConfidenceCriteria` is constructed each round from `self.generation_config`, so it was always using the initial 0.4 threshold regardless of what the ROC-curve adaptation computed. This caused the assistant to always generate the maximum number of speculative tokens per round, making assisted generation slower than non-assisted on the distil-whisper test. Fix: also update `self.generation_config.assistant_confidence_threshold` whenever the adaptive threshold is updated, so each new round's ConfidenceCriteria reflects the current best threshold. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Whisper] Format: wrap long EncoderDecoderCache line Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AudioFlamingo3] Fix is_updated UnboundLocalError in cross-attention (sync with modular) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
* Allow tokenizers 0.23.1 Closes huggingface#45736 * Fix RobertaProcessing kwargs for tokenizers 0.23.1 tokenizers 0.23.1 made RobertaProcessing args positional-only. Convert the 5 call sites (convert_slow_tokenizer x2, clip, layoutlmv3, roberta) to positional (sep, cls, trim_offsets, add_prefix_space) — works on 0.22.x and 0.23.1. * Tolerate dropped overflow segment in max-encoding-length tests tokenizers 0.23.1 early-exits right-direction truncation (huggingface/tokenizers#1990), so return_overflowing_tokens may omit the trailing overflow segment when it is shorter than the window. Validate the overflow content only when that segment is present; truncation is always checked. Keeps passing on 0.22.x (segment always present). * Drop overflow-segment assertions in max-encoding-length tests The trailing overflow segment is not essential to validate here, and 0.23.1's truncation early-exit (huggingface/tokenizers#1990) may omit it. Keep the truncation checks; remove the now-dead overflow expectations. * Apply suggestion from @ArthurZucker * Fix HerbertTokenizer: pass BertProcessing sep/cls positionally tokenizers.processors.BertProcessing.__new__ does not accept 'sep' and 'cls' keyword arguments, raising TypeError on tokenizer init. Pass the (token, id) tuples positionally instead. * Update dependency versions table for tokenizers 0.23.1 pin * Remove accidentally committed test_opt tokenizer artifacts * Fix ty type errors in tokenization files Run with tokenizers 0.23.1 (new type stubs). Fixes: - add missing '| None' to params defaulting to None (truncation, udop text/text_target) across layoutlmv2/v3, layoutxlm, luke, markuplm, mluke, roc_bert, pop2piano, udop (invalid-parameter-default) - pop2piano: offset_sec annotated int but defaults to 0.0 -> float - bertweet: import html.entities (submodule not pulled in by 'import html') - tapas: tuple[float, Ellipsis] -> tuple[float, ...] (invalid-type-form) * Pass BertProcessing/RobertaProcessing args positionally in call sites merged from main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.