[Ernie 4.5] Ernie VL models

docs/source/en/_toctree.yml

@@ -1050,6 +1050,8 @@
         title: EdgeTamVideo
       - local: model_doc/emu3
         title: Emu3
+      - local: model_doc/ernie4_5_vl
+        title: ernie4_5_vl
       - local: model_doc/evolla
         title: Evolla
       - local: model_doc/flava

docs/source/en/model_doc/ernie4_5_vl.md

@@ -0,0 +1,97 @@
+<!--Copyright 2025 The Qwen Team and The HuggingFace Inc. team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+<img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
+<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">    </div>
+</div>
+
+# ernie4_5_vl
+
+## Overview
+
+The ernie4_5_vl model was proposed in [<INSERT PAPER NAME HERE>](<INSERT PAPER LINK HERE>) by <INSERT AUTHORS HERE>.
+<INSERT SHORT SUMMARY HERE>
+
+The abstract from the paper is the following:
+
+In this report, we propose PaddleOCR-VL, a SOTA and resource-efficient model tailored for document parsing. Its core component is PaddleOCR-VL-0.9B, a compact yet powerful vision-language model (VLM) that integrates a NaViT-style dynamic resolution visual encoder with the ERNIE-4.5-0.3B language model to enable accurate element recognition. This innovative model efficiently supports 109 languages and excels in recognizing complex elements (e.g., text, tables, formulas, and charts), while maintaining minimal resource consumption. Through comprehensive evaluations on widely used public benchmarks and in-house benchmarks, PaddleOCR-VL achieves SOTA performance in both page-level document parsing and element-level recognition. It significantly outperforms existing solutions, exhibits strong competitiveness against top-tier VLMs, and delivers fast inference speeds. These strengths make it highly suitable for practical deployment in real-world scenarios.
+
+Tips:
+
+<INSERT TIPS ABOUT MODEL HERE>
+
+This model was contributed by [INSERT YOUR HF USERNAME HERE](https://huggingface.co/<INSERT YOUR HF USERNAME HERE>).
+The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>).
+
+
+## Ernie4_5_VLConfig
+
+[[autodoc]] Ernie4_5_VLConfig
+
+## Ernie4_5_VLTextConfig
+
+[[autodoc]] Ernie4_5_VLTextConfig
+
+## Ernie4_5_VLVisionConfig
+
+[[autodoc]] Ernie4_5_VLVisionConfig
+
+## Ernie4_5_VLImageProcessor
+
+[[autodoc]] Ernie4_5_VLImageProcessor
+    - preprocess
+
+## Ernie4_5_VLImageProcessorFast
+
+[[autodoc]] Ernie4_5_VLImageProcessorFast
+    - preprocess
+
+## Ernie4_5_VLVideoProcessor
+
+[[autodoc]] Ernie4_5_VLVideoProcessor
+    - preprocess
+
+## Ernie4_5_VLProcessor
+
+[[autodoc]] Ernie4_5_VLProcessor
+
+## Ernie4_5_VLTextModel
+
+[[autodoc]] Ernie4_5_VLTextModel
+    - forward
+
+## Ernie4_5_VLVisionTransformerPretrainedModel
+
+[[autodoc]] Ernie4_5_VLVisionTransformerPretrainedModel
+    - forward
+
+## Ernie4_5_VLVariableResolutionResamplerModel
+
+[[autodoc]] Ernie4_5_VLVariableResolutionResamplerModel
+    - forward
+
+## Ernie4_5_VLModel
+
+[[autodoc]] Ernie4_5_VLModel
+    - forward
+
+## Ernie4_5_VLForConditionalGeneration
+
+[[autodoc]] Ernie4_5_VLForConditionalGeneration
+    - forward

src/transformers/modeling_rope_utils.py

@@ -608,6 +608,66 @@ def _compute_llama3_parameters(
     return inv_freq_llama, attention_factor
 
 
+def _compute_ernie_3d_parameters(
+    config: PreTrainedConfig,
+    device: "torch.device",
+    seq_len: Optional[int] = None,
+    layer_type: Optional[str] = None,
+) -> tuple["torch.Tensor", float]:
+    """
+    Computes the inverse frequencies for the Ernie 4.5 VL models.
+
+    Args:
+        config ([`~transformers.PreTrainedConfig`]):
+            The model configuration. This function assumes that the config will provide at least the following
+            properties:
+            *   rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived.
+            *   hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly.
+            *   num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly.
+            *   rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following
+                keys will be accessed:
+                *   `freq_allocation` (`int`): The scaling factor applied to the inverse frequencies when 1) the
+                    wavelength is greater than `low_freq_wavelen` prior to smoothing, and 2) to all inverse frequencies
+                    during smoothing.
+            Additionally, this function will make use of the following properties if they are found in the config:
+            *   head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be
+                derived as hidden_size // num_attention_heads.
+            *   partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for
+                the first fraction of the head_dim. Defaults to 1.0.
+        device (`torch.device`):
+            The device to use for initialization of the inverse frequencies.
+        seq_len (`int`, *optional*):
+            The current sequence length. Unused for this type of RoPE.
+    Returns:
+        Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the
+        post-processing scaling factor applied to the computed cos/sin.
+    """
+    # For backward compatibility standardize the `rope_parameters_dict` if it uses old format
+    standardize_rope_params(config)
+    rope_parameters_dict = config.rope_parameters[layer_type] if layer_type is not None else config.rope_parameters
+
+    base = rope_parameters_dict["rope_theta"]
+    partial_rotary_factor = getattr(config, "partial_rotary_factor", 1.0)
+    head_dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
+    dim = int(head_dim * partial_rotary_factor)
+    attention_factor = 1.0  # Unused in this type of RoPE
+
+    # Compute the inverse frequencies
+    inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.int64).to(device=device, dtype=torch.float) / dim))
+
+    # Divide frequency allocation based on `freq_allocation`
+    # and apply necessary (pre-)rotations
+    t_dim = rope_parameters_dict["freq_allocation"]  # time dimension
+    hw_dim = inv_freq.shape[-1] - t_dim  # height and width dimension
+
+    inv_freq_3d = torch.empty_like(inv_freq)
+    # (Pre-)Rotate to avoid another rotation during the forward
+    inv_freq_3d[:hw_dim] = torch.cat([inv_freq[:-t_dim][0::2], inv_freq[:-t_dim][1::2]])
+    inv_freq_3d[-t_dim:] = inv_freq[-t_dim:]
+
+    return inv_freq_3d, attention_factor
+
+
 # This maps the "rope_type" string field in rope config to the corresponding function to compute the RoPE parameters
 # from the model config. You can append new {'rope_type': callable} pairs to this rope_parameters to enable custom RoPE
 # parameterizations, as long as the callable has the same signature.
@@ -617,6 +677,7 @@ def _compute_llama3_parameters(
     "yarn": _compute_yarn_parameters,
     "longrope": _compute_longrope_parameters,
     "llama3": _compute_llama3_parameters,
+    "ernie_3d": _compute_ernie_3d_parameters,
 }
 
 
@@ -843,6 +904,28 @@ def _validate_llama3_parameters(rope_parameters: dict, config: PreTrainedConfig,
         )
 
 
+def _validate_ernie_3d_parameters(rope_parameters: dict, config: PreTrainedConfig, ignore_keys: Optional[set] = None):
+    required_keys = {"rope_type", "rope_theta", "freq_allocation"}
+    rope_type = rope_parameters["rope_type"]
+    received_keys = set(rope_parameters.keys())
+    _check_received_keys(rope_type, received_keys, required_keys, ignore_keys=ignore_keys)
+
+    partial_rotary_factor = config.partial_rotary_factor if hasattr(config, "partial_rotary_factor") else 1.0
+    head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+    dim = int(head_dim * partial_rotary_factor)
+
+    freq_allocation = rope_parameters["freq_allocation"]
+    t_dim = freq_allocation
+    h_dim = (dim - t_dim) // 2
+    reconstructed_dim = t_dim + 2 * h_dim
+    if freq_allocation is None or not isinstance(freq_allocation, int) or reconstructed_dim != dim:
+        logger.warning(
+            "`rope_parameters`'s freq_allocation field must be an int that can evenly be split into three dimensions: "
+            f"`freq_allocation` and 2 * (dim - freq_allocation). However, we found the following splits {t_dim}, {h_dim}, {h_dim};"
+            f"this does not split evenly into the total dim of {dim} vs. {reconstructed_dim}."
+        )
+
+
 # Like `ROPE_INIT_FUNCTIONS`, this validation function mapping can be dynamically updated for custom RoPE types.
 ROPE_VALIDATION_FUNCTIONS = {
     "default": _validate_default_rope_parameters,
@@ -851,6 +934,7 @@ def _validate_llama3_parameters(rope_parameters: dict, config: PreTrainedConfig,
     "yarn": _validate_yarn_parameters,
     "longrope": _validate_longrope_parameters,
     "llama3": _validate_llama3_parameters,
+    "ernie_3d": _validate_ernie_3d_parameters,
 }
 
 
@@ -892,7 +976,7 @@ class RopeParameters(TypedDict):
             The base period of the RoPE embeddings.
         rope_type (`str`, *optional*, defaults to "default"):
             The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
-            'llama3'], with 'default' being the original RoPE implementation.
+            'llama3', 'ernie_3d'], with 'default' being the original RoPE implementation.
         factor (`float`, *optional*):
             Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
             most scaling types, a `factor` of x will enable the model to handle sequences of length x *
@@ -922,6 +1006,8 @@ class RopeParameters(TypedDict):
             Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
         high_freq_factor (`float`, *optional*):
             Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        freq_allocation (`int`, *optional*):
+            Only used with 'ernie 4.5 vl'. Absolute allocation of the head dim to the time dim in 3D RoPE
     """
 
     rope_theta: float
@@ -935,3 +1021,4 @@ class RopeParameters(TypedDict):
     long_factor: Optional[list[float]]
     low_freq_factor: Optional[float]
     high_freq_factor: Optional[float]
+    freq_allocation: Optional[int]

src/transformers/models/auto/configuration_auto.py

@@ -141,6 +141,7 @@
         ("ernie", "ErnieConfig"),
         ("ernie4_5", "Ernie4_5Config"),
         ("ernie4_5_moe", "Ernie4_5_MoeConfig"),
+        ("ernie4_5_vl", "Ernie4_5_VLConfig"),
         ("ernie_m", "ErnieMConfig"),
         ("esm", "EsmConfig"),
         ("evolla", "EvollaConfig"),
@@ -583,6 +584,7 @@
         ("ernie", "ERNIE"),
         ("ernie4_5", "Ernie4_5"),
         ("ernie4_5_moe", "Ernie4_5_MoE"),
+        ("ernie4_5_vl", "Ernie4_5_VL"),
         ("ernie_m", "ErnieM"),
         ("esm", "ESM"),
         ("evolla", "Evolla"),

src/transformers/models/auto/image_processing_auto.py

@@ -96,6 +96,7 @@
             ("efficientloftr", ("EfficientLoFTRImageProcessor", "EfficientLoFTRImageProcessorFast")),
             ("efficientnet", ("EfficientNetImageProcessor", "EfficientNetImageProcessorFast")),
             ("eomt", ("EomtImageProcessor", "EomtImageProcessorFast")),
+            ("ernie4_5_vl", ("Ernie4_5_VLImageProcessor", "Ernie4_5_VLImageProcessorFast")),
             ("flava", ("FlavaImageProcessor", "FlavaImageProcessorFast")),
             ("focalnet", ("BitImageProcessor", "BitImageProcessorFast")),
             ("fuyu", ("FuyuImageProcessor", None)),

src/transformers/models/auto/modeling_auto.py

@@ -144,6 +144,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("ernie", "ErnieModel"),
         ("ernie4_5", "Ernie4_5Model"),
         ("ernie4_5_moe", "Ernie4_5_MoeModel"),
+        ("ernie4_5_vl", "Ernie4_5_VLModel"),
         ("ernie_m", "ErnieMModel"),
         ("esm", "EsmModel"),
         ("evolla", "EvollaModel"),
@@ -1023,6 +1024,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("deepseek_vl", "DeepseekVLForConditionalGeneration"),
         ("deepseek_vl_hybrid", "DeepseekVLHybridForConditionalGeneration"),
         ("emu3", "Emu3ForConditionalGeneration"),
+        ("ernie4_5_vl", "Ernie4_5_VLForConditionalGeneration"),
         ("evolla", "EvollaForProteinText2Text"),
         ("florence2", "Florence2ForConditionalGeneration"),
         ("fuyu", "FuyuForCausalLM"),

src/transformers/models/auto/processing_auto.py

@@ -67,6 +67,7 @@
         ("dia", "DiaProcessor"),
         ("edgetam", "Sam2Processor"),
         ("emu3", "Emu3Processor"),
+        ("ernie4_5_vl", "Ernie4_5_VLProcessor"),
         ("evolla", "EvollaProcessor"),
         ("flava", "FlavaProcessor"),
         ("florence2", "Florence2Processor"),

src/transformers/models/auto/tokenization_auto.py

@@ -236,6 +236,7 @@
         ("ernie", ("BertTokenizer", "BertTokenizerFast" if is_tokenizers_available() else None)),
         ("ernie4_5", (None, "LlamaTokenizerFast" if is_tokenizers_available() else None)),
         ("ernie4_5_moe", (None, "LlamaTokenizerFast" if is_tokenizers_available() else None)),
+        ("ernie4_5_vl", (None, "LlamaTokenizerFast" if is_tokenizers_available() else None)),
         ("ernie_m", ("ErnieMTokenizer" if is_sentencepiece_available() else None, None)),
         ("esm", ("EsmTokenizer", None)),
         (

src/transformers/models/auto/video_processing_auto.py

@@ -45,6 +45,7 @@
 else:
     VIDEO_PROCESSOR_MAPPING_NAMES = OrderedDict(
         [
+            ("ernie4_5_vl", "Ernie4_5_VLVideoProcessor"),
             ("glm4v", "Glm4vVideoProcessor"),
             ("instructblip", "InstructBlipVideoVideoProcessor"),
             ("instructblipvideo", "InstructBlipVideoVideoProcessor"),

src/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py

@@ -316,11 +316,12 @@ def forward(self, hidden_states):
 
 
 class Ernie4_5_MoeExperts(nn.ModuleList):
-    def __init__(self, config):
+    def __init__(self, config, intermediate_size=None):
         super().__init__()
         self.num_experts = config.moe_num_experts
+        intermediate_size = config.moe_intermediate_size if intermediate_size is None else intermediate_size
         for _ in range(self.num_experts):
-            self.append(Ernie4_5_MoeMLP(config, config.moe_intermediate_size))
+            self.append(Ernie4_5_MoeMLP(config, intermediate_size))
 
     def forward(
         self, hidden_states: torch.Tensor, selected_experts: torch.Tensor, routing_weights: torch.Tensor
@@ -368,8 +369,8 @@ def route_tokens_to_experts(self, hidden_states):
             routing_weights = routing_weights / torch.clamp(
                 routing_weights.sum(dim=-1, keepdim=True), min=self.norm_min
             )
-        routing_weights = routing_weights.to(router_logits.dtype)
-        return selected_experts, routing_weights
+        routing_weights = routing_weights.to(hidden_states.dtype)
+        return selected_experts, routing_weights, router_logits
 
     def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
         batch_size, sequence_length, _ = hidden_states.shape
@@ -378,7 +379,7 @@ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
         if self.shared_experts is not None:
             shared_output = self.shared_experts(hidden_states)
 
-        selected_experts, routing_weights = self.route_tokens_to_experts(hidden_states)
+        selected_experts, routing_weights, _ = self.route_tokens_to_experts(hidden_states)
         final_hidden_states = self.experts(hidden_states, selected_experts, routing_weights)
 
         if self.shared_experts is not None:

src/transformers/models/ernie4_5_moe/modular_ernie4_5_moe.py

@@ -97,11 +97,12 @@ def forward(self, hidden_states):
 
 
 class Ernie4_5_MoeExperts(nn.ModuleList):
-    def __init__(self, config):
+    def __init__(self, config, intermediate_size=None):
         super().__init__()
         self.num_experts = config.moe_num_experts
+        intermediate_size = config.moe_intermediate_size if intermediate_size is None else intermediate_size
         for _ in range(self.num_experts):
-            self.append(Ernie4_5_MoeMLP(config, config.moe_intermediate_size))
+            self.append(Ernie4_5_MoeMLP(config, intermediate_size))
 
     def forward(
         self, hidden_states: torch.Tensor, selected_experts: torch.Tensor, routing_weights: torch.Tensor
@@ -149,8 +150,8 @@ def route_tokens_to_experts(self, hidden_states):
             routing_weights = routing_weights / torch.clamp(
                 routing_weights.sum(dim=-1, keepdim=True), min=self.norm_min
             )
-        routing_weights = routing_weights.to(router_logits.dtype)
-        return selected_experts, routing_weights
+        routing_weights = routing_weights.to(hidden_states.dtype)
+        return selected_experts, routing_weights, router_logits
 
     def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
         batch_size, sequence_length, _ = hidden_states.shape
@@ -159,7 +160,7 @@ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
         if self.shared_experts is not None:
             shared_output = self.shared_experts(hidden_states)
 
-        selected_experts, routing_weights = self.route_tokens_to_experts(hidden_states)
+        selected_experts, routing_weights, _ = self.route_tokens_to_experts(hidden_states)
         final_hidden_states = self.experts(hidden_states, selected_experts, routing_weights)
 
         if self.shared_experts is not None: