Performance differences on visual grounding

[Chongyu-hub]

Hi, in my reproduction, the Scan2Cap results are lower(CIDER = 76.3) than those reported in your paper (CIDER = 78.6). I finetuned Qwen2.5-VL-3B-Instruct, but the training loss seems to be higher than what you mentioned. Do you have any suggestions or recommendations?
my config:

#!/bin/bash
# Complete QwenVL Training Launch Script with Full Parameter Documentation

# ======================
# Distributed Configuration
# ======================
MASTER_ADDR="127.0.0.1"                     # [Required] Master node IP for multi-GPU training
MASTER_PORT=$(shuf -i 20000-29999 -n 1)     # Random port to avoid conflicts
NPROC_PER_NODE=8                            # Use first 7 GPUs (GPU 0-6)
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7   # Limit visible GPUs to first 7 cards

# ======================
# Path Configuration
# ======================
MODEL_PATH="ckpt/Qwen2.5-VL-3B-Instruct"  # [ModelArguments] Pretrained model path
GEOMETRY_ENCODER_TYPE="vggt"
GEOMETRY_ENCODER_PATH="ckpt/VGGT-1B"
OUTPUT_DIR="output"                   # Directory for saving checkpoints
CACHE_DIR="./cache"                        # [TrainingArguments] Cache directory for models
mkdir -p $OUTPUT_DIR
DATE=$(TZ="Asia/Shanghai" date "+%Y%m%d")
# ======================
# Model Configuration
# ======================
DATASETS="scan2cap,scanrefer,scannet_det"                  # [DataArguments] Dataset with sampling rate

# ======================
# Training Hyperparameters
# ======================
export NCCL_NVLS_ENABLE=0
LR=1e-5
total_batch_size=64
GRADIENT_ACCUMULATION_STEPS=$(($total_batch_size / $NPROC_PER_NODE))

torchrun --nproc_per_node=$NPROC_PER_NODE \
            --master_addr=$MASTER_ADDR \
            --master_port=$MASTER_PORT \
            src/qwen_vl/train/train_qwen.py \
            --model_name_or_path $MODEL_PATH \
            --tune_mm_llm True \
            --tune_mm_vision False \
            --tune_mm_mlp False \
            --dataset_use $DATASETS \
            --output_dir $OUTPUT_DIR \
            --cache_dir $CACHE_DIR \
            --bf16 \
            --per_device_train_batch_size 1 \
            --gradient_accumulation_steps $GRADIENT_ACCUMULATION_STEPS \
            --learning_rate $LR \
            --mm_projector_lr 1e-5 \
            --vision_tower_lr 1e-6 \
            --optim adamw_torch \
            --model_max_length 12800 \
            --data_flatten False \
            --max_pixels $((576*28*28)) \
            --min_pixels $((16*28*28)) \
            --base_interval 2 \
            --video_max_frames 8 \
            --video_min_frames 4 \
            --video_max_frame_pixels $((1664*28*28)) \
            --video_min_frame_pixels $((256*28*28)) \
            --num_train_epochs 1 \
            --warmup_ratio 0.03 \
            --lr_scheduler_type "cosine" \
            --weight_decay 0.01 \
            --logging_steps 10 \
            --save_steps 1000 \
            --save_total_limit 1 \
            --deepspeed "scripts/zero2_opt.json" \
            --gradient_checkpointing \
            --dataloader_num_workers 4 \
            --group_by_modality_length true \
            --seed 0 \
            --report_to "none" \
            --use_geometry_encoder True \
            --geometry_encoder_type $GEOMETRY_ENCODER_TYPE \
            --geometry_encoder_path $GEOMETRY_ENCODER_PATH \
            --feature_fusion_method "add" \
            > ${OUTPUT_DIR}/train_${DATE}.log 2>&1

my loss:

{'train_runtime': 44470.1427, 'train_samples_per_second': 5.523, 'train_steps_per_second': 0.086, 'train_loss': 3.21557123374591, 'epoch': 1.0}

[zd11024]

This is because our codebase has undergone several updates, leading to some discrepancies between the current results and those reported in the paper. For the 3B model, increasing the learning rate from 1e-5 to 2e-5 yields better performance, while for the 7B model, using lr=1e-5 produces superior results.

[Chongyu-hub]

This is because our codebase has undergone several updates, leading to some discrepancies between the current results and those reported in the paper. For the 3B model, increasing the learning rate from 1e-5 to 2e-5 yields better performance, while for the 7B model, using lr=1e-5 produces superior results.

Thanks a lot ! I will try it