Skip to content

[GenEval] feat: add multi-gpu in evaluations & tqdm display the progress#22

Open
lxazjk wants to merge 2 commits into
djghosh13:mainfrom
lxazjk:feat/multiprocess
Open

[GenEval] feat: add multi-gpu in evaluations & tqdm display the progress#22
lxazjk wants to merge 2 commits into
djghosh13:mainfrom
lxazjk:feat/multiprocess

Conversation

@lxazjk

@lxazjk lxazjk commented Mar 17, 2026

Copy link
Copy Markdown

Summary

geneval/evaluation/evaluate_images.py previously evaluated images sequentially on a single GPU, which is slow for large-scale runs and provides limited visibility into progress.
This PR adds a simple, low-intrusion multi-GPU sharding mode to improve throughput and adds progress bars for better UX.

What’s changed

Multi-GPU evaluation via multiprocessing sharding

Collects all (image_path, metadata) tasks first, then splits them across GPUs and runs one process per GPU.
Each worker sets its CUDA device, loads models locally, evaluates its shard, and returns results to the main process for aggregation.

Progress reporting with tqdm Single-GPU mode shows a single overall progress bar.

Multi-GPU mode shows per-GPU worker progress bars.
Stability improvements for spawn + CLIP color classification
Refactors the crop dataset used by CLIP color classification to avoid relying on global variables in DataLoader worker processes.
Adds clip_num_workers option and defaults to 0 when num_gpus > 1 to avoid nested multiprocessing issues.

How to run

Single GPU

python geneval/evaluation/evaluate_images.py <imagedir> --outfile results.jsonl --model-path <ckpt_dir>                                                                                     

Multi GPU

python geneval/evaluation/evaluate_images.py <imagedir> --outfile results.jsonl --model-path <ckpt_dir> --options num_gpus=8 gpu_ids=0,1,2,3,4,5,6,7 clip_num_workers=0 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant