Skip to content

Fix loop variable shadowing in validation that corrupts sample iteration#811

Open
Mr-Neutr0n wants to merge 1 commit intozai-org:mainfrom
Mr-Neutr0n:fix/validation-loop-variable-shadowing
Open

Fix loop variable shadowing in validation that corrupts sample iteration#811
Mr-Neutr0n wants to merge 1 commit intozai-org:mainfrom
Mr-Neutr0n:fix/validation-loop-variable-shadowing

Conversation

@Mr-Neutr0n
Copy link

Summary

  • Fixed a variable shadowing bug in Trainer.validate() where the inner for i, ... in enumerate(validation_artifacts) loop shadows the outer for i in range(num_validation_samples) loop variable.
  • After the inner loop completes, i retains the last enumerate index rather than the outer sample index, which can cause incorrect validation behavior with multiple samples.
  • Renamed the inner loop variable from i to artifact_idx to eliminate the shadowing.

Details

In finetune/trainer.py, the validate() method has a loop over validation samples:

for i in range(num_validation_samples):       # outer loop (line 540)
    prompt = self.state.validation_prompts[i]  # uses i
    image = self.state.validation_images[i]    # uses i
    video = self.state.validation_videos[i]    # uses i
    ...
    for i, (artifact_type, artifact_value) in enumerate(validation_artifacts):  # shadows i! (line 591)
        artifacts.update({f"artifact_{i}": ...})

In Python, for loop variables are scoped to the enclosing function, not the loop body. The inner for i overwrites the outer i. While the outer for i in range() reassigns i at the top of each iteration, this shadowing is a correctness hazard that can produce subtle bugs if the code is modified, and violates the principle of least surprise.

Test plan

  • Verified only the inner loop variable name changes via git diff
  • Run validation with multiple samples to confirm correct indexing
  • Run with DeepSpeed ZeRO stage != 3 to confirm process distribution works

In the validate() method, the inner loop `for i, ... in enumerate(validation_artifacts)`
shadows the outer loop variable `for i in range(num_validation_samples)`. After the inner
loop completes, `i` holds the last enumerate index instead of the outer sample index.
This causes subsequent validation samples to load incorrect prompts, images, and videos,
and breaks DeepSpeed multi-GPU process distribution logic that depends on `i`.

Renamed the inner variable to `artifact_idx` to prevent shadowing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant