question about training of AlignEvaluator

[JoshonSmith]

As illustrated in section 2.4, The AlignEvaluator is trained on (reprompt, image) pairs. But The GRPO Training Loop in section 2.3, AlignEvaluator calculates a scalar reward $r_i$ for each pair ($p_i, I_i$). Here, the $p_i$ denotes user prompt, not reprompt.

Would you please explain how this part was considered?

[ximinng]

Hi @JoshonSmith,

That's an excellent question!

You are correct, and this difference is intentional. It relates to the distinct goals during the AlignEvaluator's training versus its application within the GRPO Training Loop.

Here’s a breakdown of the reasoning:

1. In the GRPO Training Loop (Section 2.3): The ultimate objective is to train a generation model that aligns well with the original user's intent, which is represented by the user prompt $p_i$. Therefore, the reward model (AlignEvaluator) must assess how well the generated image $I_i$ aligns with that original prompt $p_i$. This is why we calculate the reward as $r_i = \text{AlignEvaluator}(p_i, I_i)$.

2. When Training the AlignEvaluator (Section 2.4): The goal here is to create a highly discerning and accurate "judge." To do this, we need to train it on the best possible data. The reprompt is a more detailed, descriptive, and structured version of the user prompt, which often clarifies ambiguities and explicitly mentions the key "evaluation criteria". By training on (reprompt, image) pairs, we provide a clearer and more robust signal, teaching the AlignEvaluator to be a much more capable expert in fine-grained text-image alignment.

In summary:

We train the AlignEvaluator using the higher-quality, more detailed (reprompt, image) pairs to make it a powerful and accurate "judge."

Then, we use this highly-trained "judge" within the GRPO loop to evaluate how well the generated images $I_i$ fulfill the original user's request $p_i$.

The purpose in each stage is different, which is why the prompt input varies.

Hope this clarifies things!