Strategy for applying rCM with Autoregressive Student

[lapp0]

Hi, thanks for your great work!

I saw causal models were on the roadmap.

For rCM, the DMD component can easily be replaced with Causal Forcing or Self Forcing to keep the rollout on manifold. However I was wondering what the planned approach was for adjusting the sCM loss component so the student can be conditioned on its own rollout? Will using a bidirectional teacher and a causal student conditioned on its own generated sequence be stable for sCM?

Is sCM too sensitive to conditioning mismatch, requiring one of these?

• a different method (sCT?)
• a teacher initialized on self-generated sequences (e.g. resampling forcing)
• perhaps for the sCM path the student and teacher don't need a self-generated sequence (teacher forced ground truth for both models)?
• something else?

Appreciate any insights.

[zhengkw18]

Causal-rCM will be released in the rCM codebase (https://github.com/NVlabs/rcm), stay tuned! We will have the best open-sourced causal training algorithms/infrastructure.

Briefly speaking, we require teacher-forcing CM plus self-forcing DMD. Both discrete-time CM (dCM) and sCM will be supported. The training process will be similar to APT2 (https://arxiv.org/pdf/2506.09350, see the attached figure). We first use teacher-forcing to turn the bidirectional model into causal, then do teacher-forcing-based CM, followed by self-forcing DMD. The difference to APT2 is that we use DMD instead of GAN in the last self-forcing stage. Joint training of teacher-forcing CM and self-forcing DMD is also viable.