Poor video quality when using I2V or V2V with world-scenario-map control in Transfer

[eadstry]

I am trying to use the Transfer model for world-scenario-map-controlled video generation. Specifically, I provide the first frame as the image condition, and use world-scenario-maps rendered from different trajectories as the control condition to generate the subsequent video.

However, I found that the generated video quality is quite poor. The video often shows abrupt changes in style and structure after the first frame.

vision.mov

side_by_side.mov

Relevant configuration

model_mode = "video2video"

resolution = "720"
aspect_ratio = "16,9"
num_frames = 240
fps = 10
num_video_frames_per_chunk = 121

num_steps = 20
guidance = 1.5
control_guidance = 3
seed = 2026

num_first_chunk_conditional_frames = 8
num_conditional_frames = 1
condition_frame_indexes_vision = [0]
condition_video_keep = "first"
share_vision_temporal_positions = True

wsm_weight = 1.0

[trungtpham]

@eadstry thanks for reporting the issue. Can you try without the image condition?

[eadstry]

I used the tools in cosmos-drive-dream to convert waymo/10017090168044687777_6380_000_6400_000 into the RDS-HQ format. I tried using blur to preserve the original video style, but it still didn't work.

Ablation via conditional control

configuration : chunk=161, steps=20, guidance=1.0, ctrl_g=2.0,first_cond=16

Blur Control

control_blur.mov

vision.mov

Wsm Control

control_wsm.mov

vision.mov

Ablation of conditional video frames

configuration ：WSM=0.8, Blur=0.2, chunk=160, steps=20, guidance=1.0, ctrl_g=2.0

first_cond 0：

vision.mov

first_cond 8：

vision.mov

first_cond 16：

vision.mov

first_cond 24：

vision.mov

first_cond 40：

vision.mov