bad memory management

[Manoa1911]

2 hour movies take more than 500 GB of system memory :(

frames = frames.to(device)

you are sending ALL the frames of the movie into video memory before even processing :( do you seriously expect it to fit ?

Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [01:30<00:00, 15.11s/it]
Traceback (most recent call last):
  File "X:\DiffVSR-master\inference_tile.py", line 298, in <module>
    main(args)
  File "X:\DiffVSR-master\inference_tile.py", line 229, in main
    vframes, fps, size, video_name = read_frame_from_videos(video_path)
  File "X:\DiffVSR-master\utils.py", line 33, in read_frame_from_videos
    frames = frames.to(device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 34.33 GiB (GPU 0; 23.99 GiB total capacity; 2.99 GiB already allocated; 19.76 GiB free; 3.00 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[xh9998]

Thank you for your trial and valuable opinions. I have optimized the code. Welcome to try, but we recommend testing on GPUs with more than 80G memory. For two-hour movies, it is strongly recommended to perform slice testing first, then large-scale processing. You can consider adding additional parallel measures.

[Manoa1911]

thank, i will test it, related: #5
80 giga video cards costs 100,000$ :D

I have a question for you, if I do segmentation code, what is the optimal ratio between spatial pixel count to temporal frame count ?
example: for a 400^2 how many frames should be allocated per segment of temporal awearness ?
is different for 800^2 movie right ? so how much temporal awearness would you recommend for each case ?
or do you recommend always bias spatial first and whatever is left in memory to allocate for temporal ?

def read_video_segment(video_path: str, start_frame: int, num_frames: int) -> Tuple[torch.Tensor, int, Tuple[int, int]]:
    cap = cv2.VideoCapture(video_path)
    fps = int(cap.get(cv2.CAP_PROP_FPS))
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame)
    frames = []

    for _ in range(num_frames):
        ret, frame = cap.read()
        if not ret:
            break
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        frame = torch.from_numpy(frame).permute(2, 0, 1).float()  # C, H, W
        frames.append(frame)

    cap.release()
    if not frames:
        return None, fps, (height, width)

    vframes = torch.stack(frames)  # T, C, H, W
    return vframes, fps, (height, width)

        print(f"Video name: {video_name}\nPrompt: {prompt}")

        cap = cv2.VideoCapture(video_path)
        fps = int(cap.get(cv2.CAP_PROP_FPS))
        total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        cap.release()

        segment_length = (fps * 8) - 4  # 7 seconds
        segment_start = 0
        all_segments = []
        global_frame_index = 0  # to name images correctly

        start_time = time.time()
        while segment_start < total_frames:
            vframes, _, _ = read_video_segment(video_path, segment_start, segment_length)
            if vframes is None:
                break

            upscaled = process_video(vframes, pipeline, prompt, args)

            # Resize output
            h, w = vframes.shape[-2:]
            upscaled = F.interpolate(upscaled, size=(h * 4, w * 4), mode='bilinear', align_corners=False)
            upscaled = (upscaled / 2 + 0.5).clamp(0, 1) * 255
            upscaled = upscaled.to(torch.uint8).cpu()

            # Save frames as images (per segment)
            if args.outputimage_path:
                outimage_path = Path(args.outputimage_path) / video_name
                outimage_path.mkdir(parents=True, exist_ok=True)
                for i, frame in enumerate(upscaled):
                    frame_path = outimage_path / f"{global_frame_index + i:06d}.png"
                    save_image_tensor(frame, frame_path)  # helper function to save single frame

            global_frame_index += upscaled.shape[0]
            all_segments.append(upscaled)

            segment_start += segment_length
        process_time = time.time() - start_time

so the question is, give more memory to --tile_size or give more memory to segment_length ?

[xh9998]

During the training process, we train with a fixed segment size of 8 frames, and we believe that maintaining it will yield better results. We also conduct training and debugging with relatively limited computing power (see the paper). If you want to make changes, you can experiment to see if the effect is acceptable. We recommend changing the memory consumption through spatial partitioning. If it is a video super-resolution task that is very large in both space and time (like in your case), it is inherently a very computationally intensive task (if you want to super-resolve within a limited time). Within the computational power of a single GPU you have, please try to make the tile size as large as possible. If there is overlap, it can be about a quarter of the tile size, or it can also be slightly reduced.

[Manoa1911]

ok I will, thank :)
is the performance supposed to take many hours per frame ? on tile size 128 it's been running 4 hours and still tile: [1/4] x [4/7]
my card is set to maximum power with +1600 memory clock and it still take so many time :x it's a GA102

[xh9998]

Not like that in my test. Check if the software version is not matched or the GPU model's speed.
I think if you want to carry out large-scale video processing, you can consider short-term rental of faster and larger GPUs to adapt to the task. If the card cannot be replaced, you can consider using a one-step video super-resolution model that has recently been optimized for graphics memory or effect. Whether the effect meets expectations can be tested later.
The following case of DiffVSR is for reference.

Padded frames: 5, Current shape: torch.Size([16, 3, 566, 770]) Processing the video w/ tile patches [4x3]... tile: [1/2] x [1/3] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:59<00:00, 1.18s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:58<00:00, 1.18s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:58<00:00, 1.18s/it] tile: [1/2] x [2/3] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:10<00:00, 1.41s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:10<00:00, 1.41s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:10<00:00, 1.41s/it] tile: [1/2] x [3/3] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:02<00:00, 1.26s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:02<00:00, 1.26s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:02<00:00, 1.26s/it] tile: [2/2] x [1/3] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:11<00:00, 1.43s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:11<00:00, 1.43s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:11<00:00, 1.43s/it] tile: [2/2] x [2/3] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:25<00:00, 1.70s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:24<00:00, 1.70s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:25<00:00, 1.70s/it] tile: [2/2] x [3/3] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:15<00:00, 1.50s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:14<00:00, 1.50s/it] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:14<00:00, 1.50s/it] Saved processed video "m1" to output/m1_diffvsr_n50_g5_s50.mp4, Processing time: 1386.98s

[0xga]

Inferencing with H100 GPU for 5s video, there was about 38 diffusion sampling process, and each one took about 2 mins, which took about 70 more mins totally. What's the problem on my side?

[Manoa1911]

for me it makes 43 lines between each tile message :( it verry slow, 35 hours already and still not a single frame is out :(
also you should know, at 128 tile size, the memory used is 15 GB, but at the end of each piece it's spiking to almost 24 GB, is there a way to optimize it in a way that will allow maximum memory load for both processes ? to boost 15 to 24 without OOM at whatever is using 24 ?

after 41 hours it wrote the frames :) the performance is 0.00115 fps, 16.4 kWh, 170 frames, so 0.1 kWh per frame, for 2 hour movie at 30 fps: 2 * 60 * 60 * 30 = 216000, 216000 / 170 = 1270, 1270 * 16.4 = 20828 kWh for each 2 hours movie

[Losses]

Reproduced, extremely slow.