1: Torch compiled without CUDA. 2: bad covariance_matrix

[mkultra333]

Trying to set this up on a windows 10 machine with 16 gig ram and an RTX 3070 8 gig card.

After following the instructions, my first attempt ended with an error saying torch was not compiled with CUDA:
[ERROR] Error during text-to-video generation: Torch not compiled with CUDA enabled

I installed torch and torchvision cuda versions to try and fix this, so the system was now like this:

import torch
import torchvision
print(torch.version) # Check PyTorch version
2.4.1+cu118
print(torchvision.version) # Check torchvision version
0.19.1+cu118
print(torch.cuda.is_available()) # Should return True if CUDA is enabled
True

Next attempt, tried both text to video and image to video, both times got an error to do with matrices.

[ERROR] Error during image-to-video generation: Expected parameter covariance_matrix (Tensor of shape (4, 4)) of distribution MultivariateNormal(loc: torch.Size([4]), covariance_matrix: torch.Size([4, 4])) to satisfy the constraint PositiveDefinite(), but found invalid values:
tensor([[ 1.0000, -0.3333, -0.3333, -0.3333],
[-0.3333, 1.0000, -0.3333, -0.3333],
[-0.3333, -0.3333, 1.0000, -0.3333],
[-0.3333, -0.3333, -0.3333, 1.0000]])

[mkultra333]

Installed older versions of cuda enabled torch and torchvision. That got rid of the matrix error.

(pyramid) E:\PyraF\Pyramid-Flow>pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 -f https://download.pytorch.org/whl/torch_stable.html

Was running but then ran out of memory.

ERROR] Error during text-to-video generation: CUDA out of memory. Tried to allocate 80.00 MiB. GPU 0 has a total capacty of 8.00 GiB of which 0 bytes is free. Of the allocated memory 13.30 GiB is allocated by PyTorch, and 235.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
12%|█████████▋ | 2/16 [08:39<1:00:35, 259.71s/it]
[ERROR] Error during text-to-video generation: CUDA out of memory. Tried to allocate 76.00 MiB. GPU 0 has a total capacty of 8.00 GiB of which 0 bytes is free. Of the allocated memory 13.28 GiB is allocated by PyTorch, and 259.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I'm not sure how I'm supposed to enable the low memory stuff that makes it run on 8 gig cards.

[feifeiobama]

Glad to see that you resolved the initial issues using an older PyTorch version (see #121 for more discussions).

For the CUDA out of memory error, please enable sequential CPU offloading according to #75. It will be slower, but might work with 8GB GPU memory.

[mkultra333]

I went into the code and changed pyramid_dit_for_video_gen_pipeline.py line 332

cpu_offloading: bool = True, # If true, reload device will be cuda

After that it works, and I successfully rendered a text to video. But I got a weird new issue.
Looking in the Anaconda console I can see it rendering all the seconds, I've tried 4 seconds and 16 seconds.
But then the video created doesn't match, the first was just 1 second and the other was just 5 seconds.

I don't know if I should mention that as a new issue. Is it possible to save the individual frames?

[feifeiobama]

I went into the code and changed pyramid_dit_for_video_gen_pipeline.py line 332

cpu_offloading: bool = True, # If true, reload device will be cuda

After that it works, and I successfully rendered a text to video. But I got a weird new issue. Looking in the Anaconda console I can see it rendering all the seconds, I've tried 4 seconds and 16 seconds. But then the video created doesn't match, the first was just 1 second and the other was just 5 seconds.

I don't know if I should mention that as a new issue. Is it possible to save the individual frames?

Nice observation. The temp parameter does not specify the generated video length in seconds, but the number of latent frames generated. In fact, 4 latent frames correspond to 1 second, while 16 latent frames correspond to 5 seconds. Please see #135 (comment) for details.

[mkultra333]

I see, thanks.

[mkultra333]

I guess I'd finish with the following suggestions:

1. Something to make sure the conda environment installs CUDA enabled torch and torchvision.
2. Either some way to enable cpu_offloading from the Gradio UI, or at least a note making it a bit clearer what you have to do to get it working.
3. Maybe a note on the Duration parameter explaining that it isn't seconds, but latent frames.

I'm still impressed that it managed to run on 16 gig RAM and 8 gig GPU though, well done! :)
It crashes Out Of Memory if I try to do 768p, so I can only do 384p. But that's fine, I'm amazed I can generate anything at all.

You can close this if you like.