Skip to content

README.md

Latest commit

 

History

History
93 lines (71 loc) · 2.35 KB

README.md

File metadata and controls

93 lines (71 loc) · 2.35 KB

Data Preprocessing Pipeline by Helios

This repository describes the data preprocessing pipeline used in the Helios paper. And we prepare a toy training data here.

⚙️ Requirements and Installation

Environment

# Activate conda environment
conda activate helios

🗝️ Usage

Step 1 - Prepare Metadata and Organize Videos

To train your own video generation model, create JSON files following this format:

[
    {
        "cut": [0, 81],
        "crop": [0, 832, 0, 480],
        "fps": 24.0,
        "num_frames": 81,
        "resolution": {
        "height": 480,
        "width": 832
        },
        "cap": [
        "A stunning mid-afternoon ..."
        ],
        "path": "videos/2_240_ori81.mp4"
    },
    {
        "cut": [0, 81],
        ...
    }
...
]

and arrange video files following this structure:

📦 example/
├── 📂 toy_data/
│   ├── 📂 videos
│   │   ├── 2_240_ori81.mp4
│   │   ├── 239_120_ori129.mp4.mp4
│   │   └── ...
│   └── 📄 toy_data_1.json
│
├── 📂 toy_data_2/
│   │   ├── A.mp4
│   │   ├── B.mp4
│   │   └── ...
│   └── 📄 toy_data_2.json
...

Step 2 - Prepare Autoregressive Real Data

These data can be used for training Stage-1, Stage-2, and Stage-3.

# Remember to modify the input and output paths before running
bash get_short-latents.py

Step 3 - Prepare Autoregressive ODE Data

These data can only be used for training Stage-3.

# Remember to modify the input and output paths before running
bash get_ode-pairs.sh

(Optional) Step 4 - Prepare Text Data

If you want to use the Self-Forcing training approach, prepare text embeddings:

# Remember to modify the input and output paths before running
bash get_text-embedding.sh

🔒 Acknowledgement