Reproduce the issue

[bxhsort]

I am facing an OOM (Out Of Memory) issue on an 8-GPU machine with 80GB of memory. The single-node server is consuming a large amount of GPU memory. How can I resolve this?

[Luciennnnnnn]

Hi, are you encountering OOM issues when starting the reward server or during RL training? Which model size are you using? Could you please share the detailed configuration so that we can better assist you.

[bxhsort]

你好，你在开启奖励服务器或进行现实学习训练时遇到过OOM问题吗？你用的是哪个型号尺寸？能否请您分享详细的配置，以便我们更好地协助您。

I trained using only 8 A800 GPUs and the EditScore Qwen2.5-VL 7B version. I found that when starting the server, it already consumes over 55 GB of GPU memory, which prevents OmniGen2’s RL training from running — it OOMs as soon as the server starts. If I reduce the batch size, the code runs normally.

I’d like to ask: when you mention “single machine training,” does that refer specifically to the machine used for RL training and not the machine running the server? If I run in a single-machine setup, how many GPUs should I allocate to the server in order to obtain a normal score?

[Luciennnnnnn]

Hi, in our setup, the reward server runs on a separate machine. This means training typically involves at least two machines, and it becomes quite complicated if you want the policy model and the reward model to share GPU memory.

If you only have a single machine available, I would suggest allocating 2 GPUs for the reward server and 6 GPUs for the main training, or alternatively 4 GPUs for the reward server and 4 GPUs for the main training.

To support this, you will need to make a small modification to the reward server code.

[bxhsort]

Hi, in our setup, the reward server runs on a separate machine. This means training typically involves at least two machines, and it becomes quite complicated if you want the policy model and the reward model to share GPU memory.

If you only have a single machine available, I would suggest allocating 2 GPUs for the reward server and 6 GPUs for the main training, or alternatively 4 GPUs for the reward server and 4 GPUs for the main training.

To support this, you will need to make a small modification to the reward server code.
since the loss is different from that in FlowGRPO, what kind of exploration or modification was made?

[Luciennnnnnn]

This modification is token from TempFlow-GRPO, other parts remains Flow-GRPO.

[bxhsort]

This modification is token from TempFlow-GRPO, other parts remains Flow-GRPO.

Thank you for your reply. When I train using 6 GPUs with sampling, the ratio stays at 1 and does not change.