Introduce benchmark utilities

[vwxyzjn]

Description

This PR introduces utilities that help us to run the benchmark experiments more smoothly.

Previously, we relied on a very simple mechanism for conducting benchmark experiments such as

cleanrl/benchmark/sac/mujoco.sh

Lines 3 to 7 in 443bb14

	for env_id in "HalfCheetah-v2" "Walker2d-v2" "Hopper-v2"; do
	for seed in 1 2 3; do
	poetry run python cleanrl/sac_continuous_action.py --track --capture-video --wandb-project-name cleanrl --wandb-entity openrlbenchmark --env-id $env_id --seed $seed
	done
	done

Such bash script usage is pretty straightforward but lacks flexibility and configurability. This PR introduce a command such as

OMP_NUM_THREADS=1 python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "poetry run python cleanrl/ppo.py --track --cuda False" \
    --num-seeds 3 \
    --workers 3

which will automatically run experiments withworkers=3 subprocesses. A full example can be seen here:

# export WANDB_ENTITY=openrlbenchmark
# export WANDB_PROJECT=cleanrl

OMP_NUM_THREADS=1 python -m cleanrl_utils.benchmark \
    --env-ids CartPole-v1 Acrobot-v1 MountainCar-v0 \
    --command "poetry run python cleanrl/ppo.py --track --cuda False" \
    --num-seeds 3 \
    --workers 3

poetry install -E atari
python -m cleanrl_utils.benchmark \
    --env-ids PongNoFrameskip-v4 BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4 \
    --command "poetry run python cleanrl/ppo_atari.py --track" \
    --num-seeds 3 \
    --workers 1

python -m cleanrl_utils.benchmark \
    --env-ids PongNoFrameskip-v4 BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4 \
    --command "poetry run python cleanrl/ppo_atari_lstm.py --track" \
    --num-seeds 3 \
    --workers 1

python -m cleanrl_utils.benchmark \
    --env-ids PongNoFrameskip-v4 BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4 \
    --command "poetry run python cleanrl/ppo_atari_envpool.py --track" \
    --num-seeds 3 \
    --workers 1

poetry install -E "mujoco pybullet"
python -c "import mujoco_py"
OMP_NUM_THREADS=1 python -m cleanrl_utils.benchmark \
    --env-ids HalfCheetah-v2 Walker2d-v2 Hopper-v2 \
    --command "poetry run python cleanrl/ppo_continuous_action.py --track --cuda False" \
    --num-seeds 3 \
    --workers 3

poetry install -E procgen
python -m cleanrl_utils.benchmark \
    --env-ids starpilot bossfight bigfish \
    --command "poetry run python cleanrl/ppo_procgen.py --track --track" \
    --num-seeds 3 \
    --workers 1

Types of changes

• Bug fix
• New feature
• New algorithm
• Documentation

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the documentation and previewed the changes via mkdocs serve.

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/vwxyzjn/cleanrl/9tPNxQNMcQfs1X6PgsoMTTb7YCW8
✅ Preview: https://cleanrl-git-benchmark-utilities-vwxyzjn.vercel.app

[gitpod-io]

[dosssman]

Quite useful addition. Will try to properly use it for the next benchmarks.

[vwxyzjn]

Added more documentation. Going to merge this soon.

[yooceii]

If it simply spawns multiple processes, why don't we instead write a bash script?

[vwxyzjn]

I’m open to using a bash script. But the main benefit here is pythons code is a bit easier to read and also allows me to set a maximum number of workers. I don’t know if I can Set a maximum number of workers with bash.

[yooceii]

I think u can use poetry run python cleanrl/ppo.py & to spawn a new process and run it in the background.

[vwxyzjn]

True. The issue is setting a maximum amount of workers. Imagine I have 60 commands to run, poetry run python cleanrl/ppo.py & is just gonna overflow my CPU.

[vwxyzjn]

Merging now.