Refactor replay based scripts

[vwxyzjn]

Description

This PR closes #171, closes #172, closes #168, and closes #148.

• Seed issue with dqn.py and others #171
• TD3 should also log losses for qf2 #172
• Also log episodic_length for non-PPO scripts. #168
• Investigate nn.utils.clip_grad_norm_ for DQN, DDPG, and TD3 #148

Types of changes

• Bug fix

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the documentation and previewed the changes via mkdocs serve.
• I have updated the tests accordingly (if applicable).

If you are adding new algorithms or your change could result in performance difference, you may need to (re-)run tracked experiments. See #137 as an example PR.

• I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team (required).
• I have tracked applicable experiments in openrlbenchmark/cleanrl with --capture-video flag toggled on (required).
• I have added additional documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers (if applicable).
  • I have added links to the PR related to the algorithm.
  • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves (in PNG format with width=500 and height=300).
  • I have added links to the tracked experiments.
• I have updated the tests accordingly (if applicable).

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/vwxyzjn/cleanrl/7NBoCmqbCsrTeVtqKRpZUxAGFZ8N
✅ Preview: https://cleanrl-git-refactor-replay-based-scripts-vwxyzjn.vercel.app

[gitpod-io]

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
cleanrl	✅ Ready (Inspect)	Visit Preview	May 9, 2022 at 9:16PM (UTC)

[vwxyzjn]

Here are the benchmarked results. Looks like DQN in this PR gets better performance in Atari games, slightly worse results in MountainCar-v0. SAC in this PR gets a performance boost.

Given these results, I recommend we merge this PR. This PR obtains overall better performance and removes an unverified code-level optimization: gradient norm clipping for DQN.

@dosssman and @yooceii, does the result from this PR make sense to you? If it does, I will make updates to the docs and ultimately remove the old experiments. I think after this we would be ready for the 1.0 release.

CC @araffin who might be interested in this :)

Atari games

Classic control

MuJoCo

[dosssman]

Looking good on my side.
Thanks for the great work.

[vwxyzjn]

Merging now.