[Feature Request] RAINBOW

[araffin]

Important Note: We do not do technical support, nor consulting and don't answer personal questions per email.
Please post your question on the RL Discord, Reddit or Stack Overflow in that case.

🚀 Feature

Implement RAINBOW (DQN + all extensions: noisy-net, double dqn, dueling, c51, prioritized experience replay).

closes #487 if it is implemented.

Motivation

This would allow to have DQN with all extensions while keeping DQN code simple.

• I have checked that there is no similar issue in the repo (required)

[Sino-Huang]

Any updates about the rainbow implementation?

[araffin]

Any updates about the rainbow implementation?

Contributions are welcomed ;) (if you do so, please read the contributing guide from SB3-Contrib, it explains how to test new algorithms)
It is planned but not a priority.

EDIT: QR-DQN is available in SB3-Contrib, and double DQN is also available if needed (currently as an exercise)

[jenschwich]

Hi there, I hope you're having a great day. I was reading through #487 and it leads me here. I see that the double-DQN is now available as an exercise. I looked through sb3 and sb3-contrib repositories, but I couldn't find any code or reference to it. Would you be able to point me in the direction of the material or pseudo code that explains the implementation of double-DQN in sb3? I'd really appreciate it. Thank you!

[qgallouedec]

Hi @jenschwich, check out https://github.com/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/README.md

[noahfarr]

Hi,
do you already know how long it will take until Rainbow is available? I would like to use it for a project I'm currently working on. Is there already a preliminary implementation that maybe just hasn't been tested thoroughly? Is there something I could do to contribute to Rainbow being available faster?

[araffin]

Is there already a preliminary implementation that maybe just hasn't been tested thoroughly?

If you read this issue and the linked one, you will see components that have been implemented but not fully tested.

Is there something I could do to contribute to Rainbow being available faster?

Yes, we need help for #1622 to debug and be able to reproduce the original results (and maybe re-write this implementation).
And we need the same for the rest of the components (c51, dueling, double, n-steps), please read the contributing guide for SB3 contrib for the different steps needed.

As I wrote in #1622 (comment), the plan is to keep SB3 DQN implementation as-is (vanilla) but have a RAINBOW implementation with all the tricks too on the side.