Huggingface Integration

[vwxyzjn]

Description

This PR closes #110. https://huggingface.co/cleanrl/CartPole-v1-dqn-seed1 is an example model page.

Types of changes

• Bug fix
• New feature
• New algorithm
• Documentation

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the documentation and previewed the changes via mkdocs serve.
• I have updated the tests accordingly (if applicable).

If you are adding new algorithms or your change could result in performance difference, you may need to (re-)run tracked experiments. See #137 as an example PR.

• I have contacted vwxyzjn to obtain access to the openrlbenchmark W&B team (required).
• I have tracked applicable experiments in openrlbenchmark/cleanrl with --capture-video flag toggled on (required).
• I have added additional documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers (if applicable).
  • I have added links to the PR related to the algorithm.
  • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves (in PNG format with width=500 and height=300).
  • I have added links to the tracked experiments.
  • I have updated the overview sections at the docs and the repo
• I have updated the tests accordingly (if applicable).

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
cleanrl	✅ Ready (Inspect)	Visit Preview	Jan 4, 2023 at 8:20PM (UTC)

[vwxyzjn]

The integration also makes it easier to just run models, such as

cleanrl/cleanrl_utils/evals/dqn_eval.py

Lines 43 to 57 in 4074eee

	from huggingface_hub import hf_hub_download
	from cleanrl.dqn import QNetwork, make_env
	model_path = hf_hub_download(repo_id="cleanrl/CartPole-v1-dqn-seed1", filename="q_network.pth")
	evaluate(
	model_path,
	make_env,
	"CartPole-v1",
	eval_episodes=10,
	run_name=f"eval",
	QNetwork=QNetwork,
	device="cpu",
	epsilon=0.05,
	capture_video=False,
	)

[vwxyzjn]

CC @ThomasSimonini for review :) Thanks!

[kinalmehta]

LGTM.

[simoninithomas]

First, I want to thank you for this integration and all the work behind 🤗. The result is fantastic, especially the model card with visible hyperparameters.

I gave some insights based on Omar and Lucain and my review on how we can improve the push_to_hub part. I'm happy to help with this.**

In addition to this, we can:

1. Having a downstream part ( load_from_hub ), I can help with that.
2. Generating a json/yaml file containing the hyperparameters for reproducibility?
3. Adding the library to our Hub list so that it creates a tag for people searching for cleanrl models.

[vwxyzjn]

@kinalmehta @simoninithomas and @Wauplin, thanks for the review. The CommitOperation suggestion is really helpful. Regarding some further comments:

Having a downstream part ( load_from_hub ), I can help with that.
Appreciate the help! That said, we are only downloading a single file from the hub, so having a customized load_from_hub might be unnecessary, right?

cleanrl/cleanrl_utils/evals/dqn_eval.py

Line 48 in b430540

model_path = hf_hub_download(repo_id="cleanrl/CartPole-v1-dqn-seed1", filename="q_network.pth")

Generating a json/yaml file containing the hyperparameters for reproducibility?

Are you thinking of loading from the yaml file somehow to run the script like python dqn.py --load-yaml hyper.yaml?

Adding the library to our Hub list so that it creates a tag for people searching for cleanrl models.

That would be great! Thank you!

[simoninithomas]

Hi @vwxyzjn , yes for yaml I was thinking what you mentioned.

That said, we are only downloading a single file from the hub, so having a customized load_from_hub might be unnecessary, right?

Yes and no, because it has two advantages:

1. We are able to count how many download of the model each month.
   

2. We can cache the model without using hf_hub_download directly.

For instance with SB3 integration here's the code for load_from_hub:

def load_from_hub(repo_id: str, filename: str) -> str:
    """
    Download a model from Hugging Face Hub.
    :param repo_id: id of the model repository from the Hugging Face Hub
    :param filename: name of the model zip file from the repository
    """
    try:
        from huggingface_hub import hf_hub_download
    except ImportError:
        raise ImportError(
            "You need to install huggingface_hub to use `load_from_hub`. "
            "See https://pypi.org/project/huggingface-hub/ for installation."
        )

    # Get the model from the Hub, download and cache the model on your local disk
    downloaded_model_file = hf_hub_download(
        repo_id=repo_id,
        filename=filename,
        library_name="huggingface-sb3",
        library_version="2.1",
    )

    return downloaded_model_file

[simoninithomas]

FIY From our side, we started to work on the frontend integration 🤗
huggingface/hub-docs#447

[vwxyzjn]

Thank you @simoninithomas

We are able to count how many download of the model each month.

Does this mean hf_hub_download(repo_id="cleanrl/CartPole-v1-dqn-seed1", filename="q_network.pth") would not trigger the download stats?

We can cache the model without using hf_hub_download directly.

Does hf_hub_download not cache models? I ran the dqn_eval.py and noticed the download progress bar only presents once and it did not appear again during the second run, so I assumed hf_hub_download caches automatically.

FIY From our side, we started to work on the frontend integration 🤗
huggingface/hub-docs#447

Awesome thanks! :)

[Wauplin]

Hi @vwxyzjn

Does this mean hf_hub_download(repo_id="cleanrl/CartPole-v1-dqn-seed1", filename="q_network.pth") would not trigger the download stats?

I'll let @simoninithomas answer on that as I am 100% sure what is counted in # downloads / month.
Worth noticing that the example from @simoninithomas uses 2 kwargs library_name and library_version to make the Hub know which lib is downloading the model (e.g. a cleanrl user and not a random user).

Does hf_hub_download not cache models?

Yes it does ! No matter if you use hf_hub_download or snapshot_download , your files will be downloaded only once.

[vwxyzjn]

@simoninithomas @kinalmehta @Wauplin thanks so much for helping with this PR. I think everything looks good at this point. We also have a good notebook ready to go https://colab.research.google.com/github/vwxyzjn/cleanrl/blob/hf-integration/docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb. Documentation can be previewed at https://cleanrl-git-hf-integration-vwxyzjn.vercel.app/get-started/zoo/ (the embed link is broken in it because it's pointing to the master branch).

[vwxyzjn]

Merging this as is, subjecting to future PRs. We'd also probably use huggingface/blog#616 to make the announcement. Thanks for the great work, folks!

[Wauplin]

Congrats ! That was a big piece of work 🎉🎉

[simoninithomas]

Congratulations 👏 I was off at the end of last week. I'm preparing the blogpost for next week and we're going to have a unit using CleanRL on PPO with Edward and me using GodotRL we will have the PR this week I'll mention you to put you in the loop.