HuggingFace's model hub integration

[vwxyzjn]

Problem Description

HuggingFace's model hub has become a standard place to host trained models. Now it is expanding coverage to the RL space (see huggingface.co/sb3 as an example and DLR-RM/rl-baselines3-zoo#198) & it would be nice for us to integrate, too. I spoke with @ThomasSimonini today, and he expressed interest in working with this.

10/6/22 update: I would like to rethink how we can support saved models in CleanRL as they have become increasingly relevant (e.g., recent research on using models to bootstrap RL; see reincarnate RL)

Challenges

Huggingface and SB3 make a great fit because SB3 already provides a uniform API for training and evaluation. With CleanRL, this is tricky since CleanRL is more of a repository for educational and prototyping purposes: we don't have uniform APIs as SB3 does.

Desired Features:

• save model
• evaluate model
• upload model to HF
• load model from HF

Possible Solution

I think we can start the integration in a few selected files:

• ppo.py
• ppo_atari.py
• ppo_atari_envpool.py
• dqn_atari.py
• c51_atari.py

We can add an optional flag like the following

    parser.add_argument("--save-model-hf", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True,
        help="whether to save model to hugging face")

To add utilities like https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/rl_zoo3/push_to_hub.py, we could add a function upload_to_hf in the cleanrl_utils folder and import it. Importantly, we should only import it when the flag is turned on, so we don't make the single-file implementation dependent on the cleanrl_utils.upload_to_hf.

@ThomasSimonini has a demo here https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit8/unit8.ipynb

[vwxyzjn]

10/6/22 update: I would like to rethink how we can support saved models in CleanRL as they have become increasingly relevant (e.g., recent research on using models to bootstrap RL; see reincarnate RL). I have updated the issue description to reflect this change.

[kinalmehta]

Sounds good to me. Few queries:

• I don't see any model-saving functionality in any of the files mentioned. So we will have to add both model saving and loading functionality, right?
• if we are going with model saving and loading functionality, should we aim to take a more flexible approach, like continuing the training from a checkpoint?

FYI, I've added loading and saving functionality in #267 link as DIAYN supports using pre-trained skills to fine-tune a specific reward.