Roadmap to Stable-Baselines3 V1.0

[araffin]

This issue is meant to be updated as the list of changes is not exhaustive

Dear all,

Stable-Baselines3 beta is now out 🎉 ! This issue is meant to reference what is implemented and what is missing before a first major version.

As mentioned in the README, before v1.0, breaking changes may occur. I would like to encourage contributors (especially the maintainers) to make comments on how to improve the library before v1.0 (and maybe make some internal changes).

I will try to review the features mentioned in hill-a/stable-baselines#576 (and hill-a/stable-baselines#733)
and I will create issues soon to reference what is missing.

What is implemented?

• basic features (training/saving/loading/predict)
• basic set of algorithms (A2C/PPO/SAC/TD3)
• basic pre-processing (Box and Discrete observation/action spaces are handled)
• callback support
• complete benchmark for the continuous action case
• basic rl zoo for training/evaluating plotting (https://github.com/DLR-RM/rl-baselines3-zoo)
• consistent api
• basic tests and most type hints
• continuous integration (I'm in discussion with the organization admins for that)
• handle more observation/action spaces Add support for MultiDiscrete/MultiBinary observation spaces #4 and Add support for MultiDiscrete/MultiBinary action spaces #5 (thanks @rolandgvc)
• tensorboard integration Tensorboard integration #9 (thanks @rolandgvc)
• basic documentation and notebooks
• automatic build of the documentation
• Vanilla DQN Implement Vanilla DQN #6 (thanks @Artemis-Skade)
• Refactor off-policy critics to reduce code duplication Implement DDPG #3 (see Refactored ContinuousCritic for SAC/TD3 #78 )
• DDPG Implement DDPG #3
• do a complete benchmark for the discrete case Performance Check (Discrete actions) #49 (thanks @Miffyli !)
• performance check for continuous actions Performance check (Continuous Actions) #48 (even better than gSDE paper)
• get/set parameters for the base class (Get/set parameters and review of saving and loading #138 )
• clean up type-hints in docs Custom parser for type hints #10 (cumbersome to read)
• documenting the migration between SB and SB3 Migration guide #11
• finish typing some methods Improve typing coverage #175
• HER Implement HER #8 (thanks @megan-klaiber)
• finishing to update and clean the doc Missing Documentation #166 (help is wanted)
• finishing to update the notebooks and the tutorial Update colab notebooks #7 (I will do that, only HER notebook missing)

What are the new features?

• much cleaner base code (and no more warnings =D )
• independent saving/loading/predict for policies
• State-Dependent Exploration (SDE) for using RL directly on real robots (this is a unique feature, it was the starting point of SB3, I published a paper on that: https://arxiv.org/abs/2005.05719)
• proper evaluation (using separate env) is included in the base class (using EvalCallback)
• all environments are VecEnv
• better saving/loading (now can include the replay buffer and the optimizers)
• any number of critics are allowed for SAC/TD3
• custom actor/critic net arch for off-policy algos ([Feature request] Allow different network architectures for off-policy actor/critic #113 )
• QR-DQN in SB3-Contrib
• Truncated Quantile Critics (TQC) (see Implement Truncated Quantile Critics (TQC) #83 ) in SB3-Contrib
• @Miffyli suggested a "contrib" repo for experimental features (it is here)

What is missing?

• syncing some files with Stable-Baselines to remain consistent (we may be good now, but need to be checked)
• finish code-review of exisiting code Review of Existing Code #17

Checklist for v1.0 release

• Update Readme
• Prepare blog post
• Update doc: add links to the stable-baselines3 contrib
• Update docker image to use newer Ubuntu version
• Populate RL zoo

What is next? (for V1.1+)

• basic dict/tuple support for observations (Dictionary Observations #243 )
• simple recurrent policies? (recurrent policy implementation in ppo [feature-request] #18)
• DQN extensions (double, PER, IQN) ([Feature Request] RAINBOW #622)
• Implement TRPO (Add TRPO Stable-Baselines-Team/stable-baselines3-contrib#40)
• multi-worker training for all algorithms ([Feature request] Adding multiprocessing support for off policy algorithms #179 )
• n-step returns for off-policy algorithms [feature-request] N-step returns for TD methods #47 (@partiallytyped )
• SAC discrete [Feature request] Implement SAC-Discrete #157 (need to be discussed, benefit vs DQN+extensions?)
• Energy Based Prioritisation? (@RyanRizzo96)
• implement action_proba in the base class?
• test the doc snippets Sphinx doc tests support #14 (help is welcomed)
• noisy networks (https://arxiv.org/abs/1706.10295) @partiallytyped ? exploration in parameter space? ([Feature Request] RAINBOW #622)
• Munchausen Reinforcement Learning (MDQN) (probably in the contrib first, e.g. [WIP] MDQN pfnet/pfrl#74)

side note: should we change the default start_method to fork? (now that we don't have tf anymore)

[m-rph]

Maybe N-step returns for TD3 (and DDPG) and DQN (and friends) ? If it's implemented in the experience replay, then it is likely plug and play for TD3 and DQN, an implementation for SAC probably requires extra effort.

Perhaps at a later time, e.g. V1.1+ retrace, tree backup, Q(lambda), Importance Sampling for n-step returns?
If retrace and friends are planned for later, then it should be taken into consideration when implementing n-steps.

[Miffyli]

@partiallytyped

Yup, that would be v1.1 thing but indeed planned. Should probably go over original SB issues to gather all these suggestions at some point.

[m-rph]

Perhaps a discrete version of SAC for v1.1+?
https://arxiv.org/abs/1910.07207

Edit: I can implement this, and add types to the remaining methods after my finals (early June).

[rolandgvc]

I will start working on the additional observation/action spaces this weekend 👍

[jkterry1]

Will the stable baselines 3 repo/package replace the existing stable baselines one, or will all this eventually be merged into the normal stable baselines repo?

[Miffyli]

@justinkterry

There are no plans to merge/combine the two repositories. Stable-baselines will continue to exist, and continue to receive bug-fixes and the like for some time before it is archived.

[jkterry1]

@Miffyli Thank you. Will it remain as "pip3 install stable-baselines," or become something like "pip3 install stable-baselines3"?

[Miffyli]

@justinkterry

You can already install sb3 with pip3 install stable-baselines3. The original repo will stay as pip3 install stable-baselines.

[AdamGleave]

Minor point but I wonder if we should rename BaseRLModel to BaseRLAlgorithm and BasePolicy to BaseModel, given that BasePolicy is more than just a policy?

[araffin]

Minor point but I wonder if we should rename BaseRLModel to BaseRLAlgorithm and BasePolicy to BaseModel, given that BasePolicy is more than just a policy?

Good point, BaseModel and BaseRLAlgorithm are definitely better names ;)

[jdily]

for visualization, probably using something like weights & biases (https://www.wandb.com/) is an option?
So that no need for tensorflow dependency.
I can help to add functions to do that.

[araffin]

for visualization, probably using something like weights & biases (https://www.wandb.com/) is an option?

correct me if I'm wrong but W&B does not work offline, no? This is really important as you don't want your results to be published when you do private work.

This could be also implemented either as a callback (cf doc) or a new output for the logger. But sounds more like a "contrib" module to me.

[m-rph]

Perhaps an official shorthand for stable-baselines and stable-baselines3 e.g. sb and sb3?

import stable_baselines3 as sb3

[ManifoldFR]

Is it necessary to continue to provide the interface for vectorized environments inside of this codebase?
They were contributed upstream back to gym in this PR. After that PR was merged, packages such as PyBullet (pybullet_envs) started providing vectorized variants of their own environments using the interface from gym which should be the same as the one here (for now)

[Miffyli]

@ManifoldFR Somehow that has eluded my attention. Looks like a good suggestion! Less repeated code is better, as long it fits in stable-baselines functions too.

@araffin thoughts (you have most experience doing the eval/wrap functions)? I imagine the hardest part is to update all wrappers that work on vectorized environments.

[ManifoldFR]

I happened onto it by chance because it's not documented anywhere inside of gym's docs, the openai people ported it from their own baselines repo with barely any notification of the change to end users.

[araffin]

I was aware of this (wrote some comments at that time https://github.com/openai/gym/pull/1513/files#r293899941) but I would argue against for different reasons:

• we rely on some specific features (set_attr, get_attr)
• openai version is undocumented and we don't know if they gonna break that feature (which is central in SB3) in a future release (I don't want to write new monkey patch like hill-a/stable-baselines@678f803)
• we can directly tweak that feature to fit our needs (and don't wait for a review and release by OpenAI)

So, in short, I would be in favor only if OpenAI way of maintaining Gym was more reliable.

PS: thanks @ManifoldFR for bringing that up ;)

[ManifoldFR]

I see, it figures that OpenAI changing things unilaterally without documentation would be a problem. I guess ensuring stable-baselines3's code doesn't break when running vectorized envs derived from Gym's implementation would be safer and easier instead...

[cosmir17]

@partiallytyped I think you are already aware but I am also mentioning here. I found a source code example for the original SAC Discrete Implementation paper (the one you found).

The author also publicised his code.
https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch/blob/master/agents/actor_critic_agents/SAC_Discrete.py

Hope these help,
Sean

[araffin]

I found the original example for the SAC Discrete Implementation plan.
Can the following paper be considered?
https://arxiv.org/abs/1910.07207

We already have an issue for that #157

[cosmir17]

@araffin I wasn't asking if we can implement it. Given that it has been decided as shown on the load-map and PartialTyped volunteered here, I was giving him or the team a resource. #1 (comment)
Actually, I missed the footnote
need to be discussed, benefit vs DQN+extensions?
I am posting my suggestion in the page now.

[araffin]

First release candidate is out: https://github.com/DLR-RM/stable-baselines3/releases/tag/v1.0rc0
100+ trained rl models will be published soon: DLR-RM/rl-baselines3-zoo#69