[Feature Request] Add support for gymnasium

[tobirohrer]

🚀 Feature

Add support of gymnasium (the replacement of gym). More details about the transition from gym to gymnasium can be found here.

Motivation

From gymnasium:
"Gym did a lot of things very well, but OpenAI didn’t devote substantial resources to it beyond its initial release. The maintenance of Gym gradually decreased until Gym became wholly unmaintained in late 2020. In early 2021, OpenAI gave us control over the Gym repository. [...]

This brings us to Gymnasium. It’s essentially just our fork of Gym that will be maintained going forward. It can be trivially dropped into any existing code base by replacing import gym with import gymnasium as gym, and Gymnasium 0.26.2 is otherwise the same as Gym 0.26.2."(see gymnasium for more information)

Pitch

• Add support for gymnasium, so it can be used with sb3
• Facilitate the usage of gymnasium instead of gym by updating the docs
• Create a migration guide

Alternatives

Stick with gym (IMO not really an alternative due to lack of maintenance in the future)

Additional context

• Currently, there is a PR to add support for gym v0.26 to sb3. There will be probably dependencies to this PR

Checklist

• I have checked that there is no similar issue in the repo

[araffin]

Hello,

Currently, there is a #780 to add support for gym v0.26 to sb3. There will be probably dependencies to this PR

yes, that's actually the plan to switch to gymnasium in that PR.

Create a migration guide

this should be done in gym.

[tobirohrer]

Okay, so we can close this issue?
Then I would try to help in #780 to add support for gymnasium

[qgallouedec]

I've made this clearer in #780 (comment).

[qgallouedec]

Closing as duplicate of #871