Migrate to gymnasium

[Rocamonde]

No description provided.

[Rocamonde]

@ernestum could you take an initial look at this? There's still some errors to fix but seems like most of the stuff is updated now.

[Rocamonde]

Also @AdamGleave introducing some API changes here, since now that gym has generic types a lot of our weird workarounds are no longer necessary. In particular, at the ABC top-level, I am keeping inheritance of observation_space and action_space type declaration instead of creating our custom private variable etc. Users have to create these replacements on implementation. I am also migrating the old random number generation to the new interface and adding proper generic typing to numpy where appropriate.

Also, since Discrete is an int64, the state and action values now are also int64.

[Rocamonde]

Also we seem to be getting a CI error:

Either git or ssh (required by git to clone through SSH) is not installed in the image. 
Falling back to CircleCI's native git client but the behavior may be different from official git. 
If this is an issue, please use an image that has official git and ssh installed.

Don't think this is related to my changes?

[ernestum]

Looks good so far. Thanks for the fast progress on this one!

[ernestum]

I don't think the CI error should be due to your changes.

[RedTachyon]

I left a few comments, but it wasn't necessarily 100% comprehensive

One thing to consider, which I don't have a good answer for right now (would need to spend some more time getting used to the library) -- when we hit the end of a fixed-horizon env, should it be a terminated or truncated situation?

The first-order approximation (in "regular" RL) is that if you're hitting a time limit, it's truncation; if you're reaching a terminal state, it's termination.

The second-order approximation is that if, in regular RL training, you'd use the value estimation in the final step to get the full discounted reward estimate, then that's truncation.

Fixed-horizon envs might have a weird interaction with this. In the end, the most important thing is probably that it's consistent between here and imitation. This might also slightly push it towards using truncated, since there are many environments (like Half-Cheetah, for example) which will use that semantic automatically

[AdamGleave]

@Rocamonde what's the current status on this? Planning on having someone take a look at the Gymnasium PR in SB3 next week.

[Rocamonde]

@Rocamonde what's the current status on this? Planning on having someone take a look at the Gymnasium PR in SB3 next week.

Currently quite swamped with other things and have not been able to look at this. I expect that I will have a bit more time in the future.

[ernestum]

Closing this in favor of #73