[Feature] Add OpenSpielWrapper and OpenSpielEnv

[kurtamohler]

Description

Adds environment wrapper classes for OpenSpiel.

OpenSpielWrapper.reset supports resetting to a specified state.

Motivation and Context

Part of #2133

• Someone has raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2345

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 7 Unrelated Failures

As of commit 1c90d04 with merge base e82a69f ():

NEW FAILURES - The following jobs have failed:

• Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 0ebcf6625dffb13eaf98fed4e3b81ab80ddd590a16eefa21b00697c196510f58 /exec failed with exit code 139
• Libs Tests on Linux / unittests-gym (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t fb62162595af9a039c6fae565f9fb1c8e44e791df21e76ed1e7852856acd0fcc /exec failed with exit code 1
• Libs Tests on Linux / unittests-robohive (3.9, 12.1) / linux-job (gh)
  test/test_libs.py::TestRoboHive::test_robohive[franka_slide_random-v3-True-True]

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda11_8 (gh) (detected as infra flaky with no log or failing log classifier)
• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda12_1 (gh) (detected as infra flaky with no log or failing log classifier)
• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda12_4 (gh) (detected as infra flaky with no log or failing log classifier)
• Continuous Benchmark (PR) / CPU Pytest benchmark (gh) (detected as infra flaky with no log or failing log classifier)
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cpu (gh) (trunk failure)
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestKLRewardTransform::test_kl_lstm

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kurtamohler]

A few notes:

categorical_action_encoding=False is not supported yet.

Also, some of the games in OpenSpiel do not work properly in OpenSpielWrapper because the action spec assumes a discrete space of pyspiel.Game.num_distinct_actions() (see OpenSpiel API reference). However, for some of the games, pyspiel.State.legal_actions() can return more actions than pyspiel.Game.num_distinct_actions(). I suppose to support those games we need to allow the action spec's size to change at each step?

[vmoens]

Great work thanks for that, I left a couple of comments already.
Is the TODO blocking on this?

[vmoens]

let's put a flag around that to make the state optional. Maybe "stateless=True"?

[kurtamohler]

What if we call the flag output_state? I think that name makes it more obvious what exactly it does--it adds the state to the output of reset and step

[kurtamohler]

Added the output_state flag, let me know if you'd rather call it stateless or something else

[kurtamohler]

I renamed to return_state, because that's what PettingZooWrapper calls it

[kurtamohler]

At the moment, this only supports games where all actions are taken by the players, like in chess or tic-tac-toe. But I've realized that OpenSpiel also has a concept of chance nodes, where a random non-player action is taken. For instance, in Kuhn poker, the initial dealing of the cards is a chance node. In liar's dice, the outcome of rolling dice is a chance node. OpenSpiel has some methods to obtain all the possible chance actions and the associated probability distribution (shown in this example).

For now, I'll raise an error if a loaded game contains chance nodes and leave it as a TODO. I might wait to add support for it in a follow-up PR--unless you would prefer for me to add it in this PR.

[vmoens]

I suppose to support those games we need to allow the action spec's size to change at each step?

Yes I think having a dynamic space would be the way to go. See #2143

[vmoens]

OpenSpiel has some methods to obtain all the possible chance actions and the associated probability distribution (shown in this example).

That's an amazing feature. Happy to integrate it separately!

[kurtamohler]

I suppose to support those games we need to allow the action spec's size to change at each step?

Yes I think having a dynamic space would be the way to go. See #2143

Actually, I'm not so sure that we would need dynamic action specs after all. It is true that pyspiel.State.legal_actions() can return a different number of actions than pyspiel.Game.num_distinct_actions(), but I'm pretty sure (not 100% sure) that only happens when it's a chance node, in which case the length of legal_actions() is pyspiel.Game.max_chance_outcomes() instead. Once I add support for chance nodes, it will have its own action spec separate from the players' action specs, and I think all action specs will maintain the same shape throughout the game.

[kurtamohler]

I think I've addressed everything that needed to be fixed so far. Let me know if there is anything else

[vmoens]

Amazing thanks a mil for this!