Make RecordEpisodeStatistics work with VectorEnv #2296

vwxyzjn · 2021-08-05T03:44:17Z

Following up with #2279, this PR makes RecordEpisodeStatistics work with VectorEnv as well. That is

import gym
from gym.vector import SyncVectorEnv

def make_env(gym_id, seed):
    def thunk():
        env = gym.make(gym_id)
        env = gym.wrappers.RecordEpisodeStatistics(env)
        env.seed(seed)
        env.action_space.seed(seed)
        env.observation_space.seed(seed)
        return env
    return thunk

envs = SyncVectorEnv([make_env("CartPole-v1", 1 + i) for i in range(2)])
envs.reset()
for i in range(100):
    _, _, _, infos = envs.step(envs.action_space.sample())
    for info in infos:
        if "episode" in info.keys():
            print(f"i, episode_reward={info['episode']['r']}")
            break

will produce the same results as

import gym
from gym.vector import SyncVectorEnv

def make_env(gym_id, seed):
    def thunk():
        env = gym.make(gym_id)
        env.seed(seed)
        env.action_space.seed(seed)
        env.observation_space.seed(seed)
        return env
    return thunk

envs = SyncVectorEnv([make_env("CartPole-v1", 1 + i) for i in range(2)])
envs = gym.wrappers.RecordEpisodeStatistics(envs)
envs.reset()
for i in range(100):
    _, _, _, infos = envs.step(envs.action_space.sample())
    for info in infos:
        if "episode" in info.keys():
            print(f"i, episode_reward={info['episode']['r']}")
            break

The reason why this PR is important is that some envs don't allow you to insert a RecordEpisodeStatistics in the make_env function. The procgen environment is one such example. So this PR will allow you to do something like this:

import gym
from procgen import ProcgenEnv

envs = ProcgenEnv(num_envs=2, env_name="coinrun", num_levels=0, start_level=0, distribution_mode='hard')
envs = gym.wrappers.RecordEpisodeStatistics(envs)
envs.reset()
for i in range(100):
    _, _, _, infos = envs.step(envs.action_space.sample())
    for info in infos:
        if "episode" in info.keys():
            print(f"i, episode_reward={info['episode']['r']}")
            break

jkterry1 · 2021-08-05T03:56:25Z

Could you please add tests before I merge?

vwxyzjn · 2021-08-05T04:15:24Z

@jkterry1 Could you approve workflows?

vwxyzjn · 2021-08-05T04:22:06Z

Hey @jkterry1 can you approve the workflows again

tristandeleu · 2021-08-05T17:42:40Z

gym/wrappers/record_episode_statistics.py

-        self.episode_length = 0
-        return observation
+        observations = super(RecordEpisodeStatistics, self).reset(**kwargs)
+        self.episode_returns = np.zeros(self.num_envs, dtype=np.float32)


self.num_envs is not defined here if env is a VectorEnv instance.

I don’t follow. VectorEnv has a num_envs attribute, right?

Oh right the wrapper inherits the properties from env, sorry my bad!

gym/wrappers/record_episode_statistics.py

tristandeleu · 2021-08-05T17:55:41Z

gym/wrappers/test_record_episode_statistics.py

+@pytest.mark.parametrize("env_id", ["CartPole-v0"])
+def test_record_episode_statistics_with_vectorenv(env_id):
+    envs = gym.vector.make(env_id, asynchronous=False)


With the corresponding imports

from gym.vector import AsyncVectorEnv, SyncVectorEnv from gym.vector.tests.utils import make_env

Suggested change

@pytest.mark.parametrize("env_id", ["CartPole-v0"])

def test_record_episode_statistics_with_vectorenv(env_id):

envs = gym.vector.make(env_id, asynchronous=False)

@pytest.mark.parametrize("klass", [SyncVectorEnv, AsyncVectorEnv])

@pytest.mark.parametrize("num_envs", [1, 4])

def test_record_episode_statistics_with_vectorenv(klass, num_envs):

env_fns = [make_env("CartPole-v0", i) for i in range(num_envs)]

envs = klass(env_fns)

Unfortunately it’s gonna fail with AsyncVectorEnv because the envs.env.envs[0].spec.max_episode_steps is inaccessible. Maybe I should just hardcore a 201 instead of envs.env.envs[0].spec.max_episode_steps? Do we really need the test case with AsyncVectorEnv?

Oh sorry I didn't see that you were looping over that later. Then you can ignore this (maybe keeping the parametrization for num_envs?).

Done. Would you mind Allowing the GitHub action workflow runs? I have some weird setup That makes it difficult to run test cases locally….

Co-authored-by: Tristan Deleu <[email protected]>

vwxyzjn

Thank you @tristandeleu so much for the review. I just have one comment regarding the test cases.

vwxyzjn · 2021-08-05T18:02:49Z

gym/wrappers/test_record_episode_statistics.py

+@pytest.mark.parametrize("env_id", ["CartPole-v0"])
+def test_record_episode_statistics_with_vectorenv(env_id):
+    envs = gym.vector.make(env_id, asynchronous=False)


Unfortunately it’s gonna fail with AsyncVectorEnv because the envs.env.envs[0].spec.max_episode_steps is inaccessible. Maybe I should just hardcore a 201 instead of envs.env.envs[0].spec.max_episode_steps? Do we really need the test case with AsyncVectorEnv?

vwxyzjn · 2021-08-05T18:04:49Z

gym/wrappers/record_episode_statistics.py

-        self.episode_length = 0
-        return observation
+        observations = super(RecordEpisodeStatistics, self).reset(**kwargs)
+        self.episode_returns = np.zeros(self.num_envs, dtype=np.float32)


I don’t follow. VectorEnv has a num_envs attribute, right?

…s://github.com/vwxyzjn/gym into make-RecordEpisodeStatistics-work-with-vec-env

* Make RecordEpisodeStatistics work with VectorEnv * fix test cases * fix lint * add test cases * fix linting * fix tests * fix test cases... * Update gym/wrappers/record_episode_statistics.py Co-authored-by: Tristan Deleu <[email protected]> * fix test cases * fix test cases again Co-authored-by: Tristan Deleu <[email protected]>

Make RecordEpisodeStatistics work with VectorEnv

edd8b72

vwxyzjn added 2 commits August 5, 2021 00:13

fix test cases

1419822

fix lint

aed31bf

vwxyzjn added 2 commits August 5, 2021 00:21

add test cases

0f25b42

fix linting

225e130

vwxyzjn added 2 commits August 5, 2021 00:56

fix tests

d48114b

fix test cases...

a3e60dd

tristandeleu reviewed Aug 5, 2021

View reviewed changes

Update gym/wrappers/record_episode_statistics.py

e6df452

Co-authored-by: Tristan Deleu <[email protected]>

vwxyzjn commented Aug 5, 2021

View reviewed changes

vwxyzjn added 3 commits August 5, 2021 14:26

fix test cases

c97d946

Merge branch 'make-RecordEpisodeStatistics-work-with-vec-env' of http…

d51328c

…s://github.com/vwxyzjn/gym into make-RecordEpisodeStatistics-work-with-vec-env

fix test cases again

72ac78d

jkterry1 merged commit 1397e70 into openai:master Aug 5, 2021

This was referenced Aug 24, 2021

[Do not merge] Test against gym master ray-project/ray#17829

Closed

Note- Gym is deprecating the monitor wrapper in the next release DLR-RM/stable-baselines3#551

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make RecordEpisodeStatistics work with VectorEnv #2296

Make RecordEpisodeStatistics work with VectorEnv #2296

vwxyzjn commented Aug 5, 2021 •

edited

Loading

jkterry1 commented Aug 5, 2021

vwxyzjn commented Aug 5, 2021

vwxyzjn commented Aug 5, 2021

tristandeleu Aug 5, 2021

vwxyzjn Aug 5, 2021

tristandeleu Aug 5, 2021

tristandeleu Aug 5, 2021

vwxyzjn Aug 5, 2021 •

edited

Loading

tristandeleu Aug 5, 2021

vwxyzjn Aug 5, 2021

tristandeleu Aug 5, 2021

vwxyzjn left a comment

vwxyzjn Aug 5, 2021 •

edited

Loading

vwxyzjn Aug 5, 2021

-@pytest.mark.parametrize("env_id", ["CartPole-v0"])
-def test_record_episode_statistics_with_vectorenv(env_id):
-    envs = gym.vector.make(env_id, asynchronous=False)
+@pytest.mark.parametrize("klass", [SyncVectorEnv, AsyncVectorEnv])
+@pytest.mark.parametrize("num_envs", [1, 4])
+def test_record_episode_statistics_with_vectorenv(klass, num_envs):
+    env_fns = [make_env("CartPole-v0", i) for i in range(num_envs)]
+    envs = klass(env_fns)

Make RecordEpisodeStatistics work with VectorEnv #2296

Make RecordEpisodeStatistics work with VectorEnv #2296

Conversation

vwxyzjn commented Aug 5, 2021 • edited Loading

jkterry1 commented Aug 5, 2021

vwxyzjn commented Aug 5, 2021

vwxyzjn commented Aug 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vwxyzjn Aug 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vwxyzjn left a comment

Choose a reason for hiding this comment

vwxyzjn Aug 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vwxyzjn commented Aug 5, 2021 •

edited

Loading

vwxyzjn Aug 5, 2021 •

edited

Loading

vwxyzjn Aug 5, 2021 •

edited

Loading