[CI] Fix Minari tests #2419

younik · 2024-09-04T13:05:54Z

Description

With Minari 0.5.0 release, we regrouped dependencies to install only the required ones depending on user usage.
Moreover, we restructured the datasets to be hierarchical. This caused the torch/rl tests to fail; this PR should fix them.

I changed the tests to rely on Minari for getting the last version of the datasets.

Also, Minari 0.5.0 adds support for Arrow datasets, I am happy to integrate it in torch/rl.

Motivation and Context

Fixes failing CI due to new version of Minari.

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

I have read the CONTRIBUTION guide (required)
My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

pytorch-bot · 2024-09-04T13:05:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2419

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 14 Unrelated Failures

As of commit cde51e9 with merge base 0326c41 ():

NEW FAILURES - The following jobs have failed:

Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
RuntimeError: Command docker exec -t cc89772380fda850192e3c442d90493301e35f9763e8d232490d0bc26665a612 /exec failed with exit code 134
Lint / c-source / linux-job (gh)
Lint / python-source-and-configs / linux-job (gh)
Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
test/test_exploration.py::TestConsistentDropout::test_consistent_dropout[device0-True-0.5]
Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
test/test_distributed.py::TestRayCollector::test_distributed_collector_updatepolicy[False-MultiSyncDataCollector]
Wheels / build-wheel-windows (3.10, 3.10.3) (gh)
Wheels / build-wheel-windows (3.11, 3.11) (gh)
Wheels / build-wheel-windows (3.12, 3.12) (gh)
Wheels / build-wheel-windows (3.9, 3.9) (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

Build Aarch64 Linux Wheels / pytorch/rl (pytorch/rl, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cpu-aarch64 (gh) (detected as infra flaky with no log or failing log classifier)
Build Aarch64 Linux Wheels / pytorch/rl (pytorch/rl, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda-aarch64cuda-aarch64 (gh) (detected as infra flaky with no log or failing log classifier)
Continuous Benchmark (PR) / CPU Pytest benchmark (gh) (detected as infra flaky with no log or failing log classifier)
Continuous Benchmark (PR) / GPU Pytest benchmark (gh) (detected as infra flaky with no log or failing log classifier)
Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
Unit-tests on Linux / tests-cpu-oldget (3.12) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vmoens

LGTM thanks for owning this!

vmoens · 2024-09-04T14:04:31Z

We now have multiple. FAILED test/test_libs.py::TestMinari::test_load[D4RL/door/human-v2-True] - ModuleNotFoundError: No module named 'gymnasium_robotics' in the CI.

The linter (pre-commit) is also unhappy.

rl/CONTRIBUTING.md

Lines 34 to 52 in 0326c41

    
           ## Formatting your code 
        
           **Type annotation** 
        
           TorchRL is not strongly-typed, i.e. we do not enforce type hints, neither do we check that the ones that are present are valid. We rely on type hints purely for documentary purposes. Although this might change in the future, there is currently no need for this to be enforced at the moment. 
        
           **Linting** 
        
           Before your PR is ready, you'll probably want your code to be checked. This can be done easily by installing 
        
           ``` 
        
           pip install pre-commit 
        
           ``` 
        
           and running 
        
           ``` 
        
           pre-commit run --all-files 
        
           ``` 
        
           from within the torchrl cloned directory. 
        
           You can also install [pre-commit hooks](https://pre-commit.com/) (using `pre-commit install` 
        
           ). You can disable the check by appending `-n` to your commit command: `git commit -m <commit message> -n`

younik · 2024-09-04T16:59:44Z

We now have multiple. FAILED test/test_libs.py::TestMinari::test_load[D4RL/door/human-v2-True] - ModuleNotFoundError: No module named 'gymnasium_robotics' in the CI.

The linter (pre-commit) is also unhappy.

rl/CONTRIBUTING.md

Lines 34 to 52 in 0326c41

## Formatting your code

**Type annotation**

TorchRL is not strongly-typed, i.e. we do not enforce type hints, neither do we check that the ones that are present are valid. We rely on type hints purely for documentary purposes. Although this might change in the future, there is currently no need for this to be enforced at the moment.

**Linting**

Before your PR is ready, you'll probably want your code to be checked. This can be done easily by installing

```

pip install pre-commit

```

and running

```

pre-commit run --all-files

```

from within the torchrl cloned directory.

You can also install [pre-commit hooks](https://pre-commit.com/) (using `pre-commit install`

). You can disable the check by appending `-n` to your commit command: `git commit -m <commit message> -n`

Ups, my env was contaminated.
The dependency error is due to a line incorrectly unreachable in 0.4.3. However, throwing a dependency error during download is not the intended behavior, and I will fix it in Minari. I will release it this week.

younik · 2024-09-06T20:22:29Z

After further analysis, the issue was with remote datasets lacking spaces, causing Minari to fall back to env_spec. This has been fixed. Could you please re-run the Minari tests in CI? @vmoens

vmoens · 2024-09-10T07:56:20Z

There seems to be another error in the CI now:
For one dataset, we use a transform that expects a nested ("observation", "observation") entry but "observation" is already a leaf (so there can't be any nesting). Do you know what is happening?

younik · 2024-09-12T10:51:32Z

test/test_libs.py

-    # We rely on sorting the keys as v0 < v1 but if the version is greater than 9 this won't work
-    total_keys = sorted(minari.list_remote_datasets())
-    assert not any(
-        key[-2:] == "10" for key in total_keys
-    ), "You should adapt the Minari test scripts as some dataset have a version >= 10 and sorting will fail."
-    total_keys_splits = [key.split("-") for key in total_keys]
+    total_keys = sorted(
+        minari.list_remote_datasets(latest_version=True, compatible_minari_version=True)
+    )
    indices = torch.randperm(len(total_keys))[:20]
    keys = [total_keys[idx] for idx in indices]
-    keys = [
-        key
-        for key in keys
-        if "=0.4" in minari.list_remote_datasets()[key]["minari_version"]
-    ]
-
-    def _replace_with_max(key):
-        key_split = key.split("-")
-        same_entries = (
-            torch.tensor(
-                [total_key[:-1] == key_split[:-1] for total_key in total_keys_splits]
-            )
-            .nonzero()
-            .squeeze()
-            .tolist()
-        )
-        last_same_entry = same_entries[-1]
-        return total_keys[last_same_entry]
-
-    keys = [_replace_with_max(key) for key in keys]


I believed what changed is the order of _MINARI_DATASETS given the above modification.
The test work with kitchen or antmaze datasets as the observation space is a dictionary, while it doesn't work for pen datasets.
I can change the code to test only a kitchen dataset or to work with the pen one.

younik · 2024-09-15T10:45:44Z

I changed the preprocessing test to only use a dataset with Dict observation space.
I have also updated the kitchen datasets that have wrong info structure.

Fix Minari tests

2b2fa3d

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 4, 2024

vmoens added Environments Adds or modifies an environment wrapper Data Data-related PR, will launch data-related jobs labels Sep 4, 2024

Merge branch 'main' into main

643a86b

vmoens approved these changes Sep 4, 2024

View reviewed changes

fix linter

5cdc418

empty-commit

68a76bb

vmoens self-requested a review September 11, 2024 08:56

younik commented Sep 12, 2024

View reviewed changes

fix preproc test

cde51e9

vmoens merged commit 224d637 into pytorch:main Sep 17, 2024
51 of 76 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Fix Minari tests #2419

[CI] Fix Minari tests #2419

younik commented Sep 4, 2024 •

edited

Loading

pytorch-bot bot commented Sep 4, 2024 •

edited

Loading

vmoens left a comment

vmoens commented Sep 4, 2024

younik commented Sep 4, 2024

younik commented Sep 6, 2024

vmoens commented Sep 10, 2024

younik Sep 12, 2024 •

edited

Loading

younik commented Sep 15, 2024

[CI] Fix Minari tests #2419

[CI] Fix Minari tests #2419

Conversation

younik commented Sep 4, 2024 • edited Loading

Description

Motivation and Context

Types of changes

Checklist

pytorch-bot bot commented Sep 4, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2419

❌ 9 New Failures, 14 Unrelated Failures

vmoens left a comment

Choose a reason for hiding this comment

vmoens commented Sep 4, 2024

younik commented Sep 4, 2024

younik commented Sep 6, 2024

vmoens commented Sep 10, 2024

younik Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

younik commented Sep 15, 2024

younik commented Sep 4, 2024 •

edited

Loading

pytorch-bot bot commented Sep 4, 2024 •

edited

Loading

younik Sep 12, 2024 •

edited

Loading