Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

increasing memory consumption when using object_randomization #8

Open
rojas70 opened this issue Dec 7, 2021 · 1 comment
Open

increasing memory consumption when using object_randomization #8

rojas70 opened this issue Dec 7, 2021 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@rojas70
Copy link

rojas70 commented Dec 7, 2021

In picking.py,
When the self.object_randomization flag is set to True, _reset_internal() also activate the self.hard_reset flag on.

The flag is used in environments/base.py:reset().
When self.hard_reset is on, the following code is called starting on L311:

        if self.hard_reset and not self.deterministic_reset: #TODO: investigate increasing memory consumption when calling this every rollout
            self._destroy_viewer()
            self._load_model()              #  Create a manipulation task objec (arena/robot/object/placement of objects/goal objects)
            self._postprocess_model()
            self._initialize_sim()

When this is called at the end of every rollout, our memory usage increases significantly over the lifetime of the experiment, always resulting in an out of RAM segfault.

If we only randomize every 200 epochs we see this stabilizes, but we have not figured out the underlying cause.

@rojas70 rojas70 added the bug Something isn't working label Dec 7, 2021
@rojas70 rojas70 added this to the complete_script_no_her milestone Dec 7, 2021
@rojas70
Copy link
Author

rojas70 commented Jan 17, 2022

By only doing hard resets when ALL objects are picked up and not every reset, this number seems stable... will keep monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants