direct mujo'co control ?

[lo-th]

Hi very cool libs
just for info
is possible to add or remove object on fly ?
or is only xml init scene .

[josephrocca]

Just gathering some relevant comments on this:

From the official Unity Plugin docs:

Because the MuJoCo library doesn’t (yet) expose an API for scene editing, adding and removing MuJoCo components causes complete scene recreation. This can be expensive for large models or if it happens frequently. We expect this performance limitation to be lifted in future versions of MuJoCo.

google-deepmind/dm_control#54 (comment) :

There are various hacky workarounds that people sometime use in order to avoid recompilation. For example you can define extra bodies/geoms etc. in the XML and "hide" them (e.g. by placing them really far away from the rest of the model). Then later on when they are needed you can move them into the scene.

google-deepmind/mujoco#364 (comment) :

Question: "I am also interested in dynamically adding new objects into the scenes while simulating. [...] I found that physics.reload_from_mjcf_model is time consuming(~100ms on Colab). So is it possible to add objects to current simulation at runtime without reload(recompile?) the model?"

Response: "Currently PyMJCF is the only convenient way to do this from Python, and indeed it is not very fast. As I said above, the new and improved compiler is on our roadmap and in the future will be the recommended way to do this. [...] We're currently in the design phase. So definitely not this year, but H1 2023 seems not unlikely."

So it seems like this might become possible soon.

(Also, the title of this issue should be changed to something more descriptive like "Possible to add/remove objects on the fly?")

[zalo]

It would basically require creating a copy of the current MuJoCo simulation state, but with the desired objects added or removed.

It's possible, but a bit of a pain the butt... it would need a function to manually step through all the objects in a scene, and correctly copy them and their secondary simulation elements (+ indices!) over.

Kind of like merging two meshes together into one mesh and keeping the triangle indices sorted, but with like a hundred indexed attributes....

The best idea might be to autogen this function given the metadata that is present in the Python Binding Generator... all the fields of mj_model and mj_state and their relative struct sizes are all present...

[josephrocca]

@zalo Did you see the quote in my comment that suggests that the MuJoCo team are working on supporting dynamically-added objects? Seems like it might just be worth waiting for them to implement that?

(btw, your comment brings to mind morphdom, but I haven't looked into it enough to know whether it works on arbitrary xml/html, or requires actual DOM object and browser API stuff - guessing the latter)

[zalo]

@zalo Did you see the quote in my comment that suggests that the MuJoCo team are working on supporting dynamically-added objects? Seems like it might just be worth waiting for them to implement that?

Yeah, but I'm not holding my breath for an H1 2023 release 😅

(btw, your comment brings to mind morphdom, but I haven't looked into it enough to know whether it works on arbitrary xml/html, or requires actual DOM object and browser API stuff - guessing the latter)

I'm a little averse to going through the XML representation, since (while it does dramatically simplify the implementation) it's way slower than one would expect for simple object addition/subtraction... and, while I'd love it if lo-th added MuJoCo to Phy, there are enough idiosyncrasies with the way MuJoCo does things, that I doubt that this would be a drop-in implementation...

Besides, he already has Havok and PhysX in there, which are extremely solid engines 😎
I'd expect MuJoCo to only be better at articulated bodies and soft-contact resolution... though, I'd be very curious to see how the Keva/Box Stacking demos compare...