Contruct a model programatically

[BenjaminNavarro]

Is your feature request related to a problem? Please describe.
Right now it seems that mujoco can only load a model from a file (mjb/urdf) and that there is no way to construct a model just using API calls.

This limits me in two ways.

First, in my work things are highly flexible and have a model file matching each configuration is doable but now always practical. I can always turn my internal representation of the scene into a urdf and pass that to mujoco but it would be much simple if I could just call the API to add bodies, joints, etc.

The other, I would say bigger, issue is to add things dynamically to the model. For instance a user wants to test his controller in an evolving environment, it might be tempted to add some objects to the scene while the simulation is running. As far as I know this is not possible with the current API without creating and loading a complete new model, which can cause issues as the already existing objects state won't be preserved. You can always try to prepare a bunch of objects that you hide somewhere to circumvent the problem but this is not super ergonomic nor efficient as these objects have to be simulated even when not in use.

Describe the solution you'd like
An API that allows me to create the scene I want without having to go through an intermediate text format and being able to alter the scene in real time using the same API.

Describe alternatives you've considered
Using a different simulator but loosing the performance of mujoco

Additional context
Everything becomes more and more dynamic in robotics and being able to handle such dynamic scenarios seems very important to me in order to stay relevant.

[kevinzakka]

FYI, you can currently do this with dm_contol's PyMJCF.

[BenjaminNavarro]

Ok but I was speaking about the native C/C++ API, not the Python wrapper. I should have made myself clear on that.

[yuvaltassa]

Hi,

The API that you are asking for already exists. It is the model compiler which lives in src/user/.

In fact composite bodies are entirely implemented using this API.

You are very welcome to use this API yourself as you see fit, however:

1. The reason we did not document it is that we consider it unstable. We intend to make significant upgrades to the compiler (specifically targeting better model composability) and therefore make no API stability guarantees.
2. For the same reason, we can only offer best-effort support in this regard. That said, if you can find your way around the compiler and form constructive opinions we will be happy to consider you a partner in improving this important part of MuJoCo.

[saran-t]

@yuvaltassa let's keep this issue open and use it to track any API discussion we might want to have with the community around model compilation/recompilation (and also if OP has any question while trying to use src/user/, although as Yuval said we don't intend to provide very active support around the existing API at this time)

[BenjaminNavarro]

@yuvaltassa Ok thanks, yes I didn't know this API existed since it's not mentioned in the docs. Maybe you could just add a few sentences in the docs to mention it with the current caveats, and maybe point to this issue if users want to give feedback on it?

Anyway, I'll try it soon and let you know what I think about it.

[caiqi]

@yuvaltassa I am aslo interested in dynamically adding new objects into the scenes while simulating. For example, at each step, place a new box into the scenes while keeping the states of existing boxes. I have tried PyMJCF but it seems that PyMJCF can add new objects to the mjcf model but these new added objects are not binded to the physics. That said, I have to use physics.reload_from_mjcf_model(mjcf_model) every time I modify the mjcf_model. I found that physics.reload_from_mjcf_model is time consuming(~100ms on Colab). So is it possible to add objects to current simulation at runtime without reload(recompile?) the model?

Here is a MWE:

arena = mjcf.RootElement()
arena.worldbody.add('geom', type='box', size=[0.5, 0.2, .1],rgba=[0,1,0, 1] )
physics = mjcf.Physics.from_mjcf_model(arena)
initial = PIL.Image.fromarray(physics.render())

arena.worldbody.add('geom', type='box', pos=[0.5, 0.2, 0.3], size=[0.5, 0.2, .1],rgba=[1,0,1, 1] )
second = PIL.Image.fromarray(physics.render())

physics.reload_from_mjcf_model(arena)

third = PIL.Image.fromarray(physics.render())

plt.subplot(1,3,1)
plt.imshow(initial)
plt.subplot(1,3,2)
plt.imshow(second)
plt.subplot(1,3,3)
plt.imshow(third)
plt.show()

[yuvaltassa]

Hi @caiqi,

Currently PyMJCF is the only convenient way to do this from Python, and indeed it is not very fast. As I said above, the new and improved compiler is on our roadmap and in the future will be the recommended way to do this.

Until then if you are working from Python, pyMJCF is your best bet.

[caiqi]

@yuvaltassa Thanks, I will have a try!

[joaopfg]

@BenjaminNavarro Did you make progress on this ?

@yuvaltassa @saran-t As far as I see, composite elements are discharged during the parsing. The path is Parse => Body => OneComposite => Make. The compiler doesn't seem to mention composites. I guess once they are discharged by the parser, the compiler doesn't need to know about them. So, I don't get your point about using the API used for composites creation to create bodies programmatically in an already loaded scene. Could you elaborate, please ?

The idea I have for now is to modify mj_loadXML method to allow re-load of the scene with stuff passed at runtime.

So, instead of doing:

1. mjCModel creation from mjParseXML.
2. mjCModel compilation into mjModel.

It would do:

1. mjCModel creation from mjParseXML.
2. Add custom stuff to mjCModel from data that I pass at runtime.
3. mjCModel compilation into mjModel.

The drawback of this is that I'm not really loading new stuff in a scene that is already loaded. Instead, I'm re-loading all the scene and adding my new stuff in between. So, there are obvious performance issues.

How would you suggest doing that while the new compiler is not ready yet ?

[BenjaminNavarro]

Sorry for the lack of feedback but yes, I managed to construct my scenes programmatically using the APIs in src/user.

I had to look at the parsers code and tinker a bit to understand how things should be constructed, but that can be expected since this is not a public API. Being used to URDF-like robot descriptions I struggled a bit at the beginning with the Mujoco way of first creating a tree of bodies and then attaching joints to them instead of the more traditional link->joint->link pattern but that's ok.

So I would be in favor of solving #388 and documenting this API in order for people like me to be able to build their scenes using code instead of description files where it makes sense.

If someone is interested, the code I wrote for this is here.

[joaopfg]

Thanks, @BenjaminNavarro ! I will give a look

@yuvaltassa Is there a fixed schedule to complete the upgrades in the compiler that you mentioned ? Will it arrive this year ?

[yuvaltassa]

Hi @joaopfg,

We're currently in the design phase. So definitely not this year, but H1 2023 seems not unlikely.

[Roger-Qea]

Hello, I am considering to use mujoco as the physics engine in a unity game, so I am also interesting in this issue. It is very important to create and destroy objects fastly for a game. (For example, when a gun shoots bullets in a game, a lot of objects need to be created in a short time; when these bullets hit a target, they need to be destoried quickly).
I know there is an official unity plugin, which seems can create new objects at runtime. But I have not dived into its source code to see how the new objects are constructed. Is it implemented in a fast new way? Can I use Mujoco for normal games?

[zichunxx]

I'm trying to implement the dynamic loading of XML via PyMJCF. Here are some problems I'm experiencing, I'm not sure if anyone else has met them.

Problems:

The XML document can be loaded successfully via mujoco.MjModel.from_xml_path(), but encounters a series of path parsing and tag-related issues when using mjcf.from_path(), e.g., this issue. The official tutorial focuses on introducing creating the mjcf_model from scratch, but it doesn't seem to describe how to dynamically change the model after loading some complex XML files (Please correct me if I missed it).

What I want and have tried

I want to load the XML model with Mujoco Python binding and then add or delete some geoms dynamically. I have tried some methods to realize my purpose but failed. Is it possible?

For now, the alternative approach is to load extra bodies in the XML and "hide" them (e.g. by placing them really far away from the rest of the model), then move them into the scene when they are needed, which is inspired by this comment.

Any advice would be appreciated.

Thanks.

[sangteak601]

Hi @yuvaltassa,
Is there any update on this compiler upgrades ?

[yuvaltassa]

We've completed the design phase, starting work H1 2024.

[hang0610]

I'm trying to implement the dynamic loading of XML via PyMJCF. Here are some problems I'm experiencing, I'm not sure if anyone else has met them.

Problems:

The XML document can be loaded successfully via mujoco.MjModel.from_xml_path(), but encounters a series of path parsing and tag-related issues when using mjcf.from_path(), e.g., this issue. The official tutorial focuses on introducing creating the mjcf_model from scratch, but it doesn't seem to describe how to dynamically change the model after loading some complex XML files (Please correct me if I missed it).

What I want and have tried

I want to load the XML model with Mujoco Python binding and then add or delete some geoms dynamically. I have tried some methods to realize my purpose but failed. Is it possible?

For now, the alternative approach is to load extra bodies in the XML and "hide" them (e.g. by placing them really far away from the rest of the model), then move them into the scene when they are needed, which is inspired by this comment.

Any advice would be appreciated.

Thanks.

Hi. I have the same problem when trying to dynamically modify the model in Mujoco with the Python Bindings (i.e. the APIs from import mujoco). Besides, It seems impossible to use the model constructed or modified by dm_control.mjcf with Mujoco Python Bindings as I can't find a proper way to transform the constructed model type to mujoco._structs.MjModel. Any advice?

Will the upcoming compiler upgrade solve this problem ? @yuvaltassa Thanks :)

[yuvaltassa]

The public compiler API (basically a native replacement for PyMJCF) will indeed solve all of this and work is progressing quickly now. Updates soon 🙂

[JewelryForge]

Hi @yuvaltassa,
Is there any progress or schedule for when this compiler will be released?

[yuvaltassa]

Very close! You can read the (currently private) API here. Should be in the next release or the one after (1-2 months)

[tudorjnu]

Hello @yuvaltassa! Has this been finalized? If so, is there any python example? Thank you!

[yuvaltassa]

Not finalized, but pretty close. If you can build from source look at specs_test.py, otherwise wait for the release (next week). By then there will also be better docs.