Get rendering into SceneGraph

[SeanCurtis-TRI]

Rendering should ultimately belong in SceneGraph (it is, after all, an operation on geometry -- the query is: given camera properties, what is visible?) Currently, it is embodied in the RgbdRenderer in drake/sensors. This issue tracks the work to move it from there into SceneGraph (changing things to conform to the more general SceneGraph paradigm).

The following list of anticipated PRs will track the progress for moving rendering into SceneGraph (the list may evolve if it turns out any given PR is too large or too small):

• Clean up on SceneGraph's InternalFrame class. This is largely just some cosmetic stuff that should get done before it smears all over the new changes. (Clean up InternalFrame interface and docs #9541)
• Refactor of frames and geometries; remove the public-facing API that distinguishes between "anchored" and "dynamic" geometries and introducing, formally, the world frame in SceneGraph. (Refactor internal geometry classification #9570)
• Introduce the underlying infrastructure for geometry roles (i.e., specifying what purpose(s) a geometry exists for. (Add basis of geometry roles #9813)
• Wire geometry roles into scene graph internally (do not change the public API) (Connecting geometry roles *strictly* internally. #10044)
• Update the public API so that geometry roles are exposed (Proved to be quite large.)
  • Introduce parallel API (Extend public API to provide ability to assign roles. #10134)
  • Modify call sites from old API to new API 1/2 (Modify multibody code to use geometry-role-based interface #10165)
  • Modify call sites from old API to new API 2/2 (Change non-MBP-dependendent geometry usage to explicitly use roles #10164)
  • Remove old API ([geometry] Collapse Role API - no roles implicity assigned #10233)
• Shuffle some geometry utilities around in support of render engine. (Update geometry utilities #10344)
• Add RenderEngine abstraction to geometry (no implementation yet)
  • RenderEngine pre-requisites (indices, etc.) (Introduce RenderEngine prerequisites #10351) (Introduce preliminary elements for the RenderEngine #11043)
  • Add RenderLabel infrastructure (Add RenderLabel and RenderLabel manager #11042)
  • RenderEngine (without labels) and tests. (Add the abstract RenderEngine #11216)
  • Add label image to RenderEngine (Adds label images to the RenderEngine #11331)
• Add simple VTK OpenGL RenderEngine implementation.
  • Add implementation ([render] Implementation of VTK-based RenderEngine #11463)
  • Add tests ([render] Add the unit tests for RenderEngineVtk  #11518)
  • Missing legacy "texture-for-mesh" functionality ([render] Mainstream RenderEngineVtk missing default texture behavior #11545)
• Expose rendering as part of the SceneGraph API.
  • Clean up of geometry_state_test.cc in anticipation of expansion (Clean up pass on geometry state tests #11546)
  • Introduce rendering and perception role into GeometryState ([render] Add RenderEngine and PerceptionRole to GeometryState #11553)
  • Add role removal to GeometryState (GeometryState adds role removal API. #11572)
  • Incidental clean up ([render] Patch a number of gaps in previous PRs preparatory to public API #11597)
  • Add full rendering API all the way through SceneGraph and QueryObject ([render] Expose rendering API through QueryObejct/SceneGraph #11593)
• Add SceneGraph-compatible RgbdSensor - the modern equivalent to the attic'd RgbdCamera. (Introduce RgbdSensor #11704)
• Update python bindings (Python bindings for mainstream geometry render artifacts #11825)
• Visual geometry registered through MBP has both illustration and perception roles (short term patch to get equivalent behavior to what has been exploited using dev/geometry) (MBP "visual" geometry has both illustration and perception roles #11824)
• Allow adding renderer after registering geometry (Allow adding render engine after registering geometry #11820)
• Remove dependencies in Drake on dev/geometry
  • Remove dependency on dev/geometry from ManipulationStation (Remove dev/geometry dependencies from ManipulationStation #11839)
  • Remove dependency on dev/geometry everywhere else (Remove dev Geometry from scene graph examples #11829)
• Add ImageWriter - a class for writing Drake Images (as produced by RgbdCamera) to the disk. (Add ImageWriter #9882)
• Remove geometry/dev. (Complete removal of rendering dev artifacts #11859)
• Add and wire in OSPRay renderer into SceneGraph. Moved into its own issue ([render] Create RenderEngineOspray implementation #11744)

/cc @kunimatsu-tri , @thduynguyen , @DamrongGuoy, @sherm1

[jwnimmer-tri]

I'm not sure exactly what physical design is proposed here, but I'd like to push us toward "pay for only what you need". Would the VTK, OSPRay be dependencies of SceneGraph, or would SceneGraph merely provide enough outputs or data to allow the user to attach a renderer, in cases where its needed? Since SceneGraph is such a core component of almost any simulation moving forward, I would prefer that it never hard-depend on any rendering engine, since they are finicky.

[SeanCurtis-TRI]

Short answer

This branch contains all but the last step above (https://github.com/SeanCurtis-TRI/drake/tree/gw_renderer) -- although not organized in the ordering indicated above. I'm currently decomposing it.

Long answer

So, the RgbdRender would die. It depends on RBT. A renderer has to contain the geometry registered by arbitrary sources. So, the equivalent to the renderer would move into the geometry folder. My design includes the following features:

• SceneGraph would ultimately own three different renderers (which I term low, medium, and high fidelity, corresponding to simple opengl, PBR OpenGL, and path tracing).
• RgbdCamera basically contains enough to define its intrinsic and extrinsic matrices and provides those values to the geometry::QueryObject and gets an image back (which it puts on its output port).
• Arbitrary cameras can be configured to use arbitrary rendering fidelity.
• Only those geometries that have been given the "perception" role would be registered with the various rendering engines.
• One rendering engine is shared across all cameras.

Speculative features

• Add which rendering engines any particular SceneGraph should support an construction (i.e., if you know you're only ever doing one type of fidelity, you'd like to not have to pay any runtime cost for the other, unused renderers.)

[jwnimmer-tri]

I think we'll have to explore some tweaks before we get to the rendering engine integration. I don't think its acceptable for anyone who wants to use, say, MultibodyPlant, to have to always link in VTK or OSPRay. In their own ecosystem, they may have incompatible version of those dependencies. If they were trying to use vision simulation, perhaps we can insist that they consume our VTK. If they are just doing some collision queries for a dynamics simulation, though, they should not be tainted by spurious physical dependencies on a rendering engine. Hopefully there is a way to carve out the rendering-engine dependencies into add-on classes, even if there is still a 1:1 relation from an engine to a SceneGraph and they are highly coupled with internal APIs.

[SeanCurtis-TRI]

I'm definitely sympathetic with that concern and am fully in favor of resolving it.

Would I be correct in my belief that the current rendering architecture has the same problem? It seems so to me.

[jwnimmer-tri]

On quick glance, RgbdCamera has a hard-dep on VTK, but it's decoupled enough that we could sever it if we wanted to without too much work. I don't think it spuriously bring in OSPRay though? In any case, geometry uses other than vision simulation don't bring in VTK spuriously.

[SeanCurtis-TRI]

The current branch has a similar layer of abstraction. SceneGraph only knows about the RenderEngine interface (at least in its header). And all of the nasty vtk (and other future headers) are buried in the .cc files. So, it has a similar severability.

Off the top of my head, the only obvious solution I have is a constructor that takes rendering flags combined with the use of dlload. Is this what you have in mind?

[SeanCurtis-TRI]

One last question for you @jwnimmer-tri.

I'd like to parallelize some of this stuff. For example, the introduction of the SceneGraph-compatible version of the vtk renderer in the geometry space (removal of RBT dependency and generalization of rendering). However, it will propagate the hard dependencies. Do you feel strongly enough that we defer that task listed above until we have a chance to address dependency issues?

[EricCousineau-TRI]

Off the top of my head, the only obvious solution I have is a constructor that takes rendering flags combined with the use of dlload. Is this what you have in mind?

Can I ask why you suggest dload? It seems that there's enough of a runtime abstraction that sawing off VTK / OSPRay / Godot / whatever should be simple enough?

What if you just had an option to take in a RenderEngine* of nullptr? That way you don't pay the cost for using it if you don't care?

EDIT: I see that you have fidelity flags, and a low_render_engine_ field (and I'd assume other fields to correspond to other levels).

However, that seems a bit odd. Why not have some sort of open association via strings, such that users can choose what their renderers are? (e.g. std::map<string, RenderEngine>?)
This could permit easier tuning, perhaps?

And I haven't seen how these renderers would be specified yet (from briefly looking at the code); can I ask how you envision this happening?

[SeanCurtis-TRI]

Lots of different points:

1. why dload? This is predicated upon my understanding that unless I do dload, upon loading the drake library, the OS will look for the VTK library and throw a fit if it doesn't find what it's looking for. Whereas, by using dlload, it will only have a fit if we've actively chosen to load the library.

2. Fidelity flags and engines. Yep, today there's just the one, but in my mind we'll have three (two in the near future and hopefully three in the not-too-distant future.)

3. I don't understand your "open association" proposal. I need me you to walk me through the problem that you're solving so I have context to appreciate the solution you're proposing. Unfortunately, being "a bit odd" doesn't give me enough to work with. :)

4. Specifying renderers -- there are two separate issues:

• The question of which renderers are available at all in SceneGraph is does not appear anywhere in the branch; that's a new discussion in this issue. The plan in the branch is that all renderers are available all the time (although, I had already sketched out what it would take if you didn't want to pay the run-time memory cost of populating more than a single renderer -- simple and not a problem.)
• The question of how a camera picks a renderer is in the render methods. The CameraProperties takes the fidelity flag (so when you instantiate an RGBD camera, in addition to the normal camera things, you also say which renderer you want it to use.)

[RussTedrake]

i'll throw in a vote for bringing it in more quickly and breaking the linking dependencies in a follow-on, simply for the sake of expediency.

[jwnimmer-tri]

... breaking the linking dependencies in a follow-on ...

I am all for moving quickly, but if we are going to offer an "image_output_port" on SceneGraph, we need to be sure that it is feasible to update its implementation in the future to avoid the dependencies. (Like, I will do it immediately after image_output_port gets added to master.) We don't want to put ourselves in the position of having to remove the image_output_port from SceneGraph later on (and move it to a different System) as the only path to resolving the dependency issue.

Also, if MultibodyPlant starts to depend on VTK8, we will no longer be able to use MultibodyPlant in many places within Anzu. Some parts of Anzu use third-party libraries that depend on VTK6.

[jwnimmer-tri]

Why dload? This is predicated upon my understanding that unless I do dload, upon loading the drake library, the OS will look for the VTK library and throw a fit if it doesn't find what it's looking for. Whereas, by using dlload, it will only have a fit if we've actively chosen to load the library.

That libdrake.so will always depend on VTK is on par with the current situation. That is not my complaint. My complaint is users in the bazel_drake_external workflow (which includes TRI Anzu, as well as several research labs) who build Drake from source, need to be able to avoid the dependency. It's all about the deps = [] lines in BUILD files. If I only deps on SceneGraph, I don't want VTK to be linked in. If I then also deps on VtkRenderingEngine and do SceneGraph::AddRenderer(make_unique<VtkRenderingEngine>) I've declared my desire for VTK and thus I depend on it and can use it. Different of my applications of MultibodyPlant / SceneGraph can make different build-time and run-time choices.

Separately, I also still in general support libdrake.so being sliced into smaller pieces to curate each one's dependencies, but that is not directly related here. If we have the logical class design and physical component factoring right (as proven out by drake_bazel_external use case), we can still factor libdrake.so installation at a future point.

[SeanCurtis-TRI]

Ok, I think my mental model is growing to encompass what you have in mind.

1. If I'm a "bazel_drake_external" workflow user, I can set up my project to reference specific bazel targets. The only thing my binary will depend on (in the sense of what the OS is going to look for upon loading) is the exact tree of dependencies that my declared dependencies imply. Therefore, if the scene_graph target, directly or indirectly, has a dependency on VTK, the external user is stuck.
2. However, if SceneGraph has a RenderEngine dependency (which is just an interface). And each render engine implementation is its own target, the end user can elect to also be dependent on any of the engine implementations they like, a la carte.

To summarize, the targeted workflow has the flavor that it allows end users to create a custom slice of all of Drake to be the libdrake.so and we want to give them a reasonably fine-grain ability to slice.

Do I have that basically right?

[jwnimmer-tri]

Yes, that's the gist of it.

The only detail to nitpick is that bazel_drake_external users don't use anything akin to libdrake.so -- they are (mostly) just linking all of the *.o files directly into their applications (drake/geometry/scene_graph.o, etc.). (Yes, that means the linker is fed a long list of files.)

[RussTedrake]

I also think that factoring libdrake.so has to be in our roadmap soon. Looks like we have #7294 to track it.

[SeanCurtis-TRI]

RenderEngine abstraction got stopped a few weeks ago -- requests for a design change on how RenderLabels are managed led to a rearchitecture that was interrupted by other issues. The RenderLabel rearchitecture is just about ready for a PR, so the chain will continue in the next day or so.

[SeanCurtis-TRI]

Adding RenderLabel into the abstract RenderEngine currently under review.

[SeanCurtis-TRI]

Expect RenderLabel/RenderEngine (#11331) will merge today; RenderEngineVtk will enter review today.

[SeanCurtis-TRI]

RenderEngineVtk (1/2) PR 11477 under slow review. Initial reviewer proved to be unavailable; it was reassigned. This PR doesn't have unit tests; they'll come in the immediate follow up PR.

[SeanCurtis-TRI]

Missing render engine feature added (#11545). Began 4-PR chain to get rendering exposed all the way through the SceneGraph/QueryObject API. Updated tasks accordingly.

[jwnimmer-tri]

Where is the current WIP branch for this feature? It doesn't seem to be gw_renderer anymore. Per the physical design discussion upthread, I would like to see how "Add full rendering API all the way through SceneGraph and QueryObject" affects the BUILD file dependencies.

[SeanCurtis-TRI]

Yeah -- sorry about that. That was the "raw" branch. This is the one that's been maintained and is being curated into commits: https://github.com/SeanCurtis-TRI/drake/tree/gw_renderer_remaining. Specifically, commit cd3f06b. (Edit: Sorry, the specific commit, while representative, is not reliable; I'm constantly rebasing. However, the commit message should be pretty obvious. the next three PRs I submit will directly expose rendering through the SG/QO interface. I think you'll find it benign.)

[jwnimmer-tri]

I think you'll find it benign.

Yes indeed, I confirm it avoids having MultibodyPlant depend on VTK.

It adds more instances of package dependency cycles (geometry vs geometry/render vs systems/sensors vs etc.), but those were already there. I am holding off identifying and proposing how to fix those until #10675 is resolved.

[SeanCurtis-TRI]

Sounds good.

[SeanCurtis-TRI]

The 4-PR chain to integrate the public API is in progress. FIrst merged, second under platform review. Two subsequent PRs are waiting in the wings.

[SeanCurtis-TRI]

The RgbdSensor PR is stalled to incorporate updates in terminology started in #11337.

[SeanCurtis-TRI]

The last box has been checked. Finally closing!