Add Metal support for the Ogre2 Render Engine

[srmainwaring]

Desired behaviour

Add support for the Ogre2 Metal render system when using the ogre2 render engine and allow the choice of render system to be controlled using a plugin parameter.

Alternatives considered

The alternative is to only support the GL3+ render system on all platforms. There is ongoing work to support macOS using GL3+. The proposal is that this continue in parallel for perhaps the next couple of release cycles.

Implementation suggestion

Ogre2 already supports Metal for HlmsPbs and HmlsUnlit, so the major work needed for the core material system has been done.

Outline of the tasks to implement:

• Add a plugin parameter to select the render system
• Allow the Ogre2RenderEngine to select the either GL3+ or Metal render systems
• Set Metal configuration and add Metal resource folders when initialising the render engine
• Write Metal versions of the shaders in ign-rendering/tree/ign-rendering5/ogre2/src/media/materials/programs and update the materials to automatically select the correct shader for the render system
• Update examples and demos as required

Additional context

Apple have deprecated OpenGL for macOS. While OpenGL is still available on macOS Big Sur 11.4 it is becoming increasingly difficult to support on that platform. Upstream libraries such as Ogre2 are focussing on Metal and OpenGL > 4.2 and support for legacy OpenGL will become progressively more time consuming as these libraries advance.

macOS no longer has good tools for debugging OpenGL programs: tools such as OpenGL Profiler while available with Xcode 12.5 are no longer working correctly, alternatives such as Renderdoc are not supported on macOS. On the other hand there is very good support for debugging Metal shaders from within Xcode. This may be of benefit when developing and testing new shaders which could then be ported to OpenGL from Metal.

Links to related discussions

• https://community.gazebosim.org/t/ogre-2-support-on-macos-in-ignition-fortress/912
• Examples not working on macOS with Ogre2.2 #422 (comment)

Detailed tasks

Set up Metal dev environment

• Add script to build and install libraries on macOS using same conventions as osrf/simulation/ogre2.2.rb.
• Set up Xcode projects for ign-rendering, simple_demo and simple_demo_qml. Check build and debug.

Load Metal render system

• Add parameter to enable Metal.
• Allow Ogre2RenderEngine to select GL3+ or Metal render system.
• Set Metal configuration and add Metal resource folders when initialising the render engine.

Port shaders to Metal

• depth_camera_final
• depth_camera
• gaussian_noise_depth
• gaussian_noise
• gpu_rays_1st_pass
• gpu_rays_2nd_pass
• heat_signature
• plain_color
• point
• selection_buffer
• skybox
• thermal_camera

Update examples

• Update examples and demos to be render system agnostic.

[srmainwaring]

Using Metal in the ign-rendering examples

The examples have been modified to allow the Metal render system to be set using an extra command line option:

./simple_demo ogre2 metal

The metal option is only active when the option ogre2 is specified.

Status

Running with Metal, correct behaviour

• actor_animation
• camera_tracking
• custom_scene_viewer
• heightmap
• lidar_visual
• mesh_viewer
• mouse_picking
• ogre2_demo
• particles_demo
• render_pass
• segmentation_camera
• simple_demo
• transform_control
• view_control
• visualization_demo

Running with Metal, incorrect behaviour

• simple_demo_qml (scene not rendering as relies on shared FBO with QML)
• thermal_camera (check whether table should be visible)

Not working with Metal

Not yet ported

• hello_world_plugin
• gazebo_scene_viewer

Not suitable for Metal

• custom_shaders (*)
• custom_shaders_uniforms (*)
• text_geom (*)

(*) Example is for ogre render engine only.

[srmainwaring]

@iche033 I think the Metal support is nearly there. The main remaining issue is with the GPURays. It turns out that Metal does not support the pixel format Ogre::PFG_RGB32_FLOAT used for the cubeUVTexture and elsewhere (see Ogre::MetalMappings::get( PixelFormatGpu pf )).

Edit:

It looks like the textures used in the two render passes in GpuRay will need to use Ogre::PFG_RGBA32_FLOAT to run on Metal. This is going to be a little more involved than just substituting the alternative pixel format. Is there a developer guide or internal document detailing the calculations used in the GpuRay sensor that would help while working through the code?

[iche033]

I made some changes switching away from RGB32_FLOAT format in #468. Let me know if that works. Currently I set the 1st pass to RG32_FLOAT. If that does not work, I can change it to RGBA32_FLOAT

[srmainwaring]

Thanks @iche033 .The pixel format changes solves the Metal shader compilation issue.

Edit: GPU ray is now working properly on Metal. I'll commit the changes and update the task list.

[srmainwaring]

@iche033 and @darksylinc I'd appreciate your thoughts on the best approach to use Metal with QML and in particular Scene3d or its equivalent with a view to getting ign-gui and ign-gazebo fully supported on macOS.

I've got a basic demo working using the scene from simple_demo and a modified version of the Qt Scene Graph - Metal Texture Import. This is an outline of the approach:

1. A reference to the Metal texture is obtained from the Ogre::MetalTextureGpu using getCustomAttribute. I've implemented the access as a strong reference to id<MTLTexture>. This branch below contains the changes: it's close to the one currently used by ignition-rendering but I'd post a PR to either master or v2-2 if the approach is sound.

   • branch: https://github.com/srmainwaring/ogre-next/tree/feature/v2-2-osrf-macos
   • commit: srmainwaring/ogre-next@11a3adf

2. Access to the Metal texture in ignition-rendering is handled in a similar fashion to getting the GLId for OpenGL. A new function MTLId() has been added to the Camera and RenderTarget classes. Instead of returning the id<MTLTexture> I've kept access via a void* in the interface so as to not mix objective-c and c++ types (from (1) above it's implicit however that the pointer is a strong reference to the id).

   • branch: https://github.com/srmainwaring/ign-rendering/tree/feature/ign-rendering6-metal-simple_demo_qml
   • commit: srmainwaring@a876f53

3. Finally the example renders a basic scene and imports the texture into the scene graph for rendering by the QML application. The QML application is using the Metal backend QSGRendererInterface::MetalRhi (so it's all Metal).

   • branch: https://github.com/srmainwaring/ign-rendering/tree/feature/ign-rendering6-metal-simple_demo_qml
   • commit: srmainwaring@976253c

I've left all the quad animation from the Qt example, and have not forced the render to update when Qt doesn't think anything has changed (so the camera is only updated when animating).

One of my main concerns is getting the garbage collection / ARC policy right in the interop between objective-c and c++. The example doesn't have ARC enabled, so I suspect there is a resource leak, however if I use weak references in Ogre::MetalTextureGpu I see runtime exceptions when running in Xcode (although not from the command line cmake build).

[darksylinc]

Is CFBridgingRetain & CFBridgingRelease available to you from C++? If so you can use that.

Else you'll have to wrap those calls within some Obj C code.

See MetalRenderSystem::_hlmsSamplerblockCreated and MetalRenderSystem::_hlmsSamplerblockDestroyed which put a strong Obj-C reference (id <MTLSamplerState>) into HlmsSamplerblock::mRsData which is a void*

Edit: Please note that OgreMetalRenderSystem.mm is compiled with ARC enabled. The rest of the files don't have to; but where your "bridge/wrap" code is defined should have ARC enabled

[srmainwaring]

Is CFBridgingRetain & CFBridgingRelease available to you from C++? If so you can use that.

Else you'll have to wrap those calls within some Obj C code.

See MetalRenderSystem::_hlmsSamplerblockCreated and MetalRenderSystem::_hlmsSamplerblockDestroyed which put a strong Obj-C reference (id <MTLSamplerState>) into HlmsSamplerblock::mRsData which is a void*

Edit: Please note that OgreMetalRenderSystem.mm is compiled with ARC enabled. The rest of the files don't have to; but where your "bridge/wrap" code is defined should have ARC enabled

Thanks @darksylinc that's good to know.

[srmainwaring]

Aside from the example simple_demo_qml I think that this is done as the remaining examples that have ogre2 support for OpenGL are now working with Metal.

To get simple_demo_qml running with Metal will require changes similar to those applied to ign-gui in gazebosim/gz-gui#323 or an alternative fallback (#553 may cover this?). Either way, it's probably best tracked in a standalone issue.