how can I use a custom style layer to render custom OpenGL content.

[GaoboLi]

I first wanted to customize the effect with opengl, but I didn't find any examples to refer to.

[mwilsnd]

To render arbitrary content on the map, you need to create an implementation of mbgl::style::CustomLayerHost and register it together with a custom layer instance in your map style. You can refer to this test for a minimal custom layer example.

For a more advanced use, like modifying a built-in layer, it looks like you'll need to provide an implementation of mbgl::LayerManager so you can register your modified layers. The default layer manager provided by the default platform is hard-coded so you'll need to edit that file or replace it in the build system for your chosen platform. From there you can provide custom implementations of the various layer factories.