Proposal for Underlying Api Interoperability

[cwfitzgerald]

It is decently common for people to want to integrate wgpu into a larger application that is using a graphics api or to use a library built around a graphics api to integrate with wgpu. There are many different ways and with a wide variety of resources that you could need to do interop with, so this proposal will be a set of smaller api changes that combine to have a unified picture to underlying apis.

This API is sure to evolve as it is refined, this is just a first attempt to unify it

wgpu layer

We currently only support getting the underlying types for a wgpu type if we're running on wgpu-core/hal. We should also allow this for access to the underlying WebGPU resource. To facilitate this, we should have a trait that mirrors wgpu_hal::Api but only for associated types. This is only there for allowing generic functions, so no real trait bounds need to exist.

trait Api {
   type Instance;
   type Adapter;
   type Device;
   type Texture;
   type Buffer;
   ...
}

struct WebGPU;
pub use wgpu_hal::api::*;

impl Api for WebGPU {
    type Instance = ();
    type Adapter = GpuAdapter;
    ....
}

impl Api for A where A: wgpu_hal::Api {
    // These are not wgpu_hal::*::Instance. It's the actual underlying api instance type
    type Instance = A::RawInstance;
    type Adapter = A::RawDevice;
    ....
}

Second we change the as_hal interop functions to be as_inner_api and take A: Api

impl CommandEncoder {
    fn as_inner_api<A: Api>(&self, f: impl FnOnce(&mut A::CommandBuffer) -> R) -> R;
}

Ownership

The ownership of wgpu-created resources always lies in wgpu, all client code must keep the wgpu object alive while they are using the object.

The ownership of all wgpu-imported resources will depend on the presence of a DropGuard. This drop guard is a Box<dyn FnOnce(A::Raw*)> which will be called when the resource is destroyed.

• If a drop guard is provided, wgpu does not own the resource, and it must be kept alive externally until the provided drop guard is call.
• If a drop guard is not provided, wgpu does own the resource, and it will be destroyed as if created by wgpu.

Creation

For each importable object, we will add a associated type that gives all the information wgpu-hal needs to successfully import a type.

trait Api {
    type InstanceImportDescriptor;
    type AdapterImportDescriptor;
    type DeviceImportDescriptor;
    type TextureImportDescriptor;
    type BufferImportDescriptor;
    // Note _not_ command encoder.
    type CommandBufferImportDescriptor;
}

On the wgpu level, it will look like:

impl Device {
    unsafe fn create_texture_from_inner<A: Api>(
        &self,
        raw: A::Texture,
        desc: &TextureDescriptor,
        import_desc: A::TextureImportDescriptor,
        drop_guard: DropGuard,
    );
}

This should work for all the importable objects.

Resource States

Add the following to the API trait

trait Api {
    // Provides all information to be able to issue a barrier for the state.
    type BufferState;
    // Provides all information to be able to issue a barrier for the state.
    type TextureState;
   
    fn buffer_usage_to_state(hal::BufferUses) -> Self::BufferState;
    fn texture_usage_to_state(hal::TextureState) -> Self::TextureState;
}

When using buffers and textures with external code, or external command buffers, you must follow the following flow (as seen by the queue command stream)

1. Release resource for external use.
2. External use (either via a wgpu command encoder, or via an imported command buffer)
3. Acquire resource from external use.

The release/acquire terminology could probably be improved.

Step one and three can be accomplished with the following api:

struct ResourceStates<T, U> {
    resource: T,
    usage: U,
    requirement: ResourceUsageRequirement,
}

enum ResourceUsageRequirement {
    // If the command buffer does not know the state of the object, will move it to the provided state.
    // otherwise will leave the state alone. If you are going to record your own barrier before using the resource, this will
    // reduce the total amount o barriers.
    Optional,
    // Will unconditionally bring the state of the object to the given state.
    Required,
}

impl CommandEncoder {
    // Returns the state that the resources are in. These can be converted to API
    // state using `A::*_usage_to_state`.
    //
    // All released resources must have their wgpu handle kept alive until the command re-consuming them is recorded onto a submitted command buffer.
    unsafe fn release_resources(
        &mut self,
        textures: &[ResourceStates<&TextureView, hal::TextureUses>]
        buffers: &[ResourceStates<&Buffer, hal::BufferUses>]
    ) -> (Vec<hal::TextureUses>, Vec<hal::BufferUses>);
    
    unsafe fn acquire_resource(
        &mut self,
        textures: &[(&TextureView, hal::TextureUses)]
        buffers: &[(&Buffer, hal::BufferUses)]
    );
}

[teoxoy]

I like the sound of this! Especially the flexibility of the ownership model for imported resources.

[i509VCB]

Second we change the as_hal interop functions to be as_inner_api and take A: Api

impl CommandEncoder {
    fn as_inner_api<A: Api>(&self, f: impl FnOnce(&mut A::CommandBuffer) -> R) -> R;
}

This would need to return Option<R> or panic if you pass the wrong associated type? Multiple backends in wgpu-hal being the main reason.

The ownership of all wgpu-imported resources will depend on the presence of a DropGuard. This drop guard is a Box<dyn FnOnce(A::Raw*)> which will be called when the resource is destroyed.

Do we need describe any threading guarantees about the drop guard? Also when are objects released? Currently the drop guard is Send + Sync. Assuming all the api types are copy or stateless, Send + Sync is probably fine and you'd just send the type back through a channel.

Creation

For each importable object, we will add a associated type that gives all the information wgpu-hal needs to successfully import a type.

trait Api {
    type InstanceImportDescriptor;
    type AdapterImportDescriptor;
    type DeviceImportDescriptor;
    type TextureImportDescriptor;
    type BufferImportDescriptor;
    // Note _not_ command encoder.
    type CommandBufferImportDescriptor;
}

A few of these associated types could be quite API dependent? I'm not sure how you'd import a command buffer from GL given that doesn't conceptually exist.

For Instance, Adapter and Device, do we need to provide a way for external apis to query what features/extensions wgpu expects? Vulkan is the biggest example and pain of the current process.

On the wgpu level, it will look like:

impl Device {
    unsafe fn create_texture_from_inner<A: Api>(
        &self,
        raw: A::Texture,
        desc: &TextureDescriptor,
        import_desc: A::TextureImportDescriptor,
        drop_guard: DropGuard,
    );
}

May need to handle panics or return an error from wrong API type?

Resource States

I imagine most of this is going to be Vulkan and DX12 related?

Other things:

Synchronization

Some APIs have primitives for fences and semaphores. How should wgpu allow having these be signalled when you submit commands from wgpu. There might also be an argument to allow exporting a fence/sync object from wgpu (I think EGLSync might be an example of this? Need to check)

Some use cases like using wgpu in a Wayland compositor will ideally have wgpu signal a fence I give to kernel modesetting so that when wgpu finishes rendering an atomic flip is committed. So this is probably an important thing to consider,

[AdrianEddy]

I'm strongly interested in this, I've already implemented a bunch of interop functions and was thinking about making it a separate crate like wgpu-interop or something, but it would be even better to have it directly in wgpu
If it's of any help, my implementations are here: https://github.com/gyroflow/gyroflow/tree/master/src/core/gpu in wgpu_interop*.rs, and a list of interop possibilities is here

[ids1024]

For Instance, Adapter and Device, do we need to provide a way for external apis to query what features/extensions wgpu expects? Vulkan is the biggest example and pain of the current process.

Do you mean like https://docs.rs/wgpu-hal/0.17.0/wgpu_hal/vulkan/struct.Instance.html#method.required_extensions and https://docs.rs/wgpu-hal/0.17.0/wgpu_hal/vulkan/struct.Adapter.html#method.required_device_extensions? Or something else?

It should be possible for an application creating a Vulkan instance to call Instance::from_raw and then call required_extensions, I think. I don't see an API to create an adapter from raw Vulkan types, so it would have to enumerate adapters from the instance to call required_device_extensions.

Conversely, to make use of as_hal with Vulkan when the adapter is created by wgpu, it seems like there would need to be a way to tell wgpu/wgpu_hal what additional Vulkan device/instance extensions the applications wants? Though that isn't technically necessary if the application can use *from_raw with to create the instance and device from Vulkan types.

[i509VCB]

For Instance, Adapter and Device, do we need to provide a way for external apis to query what features/extensions wgpu expects? Vulkan is the biggest example and pain of the current process.

Do you mean like https://docs.rs/wgpu-hal/0.17.0/wgpu_hal/vulkan/struct.Instance.html#method.required_extensions and https://docs.rs/wgpu-hal/0.17.0/wgpu_hal/vulkan/struct.Adapter.html#method.required_device_extensions? Or something else?

It should be possible for an application creating a Vulkan instance to call Instance::from_raw and then call required_extensions, I think. I don't see an API to create an adapter from raw Vulkan types, so it would have to enumerate adapters from the instance to call required_device_extensions.

Yes, but there are finer details like ensuring device features that are given are enabled and a way to negotiate optional ones.

Conversely, to make use of as_hal with Vulkan when the adapter is created by wgpu, it seems like there would need to be a way to tell wgpu/wgpu_hal what additional Vulkan device/instance extensions the applications wants? Though that isn't technically necessary if the application can use *from_raw with to create the instance and device from Vulkan types.

This wouldn't work since some extensions require chaining something to VkDeviceCreateInfo.

[i509VCB]

For the GL(ES) backend, what would be be the "Raw" Instance, Adapter and Queue?

For EGL there is a context in the Instance, Adapter and Queue.

Also I'd probably ask the same regarding the pipelines and commandbuffer/encoder.

For dx11, these would probably all be unit, but probably not of use right now (so unit ()).

For Metal everything seems to map well except for Instance not really existing, and Device and Adapter are the same thing pretty much.

DX12, I think d3d12::DxgiFactory in an instance and everything else maps over.

A future pull request over #4573 might mean a RawTexture on Vulkan is in fact an array of textures? Not sure where this will go.

[AdrianEddy]

A future pull request over #4573 might mean a RawTexture on Vulkan is in fact an array of textures? Not sure where this will go.

Texture arrays and multi-planar textures are a different thing, and that's not specific to Vulkan.
Multi-planar textures (used in video decoders/encoders) are a single texture, but they have a special ability to create a texture view to each plane.

• DirectX has this as DXGI_FORMAT_NV12 and friends (AYUV, P010 etc),
• Vulkan is similar and the multi-planar formats are described here
• Apple is a bit different, it requires a call to CVMetalTextureCacheCreateTextureFromImage to get a plane texture from video decoder "output texture", and the resulting textures are Metal's single-plane textures
• On OpenGL the closest thing would be OES_EGL_image_external but is also a bit different thing.

In general, video APIs which output/input these hardware video textures are: DXVA2/D3D11VA/MF (windows), VideoToolbox (apple), NVIDIA Video API (windows, linux), MediaCodec (android), AMD AMF (windows, linux), Intel Quick Sync (windows, linux), VA-API (linux, windows), VDPAU (linux), Vulkan video extension

[alexgeek]

Hi, I've seen there's some work on this and quite a few issues closed and pointing this instead.

I had a look as saw we have Texture::as_hal, I tried to use it like so but nothing in Dx12::Texture is public.

let d3d12_resource = unsafe {
        texture.as_hal::<Dx12, _, _>(|hal_texture| {
            if let Some(hal_texture) = hal_texture {
                Some(hal_texture.resource.as_ptr())
            } else {
                None
            }
        })
    };

126 |                 Some(hal_texture.resource.as_ptr())
    |                                  ^^^^^^^^ private field

Was hoping to cast it to a IDXGIResource so that I can grab the handle and share it to other processes:

    if let Some(d3d12_resource) = d3d12_resource {
        use windows::core::ComInterface;
        let dxgi_resource: IDXGIResource = d3d12_resource.cast().expect("Should be able to cast.");
        let shared_handle = unsafe { dxgi_resource.GetSharedHandle() }.unwrap();

        println!("Shared handle: {:?}", shared_handle);
    } else {
        println!("Failed to get D3D12 resource from texture");
    }

Should those fields be public in Dx12::Texture? I'm failing to see what I could actually do with the texture I get from as_hal, I've seen examples around of creating texture from raw but not seeing the reverse.

Cheers

[Vecvec]

Was hoping to cast it to a IDXGIResource so that I can grab the handle and share it to other processes

I'd like to be able to grab handles for both vulkan and dx12 (for buffers/textures). This is also quite hard for vulkan (basically need to duplicate the entire create buffer code), it would be nice to have this as a wgpu feature. It seems that this is supported in dx12 always and vulkan after 1.1 (or with VK_KHR_external_memory).

[rojsc]

I'm also interested in getting native handles that are underlying wgpu resources.

Should those fields be public in Dx12::Texture? I'm failing to see what I could actually do with the texture I get from as_hal, I've seen examples around of creating texture from raw but not seeing the reverse.

I'm not sure if I understand the full picture, but it seems that the Vulkan backend already supports this in its Vulkan::Texture::raw_handle() function:

impl Texture {
    /// # Safety
    ///
    /// - The image handle must not be manually destroyed
    pub unsafe fn raw_handle(&self) -> vk::Image {
        self.raw
    }
}

So for a Vulkan backend, calling first Texture::as_hal and then raw_handle() on the returned Vulkan Texture should be enough to obtain a vk::Image instance.

Other backend don't seem to support this though.

[Wumpf]

PRs to add similar methods to other hal backends would definitely be appreciated :)