Libmetal Shared Memory Allocation

[wjliang]

Issue Description:

When the application needs to shared data between a device or a remote processor,
it needs to:

• allocate memory for the system
• make the allocated shared memory accessible for both the applicaiton and the device/remote processor
• push and retrieve data from the shared memory

Libmetal APIs should provide the abtraction to allocate and return the shared memory across different operating systems, different remote devices and different types of memories.
The application on top of libmetal doesn't need to care about the underline platform/operating system specific memory management.

Operating Systems:

• Linux(userspace)
• Baremetal
• Zephyr (future)
• ...

Memory Type:

• Normal memory (e.g. DDR)
• Memories over PCIe (future)
• ...

Proposal

Libmetal Components involved:

• metal_io_region
• metal_shm
• metal_device

metal_io_region:

It is the minimum unit to present memory mapped address space, and it abstract how to access the address space. Will keep as what it is now.

metal_shm:

• Today, it is only used to keep track of all the registered metal_io_regions which can be used as shared memory, and allow user to get a metal_io_region from a name.

• Will need to extend this API to:

  • enable allocating shared memory from system or device
    • After the allocating shared memory operation, the application can access the memory.
    • enable assigning the shared memory to a remote device/processor
    • This API will not include the notification to the remote to tell which memory is assigned to Application/driver can use IPC for notification. E.g. RPMsg can be used to inform the remoteprocessor.
    • After assigning the shared memory to a remote:
    • the remote should be able to access memory.
    • device address of the shared memory should be available, so that application can use it to set up DMA or notify remote.

metal_device:

It is a device abstraction for libmetal application.

• Today, it abstracts the memory mapped address regions and operation to open() -- enable user to access its memory mapped address regions and close() operations.
• Will need to add the abtraction to allow to allocate/free memory from the device, and attach/detach memory to the device

Here are the use cases we are considering for now:

• UC1:

  • Linux/Baremetal application without IOMMU needs to share DDR memory to a device/coprocessor.

• UC2:

  • Linux application with IOMMU enabled needs to share DDR memory to a device/coprocessor.

Here is the proposed APIs:

• struct metal_generic_shm *metal_shm_allocate(size_t size, struct metal_device *src_dev, metal_shm_type shm_type);
  • This function will allocate size of memory from src_dev. The shm_type indicate if it is contiguous memory or not, can extend if required.
• struct metal_generic_shm *metal_shm_attach(struct metal_generic_shm *src_shm, struct metal_device *dst_dev);
  • attach the shared memory to the destination address
  • this operation will make the shared memory available to the remote. (It may map the memory differently to the source device in order to have the destination device to access it.)
  • the io_region will contain the memory address which applicaiton can be used to setup DMA or notify remote.
• metal_shm_detach(struct metal_generic_shm *dst_shm, struct metal_device *dst_dev);
  • detach the memory from the destination device
• metal_shm_free(struct metal_generic_shm *src_shm, struct metal_device *src_dev);
  • free the memory from the source device

Here is the proposal on Linux:

• Without IOMMU

  /* Step 1: Allocate shm with metal_shm_allocate() by passing source metal_device pointer.
             e.g. The source device can be ION device. It gets the DMA buf from the device.
  		   We suggest to use ION device so that libmetal can have generic implementaion
  		   to get DMA buffer.
  */
  /* Step 2: attach shm to the target device with metal_shm_attach() by passing dst_dev pointer.
                  the dest device underline Linux kernel driver will import the DMA buf and map it 
                      to make sure it is available for the remote.
  		We propose to add importing DMA buffer to UIO driver, so that the libmetal for
  		Linux can have generic implementation for DMA import.
   */
  
  

• With IOMMU

  /* Step 1: Allocate shm with metal_shm_allocate() by passing NULL src_dev pointer.
             This step will allocate some space with mmap(). And returns shm object
  		   which contains the io_region for the allocated memory.
  */
  /* Step 2: attach shm with vfio device with metal_shm_attach() by passing vfio dst_dev pointer.
             This step will call vfio iotl to map the memory for the device.
   */
  

[Raveem13]

Hi @wjliang, the issue is resolved or still in progress?

[arnopo]

Hi @Raveem13
@wjliang does not work anymore on the libmetal project
I close this one as stalled since a long time. I case of new discussion on IOMMU and/or share memory management, I propose to open a new one.