[API Proposal]: Update to the LibraryImport custom marshalling shape
#70859
Comments
AaronRobinsonMSFT
added
api-suggestion
|
Tagging subscribers to this area: @dotnet/interop-contrib Issue DetailsBackground and motivationThere has been substantial feedback regarding the current This API would also apply to all previously approved marshaller APIs. This will also apply to/supplant outstanding source generator proposals. API Proposal/// <summary>
/// Define features for a custom type marshaller.
/// </summary>
public sealed class CustomTypeMarshallerFeaturesAttribute : Attribute
{
/// <summary>
/// Desired caller buffer size for the marshaller.
/// </summary>
public int BufferSize { get; set; }
}
/// <summary>
/// Base class attribute for custom marshaller attributes.
/// </summary>
/// <remarks>
/// Use a base class here to allow doing ManagedToUnmanagedMarshallersAttribute.GenericPlaceholder, etc. without having 3 separate placeholder types.
/// For the following attribute types, any marshaller types that are provided will be validated by an analyzer to have the correct members to prevent
/// developers from accidentally typoing a member like Free() and causing memory leaks.
/// </remarks>
public abstract class CustomUnmanagedTypeMarshallersAttributeBase : Attribute
{
/// <summary>
/// Placeholder type for generic parameter
/// </summary>
public sealed class GenericPlaceholder { }
}
/// <summary>
/// Specify marshallers used in the managed to unmanaged direction (that is, P/Invoke)
/// </summary>
public sealed class ManagedToUnmanagedMarshallersAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
/// <summary>
/// Create instance of <see cref="ManagedToUnmanagedMarshallersAttribute"/>.
/// </summary>
/// <param name="managedType">Managed type to marshal</param>
public ManagedToUnmanagedMarshallersAttribute(Type managedType) { }
/// <summary>
/// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>in</c> keyword.
/// </summary>
public Type? InMarshaller { get; set; }
/// <summary>
/// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>ref</c> keyword.
/// </summary>
public Type? RefMarshaller { get; set; }
/// <summary>
/// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>out</c> keyword.
/// </summary>
public Type? OutMarshaller { get; set; }
}
/// <summary>
/// Specify marshallers used in the unmanaged to managed direction (that is, Reverse P/Invoke)
/// </summary>
public sealed class UnmanagedToManagedMarshallersAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
/// <summary>
/// Create instance of <see cref="UnmanagedToManagedMarshallersAttribute"/>.
/// </summary>
/// <param name="managedType">Managed type to marshal</param>
public UnmanagedToManagedMarshallersAttribute(Type managedType) { }
/// <summary>
/// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>in</c> keyword.
/// </summary>
public Type? InMarshaller { get; set; }
/// <summary>
/// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>ref</c> keyword.
/// </summary>
public Type? RefMarshaller { get; set; }
/// <summary>
/// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>out</c> keyword.
/// </summary>
public Type? OutMarshaller { get; set; }
}
/// <summary>
/// Specify marshaller for array-element marshalling and default struct field marshalling.
/// </summary>
public sealed class ElementMarshallerAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
/// <summary>
/// Create instance of <see cref="ElementMarshallerAttribute"/>.
/// </summary>
/// <param name="managedType">Managed type to marshal</param>
/// <param name="elementMarshaller">Marshaller type to use for marshalling <paramref name="managedType"/>.</param>
public ElementMarshallerAttribute(Type managedType, Type elementMarshaller) { }
}
/// <summary>
/// Specifies that a particular generic parameter is the collection element's unmanaged type.
/// </summary>
/// <remarks>
/// If this attribute is provided on a generic parameter of a marshaller, then the generator will assume
/// that it is a linear collection marshaller.
/// </remarks>
[AttributeUsage(AttributeTargets.GenericParameter)]
public sealed class ElementUnmanagedTypeAttribute : Attribute
{
}API Usage[ManagedToUnmanagedMarshallers(
typeof(TManaged),
InMarshaller = typeof(ManagedToUnmanaged),
RefMarshaller = typeof(Bidirectional),
OutMarshaller = typeof(UnmanagedToManaged))]
[UnmanagedToManagedMarshallers(
typeof(TManaged),
InMarshaller = typeof(UnmanagedToManaged),
RefMarshaller = typeof(Bidirectional),
OutMarshaller = typeof(ManagedToUnmanaged))]
[ElementMarshaller(
typeof(TManaged),
typeof(Element))]
public unsafe static class Marshaller // Must be static class
{
public unsafe ref struct ManagedToUnmanaged
{
public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes
public TUnmanaged ToUnmanaged() => throw null;
public void Free() => throw null; // Should not throw exceptions. Use Free instead of Dispose to avoid issues with marshallers needing to follow the Dispose pattern guidance.
}
public unsafe ref struct Bidirectional
{
public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes.
public TUnmanaged ToUnmanaged() => throw null; // Should not throw exceptions.
public void FromUnmanaged(TUnmanaged native) => throw null; // Should not throw exceptions.
public TManaged ToManaged() => throw null;
public void Free() => throw null; // Should not throw exceptions.
}
public unsafe struct UnmanagedToManaged
{
public void FromUnmanaged(TUnmanaged native) => throw null;
public TManaged ToManaged() => throw null; // Should not throw exceptions.
public void Free() => throw null; // Should not throw exceptions.
}
// Currently only support stateless. May support stateful in the future
public static class Element
{
// Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
public static TUnmanaged ConvertToUnmanaged(TManaged managed) => throw null;
public static TManaged ConvertToManaged(TUnmanaged native) => throw null;
public static void Free(TUnmanaged native) => throw null; // Should not throw exceptions.
}
}
[ManagedToUnmanagedMarshallers(typeof(ManagedToUnmanagedMarshallersAttribute.GenericPlaceholder[]),
InMarshaller = typeof(ArrayMarshaller<,>.In),
RefMarshaller = typeof(ArrayMarshaller<,>),
OutMarshaller = typeof(ArrayMarshaller<,>.Out))]
[UnmanagedToManagedMarshallers(typeof(UnmanagedToManagedMarshallersAttribute.GenericPlaceholder[]),
InMarshaller = typeof(ArrayMarshaller<,>),
RefMarshaller = typeof(ArrayMarshaller<,>),
OutMarshaller = typeof(ArrayMarshaller<,>))]
[ElementMarshaller(typeof(ElementMarshallerAttribute.GenericPlaceholder[]), typeof(ArrayMarshaller<,>))]
[CustomTypeMarshallerFeatures(BufferSize = 20)]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement> // Must be static class
where TUnmanagedElement : unmanaged
{
// Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
public static byte* AllocateContainerForUnmanagedElements(T[]? managed, out int numElements)
{
if (managed is null)
{
numElements = 0;
return null;
}
numElements = managed.Length;
return (byte*)Marshal.AllocCoTaskMem(checked(sizeof(TUnmanagedElement) * numElements));
}
public static ReadOnlySpan<T> GetManagedValuesSource(T[] managed) => managed;
public static Span<T> GetManagedValuesDestination(T[] managed) => managed;
public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements)
=> new Span<TUnmanagedElement>(nativeValue, numElements);
public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements)
=> new Span<TUnmanagedElement>(nativeValue, numElements);
public static T[] AllocateContainerForManagedElements(int length) => new T[length];
public static void Free(byte* native) => Marshal.FreeCoTaskMem((IntPtr)native);
[CustomTypeMarshallerFeatures(BufferSize = 20)] // As our buffer is typed to our native element type, we limit the buffer size based on number of elements, not number of bytes.
public unsafe ref struct In
{
private T[]? _managedArray;
private IntPtr _allocatedMemory;
private Span<TUnmanagedElement> _span;
/// <summary>
/// Initializes a new instance of the <see cref="ArrayMarshaller{T}"/>.
/// </summary>
/// <param name="array">Array to be marshalled.</param>
/// <param name="buffer">Buffer that may be used for marshalling.</param>
/// <param name="sizeOfUnmanagedElement">Size of the native element in bytes.</param>
/// <remarks>
/// The <paramref name="buffer"/> must not be movable - that is, it should not be
/// on the managed heap or it should be pinned.
/// <seealso cref="CustomTypeMarshallerFeatures.CallerAllocatedBuffer"/>
/// </remarks>
public void FromManaged(T[]? array, Span<TUnmanagedElement> buffer)
{
_allocatedMemory = default;
if (array is null)
{
_managedArray = null;
_span = default;
return;
}
_managedArray = array;
// Always allocate at least one byte when the array is zero-length.
int bufferSize = checked(array.Length * sizeof(TUnmanagedElement));
int spaceToAllocate = Math.Max(bufferSize, 1);
if (spaceToAllocate <= buffer.Length)
{
_span = buffer[0..spaceToAllocate];
}
else
{
_allocatedMemory = Marshal.AllocCoTaskMem(spaceToAllocate);
_span = new Span<TUnmanagedElement>((void*)_allocatedMemory, spaceToAllocate);
}
}
/// <summary>
/// Gets a span that points to the memory where the managed values of the array are stored.
/// </summary>
/// <returns>Span over managed values of the array.</returns>
/// <remarks>
/// <seealso cref="CustomTypeMarshallerDirection.In"/>
/// </remarks>
public ReadOnlySpan<T> GetManagedValuesSource() => _managedArray;
/// <summary>
/// Returns a span that points to the memory where the native values of the array should be stored.
/// </summary>
/// <returns>Span where native values of the array should be stored.</returns>
public Span<TUnmanagedElement> GetUnmanagedValuesDestination() => _span;
/// <summary>
/// Returns a reference to the marshalled array.
/// </summary>
public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);
/// <summary>
/// Returns the native value representing the array.
/// </summary>
public byte* ToUnmanaged() => (byte*)Unsafe.AsPointer(ref GetPinnableReference());
/// <summary>
/// Sets the native value representing the array.
/// </summary>
/// <param name="value">The native value.</param>
public void FromUnmanaged(byte* value)
{
_allocatedMemory = (IntPtr)value;
}
/// <summary>
/// Returns the managed array.
/// </summary>
public T[]? ToManaged() => _managedArray;
/// <summary>
/// Frees native resources.
/// </summary>
public void Free()
{
Marshal.FreeCoTaskMem(_allocatedMemory);
}
public static ref T GetPinnableReference(T[] managed) // Optional, allowed on all shapes
{
if (managed is null)
{
return ref Unsafe.NullRef<T>();
}
return ref MemoryMarshal.GetArrayDataReference(managed);
}
}
public unsafe ref struct Out
{
private T[]? _managedArray;
private IntPtr _allocatedMemory;
private Span<TUnmanagedElement> _span;
/// <summary>
/// Gets a span that points to the memory where the unmarshalled managed values of the array should be stored.
/// </summary>
/// <param name="length">Length of the array.</param>
/// <returns>Span where managed values of the array should be stored.</returns>
public Span<T> GetManagedValuesDestination(int length) => _allocatedMemory == IntPtr.Zero ? null : _managedArray = new T[length];
/// <summary>
/// Returns a span that points to the memory where the native values of the array are stored after the native call.
/// </summary>
/// <param name="length">Length of the array.</param>
/// <returns>Span over the native values of the array.</returns>
public ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(int length)
{
if (_allocatedMemory == IntPtr.Zero)
return default;
return _span = new Span<TUnmanagedElement>((void*)_allocatedMemory, length);
}
/// <summary>
/// Returns a reference to the marshalled array.
/// </summary>
public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);
/// <summary>
/// Sets the native value representing the array.
/// </summary>
/// <param name="value">The native value.</param>
public void FromUnmanaged(byte* value)
{
_allocatedMemory = (IntPtr)value;
}
/// <summary>
/// Returns the managed array.
/// </summary>
public T[]? ToManaged() => _managedArray;
/// <summary>
/// Frees native resources.
/// </summary>
public void Free()
{
Marshal.FreeCoTaskMem(_allocatedMemory);
}
}
}Alternative DesignsNo response RisksThis is close to an RC and is a non-trivial update to the Marshaller shape. The ASP.Net Core and WinForms repos will need to respond as well. This is introducing complexity in the development slow down following an RC release in several weeks. There is no doubt risk here. A potential escape is to go with the current shape and "make it work". Given the internal/external feedback we have received the risk is worth it from the perspective of the Interop team.
|
|
/cc @pavelsavara |
AaronRobinsonMSFT
added
api-ready-for-review
|
The existing |
I am wondering whether it would look better to replace all these attributes with just one attribute and enum. Something like: public class CustomMarshallerAttribute : Attribute
{
public CustomMarshallerAttribute(Type managedType) { }
public CustomMarshallerAttribute(Type managedType, CustomMarshallersKind kind, Type marshallerType) { }
/// <summary>
/// Placeholder type for generic parameter
/// </summary>
public sealed class GenericPlaceholder { }
}
public enum CustomMarshallersKind
{
Default,
ManagedToUnmanaged, // Optional - allows you to specify marshaller for all managed->unmanaged cases without repeating the marshaller type 3x
ManagedToUnmanagedIn,
ManagedToUnmanagedRef,
ManagedToUnmanagedOut,
UnmanagedToManaged, // Optional - allows you to specify marshaller for all unmanaged->managed cases without repeating the marshaller type 3x
UnmanagedToManagedIn,
UnmanagedToManagedRef,
UnmanagedToManagedOut,
ElementMarshaller
...
} |
Now that all marshaller features are based on pattern matching, the BufferSize that is the only thing left in public unsafe ref struct In
{
public const int StackBufferSize = 20;
}Field may be more consistent with the overall design than an attribute. (I do not have a strong opinion about it.) |
|
We had the buffer size field vs attribute in the alternate designs to make sure to bring it up during API review. (I don't think any of us had a strong opinion either.) |
|
Ok - I have missed it.
If we go with this, we may want to allow both |
|
I think the best route to go would be to allow a field or property in this case, as long as it is static and read-only. That gives some extra flexibility for some generic cases that we might find useful. |
|
I've updated the API proposal with a full list of added and removed APIs. |
I'd prefer to avoid the enum for two reasons. The first is it permits multiple instances of an attribute and instead of 1 it becomes 1 up to N where N is the number of unique enumeration values - that is difficult to reason about from my perspective. The second, and @jkoritzinsky or @elinor-fung can confirm, but we would now need to handle the complexity of what enum values are unique or valid combinations - that seems like a difficult matrix to encode and validate we get right. I am all for iterating on the single attribute but I'd prefer if it was a single attribute. |
|
I have been looking at what the code for marshallers looks like and whether it is looks good and intuitive. Here are a few examples of where I think the current proposal falls short. |
|
Simple state-less marshaller: // Per our offline conversation, this implies that the marshaler is usable for both
// managed->unmanaged and unmanaged->managed directions. Is that still correct?
// I find that confusing. The attribute says managed->unmanaged. It is not obvious
// that it implies support for unmanaged->managed too.
[ManagedToUnmanagedMarshallers(typeof(bool))]
static class WindowsBoolMarshaller
{
public static int ConvertToUnmanaged(bool b);
public static bool ConvertToManaged(int wb);
}Alternative: [CustomMarshaller(typeof(bool))]
static class WindowsBoolMarshaller
{
public static int ConvertToUnmanaged(bool b);
public static bool ConvertToManaged(int wb);
} |
|
In order to convert in both directions I think both attributes would be needed. [ManagedToUnmanagedMarshallers(typeof(bool))]
[UnmanagedToManagedMarshallers(typeof(bool))]
static class WindowsBoolMarshaller
{
public static int ConvertToUnmanaged(bool b);
public static bool ConvertToManaged(int wb);
} |
And if I want support for array element marshalling, do I have to tag it with |
|
Yes, that is required by the current design. If we were to go the |
|
ArrayMarshaller can look like this: // This is equivalent to `[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]), Kind=CustomMarshallersKind.Default, Marshaller=typeof(ArrayMarshaller<,>)]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]))]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]),
Kind = CustomMarshallersKind.ManagedToUnmanagedIn,
Marshaller = typeof(ArrayMarshaller<,>.In)]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement> // Must be static class
...We would always find the most specific marshaller for the scenario. For example, we would first look for |
|
I really dislike the multiple attribute with an enum approach. |
|
I'm not the biggest fan of that design as we'd have to validate that none of the attributes have the same Kind specified. Also, how would we specify a not-supported scenario? Explicitly set the Other than those two concerns, I like that it isn't as verbose and would allow us to expand the scenarios list in the future if need be without introducing additional types. |
Either by specifying null, or not specifying |
It's not just that. It is also what does "default" mean and when does it apply? The below case now has cases where the use is implied again and creates far more confusion. The single attribute with multiple values is for sure verbose but it is also very clear about intent and has no ambiguity about when something applies. [CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]))]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]),
Kind = CustomMarshallersKind.ManagedToUnmanagedIn,
Marshaller = ...]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]),
Kind = CustomMarshallersKind.ManagedToUnmanagedRef,
Marshaller = ...]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]),
Kind = CustomMarshallersKind.ManagedToUnmanagedOut,
Marshaller = ...]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]),
Kind = CustomMarshallersKind.Default,
Marshaller = ...]
[CustomMarshaller(typeof(CustomMarshaller.GenericPlaceholder[]),
Kind = CustomMarshallersKind.ManagedToUnmanaged,
Marshaller = ...]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement> // Must be static class**
Before we start going down that path as an argument, I'd like to see even a contrived example where we might want something new. |
|
We've talked a little about a struct marshalling source generator that automatically generates a marshaller type for a given managed struct type. There's a question that arises: in this new design with specialized marshallers, which marshaller do we use for field marshalling? Generally, we would want the same behavior as for parameters, but for some types, we want different behavior. Specifically, SafeHandle-derived types have a different marshaller for fields than they do for other scenarios. SafeHandles specifically do not support the handle value changing when they are fields in a struct. With our currently proposed "Kind"s or scenarios, which marshaller would we use for SafeHandle fields? Would we use the "element" marshaller? If so, then we'd allow arrays of SafeHandle's as well with the same logic as fields (this option would only work once we support preserving state for element marshallers). Would we use one of the Ref marshallers? If so, which one? Also, here's another (less concrete scenario): Right now, since our element marshallers are stateless, we don't have the issue of specifying state in them. When we add support for stateful element marshallers, we'll end up in a scenario where the state diagram is more complex, as unlike all of the other shapes defined in the new design doc, we will not always call each of the instance methods in one of the defined shapes. We'd only call the "managed->unmanaged" methods when the collection is passed by-val without |
Yes.
That is true. Since it isn't needed right now I think that is okay. We've called out |
Well, the proposed attribute hierarchy is optimized for cases that are not that common. Looking at the universe of all existing built-in marshallers:
The proposed attribute hierarchy is optimized for cases where you need to have a special marshaller for each scenario or direction, or some scenarios or directions are unsupported. These cases are not very common. We can certainly live with the proposed hierarchy. It is sufficient to achieve the desired functionality. I am just wondering whether we can come up with something that is simpler for the most common cases. |
That isn't the intent with the hierarchy, although one could assume that. The optimization is for clarity. Yes, there are three attribute possible, but that is it and it is very explicit about intent. It was optimized for UX and explaining the system. The proposed multi-attribute approach optimizes for less writing but again, has a large number of defaults which the interop team has been working to avoid and focus on explicit settings. It also imposes a large amount of complexity on which values supersede others, which I would argue is a less ideal UX than having to write The |
I am open to that. Improved UX to me is explicit and helping to avoid ambiguity. The enum approach reduces writing but introduces a more complex supersede construct that I find increases ambiguity. Given the majority of people are in IDEs and we have all the language services running reducing a few more |
namespace System.Runtime.InteropServices.Marshalling;
/// <summary>
/// An enumeration representing the different marshalling scenarios in our marshalling model
/// </summary>
public enum MarshalMode
{
/// <summary>
/// All scenarios. A marshaller specified with this scenario will be used if there is not a specific
/// marshaller specified for a given usage scenario.
/// </summary>
Default,
/// <summary>
/// By-value and <c>in</c> parameters in managed-to-unmanaged scenarios, like P/Invoke.
/// </summary>
ManagedToUnmanagedIn,
/// <summary>
/// <c>ref</c> parameters in managed-to-unmanaged scenarios, like P/Invoke.
/// </summary>
ManagedToUnmanagedRef,
/// <summary>
/// <c>out</c> parameters in managed-to-unmanaged scenarios, like P/Invoke.
/// </summary>
ManagedToUnmanagedOut,
/// <summary>
/// By-value and <c>in</c> parameters in unmanaged-to-managed scenarios, like Reverse P/Invoke.
/// </summary>
UnmanagedToManagedIn,
/// <summary>
/// <c>ref</c> parameters in unmanaged-to-managed scenarios, like Reverse P/Invoke.
/// </summary>
UnmanagedToManagedRef,
/// <summary>
/// <c>out</c> parameters in unmanaged-to-managed scenarios, like Reverse P/Invoke.
/// </summary>
UnmanagedToManagedOut,
/// <summary>
/// Elements of arrays passed with <c>in</c> or by-value in interop scenarios.
/// </summary>
ElementIn,
/// <summary>
/// Elements of arrays passed with <c>ref</c> or passed by-value with both <see cref="InAttribute"/> and <see cref="OutAttribute" /> in interop scenarios.
/// </summary>
ElementRef,
/// <summary>
/// Elements of arrays passed with <c>out</c> or passed by-value with only <see cref="OutAttribute" /> in interop scenarios.
/// </summary>
ElementOut
}
/// <summary>
/// Specifies which marshaller to use for a given managed type and marshalling scenario.
/// </summary>
/// <remarks>
/// This attribute should be placed on a marshaller entry-point type.
/// </remarks>
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct, AllowMultiple = true)]
public sealed class CustomMarshallerAttribute : Attribute
{
public CustomMarshallerAttribute(Type managedType, MarshalMode marshalMode, Type marshallerType);
public Type ManagedType { get; }
public MarshalMode MarshalMode { get; }
public Type MarshallerType { get; }
/// <summary>
/// Placeholder type for generic parameter
/// </summary>
public struct GenericPlaceholder
{
}
}
/// <summary>
/// Specifies that the attributed custom marshaller marshals a linear/contiguous collection.
/// </summary>
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct)]
public sealed class ContiguousCollectionMarshallerAttribute : Attribute
{
}
[CustomMarshaller(typeof(string), Scenario.Default, typeof(Utf8StringMarshaller))]
[CustomMarshaller(typeof(string), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanagedIn))]
public static class Utf8StringMarshaller
{
public static byte* ConvertToUnmanaged(string managed);
public static string ConvertToManaged(byte* unmanaged);
public static void Free(byte* unmanaged);
public ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(string managed, Span<byte> buffer);
public byte* ToUnmanaged();
public void FromUnmanaged(byte* unmanaged);
public string ToManaged();
public void Free();
}
}
[CustomMarshaller(typeof(string), Scenario.Default, typeof(AnsiStringMarshaller))]
[CustomMarshaller(typeof(string), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanagedIn))]
public static class AnsiStringMarshaller
{
public static byte* ConvertToUnmanaged(string managed);
public static string ConvertToManaged(byte* unmanaged);
public static void Free(byte* unmanaged);
public ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(string managed, Span<byte> buffer);
public byte* ToUnmanaged();
public void FromUnmanaged(byte* unmanaged);
public string ToManaged();
public void Free();
}
}
[CustomMarshaller(typeof(string), Scenario.Default, typeof(BStrStringMarshaller))]
[CustomMarshaller(typeof(string), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanagedIn))]
public static class BStrStringMarshaller
{
public static ushort* ConvertToUnmanaged(string managed);
public static string ConvertToManaged(ushort* unmanaged);
public static void Free(ushort* unmanaged);
public ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(string managed, Span<ushort> buffer);
public ushort* ToUnmanaged();
public void FromUnmanaged(ushort* unmanaged);
public string ToManaged();
public void Free();
}
}
[CustomMarshaller(typeof(string), Scenario.Default, typeof(Utf16StringMarshaller))]
public static class Utf16StringMarshaller
{
public static ushort* ConvertToUnmanaged(string managed);
public static string ConvertToManaged(ushort* unmanaged);
public static void Free(ushort* unmanaged);
public static ref char GetPinnableReference(string str);
}
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.Default, typeof(ArrayMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.ManagedToUnmanagedIn, typeof(ArrayMarshaller<,>.In))]
[ContiguousCollectionMarshaller]
public unsafe static class ArrayMarshaller<T, TUnmanagedElement>
where TUnmanagedElement : unmanaged
{
public static byte* AllocateContainerForUnmanagedElements(T[]? managed, out int numElements);
public static ReadOnlySpan<T> GetManagedValuesSource(T[] managed);
public static Span<T> GetManagedValuesDestination(T[] managed);
public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);
public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements);
public static T[] AllocateContainerForManagedElements(byte* nativeValue, int length);
public static void Free(byte* native);
public unsafe ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(T[]? array, Span<TUnmanagedElement> buffer);
public ReadOnlySpan<T> GetManagedValuesSource();
public Span<TUnmanagedElement> GetUnmanagedValuesDestination();
public ref TUnmanagedElement GetPinnableReference();
public byte* ToUnmanaged();
public void Free();
public static ref T GetPinnableReference(T[] managed);
}
}
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.Default, typeof(PointerArrayMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.ManagedToUnmanagedIn, typeof(PointerArrayMarshaller<,>.In))]
[ContiguousCollectionMarshaller]
public unsafe static class PointerArrayMarshaller<T, TUnmanagedElement>
where T : unmanaged
where TUnmanagedElement : unmanaged
{
public static byte* AllocateContainerForUnmanagedElements(T*[]? managed, out int numElements);
public static ReadOnlySpan<IntPtr> GetManagedValuesSource(T*[] managed);
public static Span<IntPtr> GetManagedValuesDestination(T*[] managed);
public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);
public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements);
public static T*[] AllocateContainerForManagedElements(byte* nativeValue, int length);
public static void Free(byte* native);
public unsafe ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(T*[]? array, Span<TUnmanagedElement> buffer);
public ReadOnlySpan<IntPtr> GetManagedValuesSource();
public Span<TUnmanagedElement> GetUnmanagedValuesDestination();
public ref TUnmanagedElement GetPinnableReference();
public byte* ToUnmanaged();
public void Free();
public static ref T* GetPinnableReference(T*[] managed);
}
}
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.Default, typeof(SpanMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.ManagedToUnmanagedIn, typeof(SpanMarshaller<,>.In))]
[ContiguousCollectionMarshaller]
public unsafe static class SpanMarshaller<T, TUnmanagedElement>
where TUnmanagedElement : unmanaged
{
public static byte* AllocateContainerForUnmanagedElements(Span<T> managed, out int numElements);
public static ReadOnlySpan<T> GetManagedValuesSource(Span<T> managed);
public static Span<T> GetManagedValuesDestination(Span<T> managed);
public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);
public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements);
public static Span<T> AllocateContainerForManagedElements(byte* nativeValue, int length);
public static void Free(byte* native);
public unsafe ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(Span<T> array, Span<TUnmanagedElement> buffer);
public ReadOnlySpan<T> GetManagedValuesSource();
public Span<TUnmanagedElement> GetUnmanagedValuesDestination();
public ref TUnmanagedElement GetPinnableReference();
public byte* ToUnmanaged();
public void Free();
public static ref T GetPinnableReference(Span<T> managed);
}
}
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.ManagedToUnmanagedIn, typeof(SpanMarshaller<,>.ManagedToUnmanaged))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.UnmanagedToManagedOut, typeof(SpanMarshaller<,>.UnmanagedToManaged))]
[ContiguousCollectionMarshaller]
public unsafe static class ReadOnlySpanMarshaller<T, TUnmanagedElement>
where TUnmanagedElement : unmanaged
{
public unsafe ref struct ManagedToUnmanagedIn
{
public static int BufferSize { get; }
public void FromManaged(ReadOnlySpan<T> array, Span<TUnmanagedElement> buffer);
public ReadOnlySpan<T> GetManagedValuesSource();
public Span<TUnmanagedElement> GetUnmanagedValuesDestination();
public ref TUnmanagedElement GetPinnableReference();
public byte* ToUnmanaged();
public void Free();
public static ref T GetPinnableReference(ReadOnlySpan<T> managed);
}
public unsafe static class UnmanagedToManagedOut
{
public static byte* AllocateContainerForUnmanagedElements(ReadOnlySpan<T> managed, out int numElements);
public static ReadOnlySpan<T> GetManagedValuesSource(ReadOnlySpan<T> managed);
public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);
}
}Shapes [CustomMarshaller(typeof(TManaged), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanaged))]
[CustomMarshaller(typeof(TManaged), Scenario.ManagedToUnmanagedRef, typeof(Bidirectional))]
[CustomMarshaller(typeof(TManaged), Scenario.ManagedToUnmanagedOut, typeof(UnmanagedToManaged))]
[CustomMarshaller(typeof(TManaged), Scenario.UnmanagedToManagedIn, typeof(UnmanagedToManaged))]
[CustomMarshaller(typeof(TManaged), Scenario.UnmanagedToManagedRef, typeof(Bidirectional))]
[CustomMarshaller(typeof(TManaged), Scenario.UnmanagedToManagedOut, typeof(ManagedToUnmanaged))]
[CustomMarshaller(typeof(TManaged), Scenario.ElementIn, typeof(Element))]
[CustomMarshaller(typeof(TManaged), Scenario.ElementRef, typeof(Element))]
[CustomMarshaller(typeof(TManaged), Scenario.ElementOut, typeof(Element))]
public unsafe static class Marshaller // Must be static class
{
public unsafe ref struct ManagedToUnmanaged
{
public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes
public TUnmanaged ToUnmanaged() => throw null;
public void Free() => throw null; // Should not throw exceptions. Use Free instead of Dispose to avoid issues with marshallers needing to follow the Dispose pattern guidance.
// We will pattern-match this method and use it when available.
// Canonical case is for GC.KeepAlive().
public void OnInvoked() => throw null;
}
public unsafe ref struct Bidirectional
{
public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes.
public TUnmanaged ToUnmanaged() => throw null; // Should not throw exceptions.
public void FromUnmanaged(TUnmanaged native) => throw null; // Should not throw exceptions.
public TManaged ToManaged() => throw null;
// ToManagedFinally requests the generator to move the unmarshalling method calls to be called as part of the GuaranteedUnmarshalling stage.
// The generator will pattern-match this method as an alternative to ToManaged().
public TManaged ToManagedFinally() => throw null;
public void Free() => throw null; // Should not throw exceptions.
public void OnInvoked() => throw null; // See notes in ManagedToUnmanaged on usage.
}
public unsafe struct UnmanagedToManaged
{
public void FromUnmanaged(TUnmanaged native) => throw null;
public TManaged ToManaged() => throw null; // Should not throw exceptions.
public void Free() => throw null; // Should not throw exceptions.
}
// Currently only support stateless. May support stateful in the future
public static class Element
{
// Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
public static TUnmanaged ConvertToUnmanaged(TManaged managed) => throw null;
public static TManaged ConvertToManaged(TUnmanaged native) => throw null;
public static void Free(TUnmanaged native) => throw null; // Should not throw exceptions.
}
}
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.Default, typeof(ArrayMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.ManagedToUnmanagedIn, typeof(ArrayMarshaller<,>.In))]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement> // Must be static class
where TUnmanagedElement : unmanaged
{
// Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
public static byte* AllocateContainerForUnmanagedElements(T[]? managed, out int numElements)
{
if (managed is null)
{
numElements = 0;
return null;
}
numElements = managed.Length;
return (byte*)Marshal.AllocCoTaskMem(checked(sizeof(TUnmanagedElement) * numElements));
}
public static ReadOnlySpan<T> GetManagedValuesSource(T[] managed) => managed;
public static Span<T> GetManagedValuesDestination(T[] managed) => managed;
public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements)
=> new Span<TUnmanagedElement>(nativeValue, numElements);
public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements)
=> new Span<TUnmanagedElement>(nativeValue, numElements);
public static T[] AllocateContainerForManagedElements(byte* nativeValue, int length) => new T[length];
public static void Free(byte* native) => Marshal.FreeCoTaskMem((IntPtr)native);
public unsafe ref struct In
{
private T[]? _managedArray;
private IntPtr _allocatedMemory;
private Span<TUnmanagedElement> _span;
public static int BufferSize { get; } = 20;
/// <summary>
/// Initializes a new instance of the <see cref="ArrayMarshaller{T}"/>.
/// </summary>
/// <param name="array">Array to be marshalled.</param>
/// <param name="buffer">Buffer that may be used for marshalling.</param>
/// <param name="sizeOfUnmanagedElement">Size of the native element in bytes.</param>
/// <remarks>
/// The <paramref name="buffer"/> must not be movable - that is, it should not be
/// on the managed heap or it should be pinned.
/// <seealso cref="CustomTypeMarshallerFeatures.CallerAllocatedBuffer"/>
/// </remarks>
public void FromManaged(T[]? array, Span<TUnmanagedElement> buffer)
{
_allocatedMemory = default;
if (array is null)
{
_managedArray = null;
_span = default;
return;
}
_managedArray = array;
// Always allocate at least one byte when the array is zero-length.
int bufferSize = checked(array.Length * sizeof(TUnmanagedElement));
int spaceToAllocate = Math.Max(bufferSize, 1);
if (spaceToAllocate <= buffer.Length)
{
_span = buffer[0..spaceToAllocate];
}
else
{
_allocatedMemory = Marshal.AllocCoTaskMem(spaceToAllocate);
_span = new Span<TUnmanagedElement>((void*)_allocatedMemory, spaceToAllocate);
}
}
/// <summary>
/// Gets a span that points to the memory where the managed values of the array are stored.
/// </summary>
/// <returns>Span over managed values of the array.</returns>
/// <remarks>
/// <seealso cref="CustomTypeMarshallerDirection.In"/>
/// </remarks>
public ReadOnlySpan<T> GetManagedValuesSource() => _managedArray;
/// <summary>
/// Returns a span that points to the memory where the native values of the array should be stored.
/// </summary>
/// <returns>Span where native values of the array should be stored.</returns>
public Span<TUnmanagedElement> GetUnmanagedValuesDestination() => _span;
/// <summary>
/// Returns a reference to the marshalled array.
/// </summary>
public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);
/// <summary>
/// Returns the native value representing the array.
/// </summary>
public byte* ToUnmanaged() => (byte*)Unsafe.AsPointer(ref GetPinnableReference());
/// <summary>
/// Frees native resources.
/// </summary>
public void Free()
{
Marshal.FreeCoTaskMem(_allocatedMemory);
}
public static ref T GetPinnableReference(T[] managed) // Optional, allowed on all shapes
{
if (managed is null)
{
return ref Unsafe.NullRef<T>();
}
return ref MemoryMarshal.GetArrayDataReference(managed);
}
}
public unsafe ref struct Out
{
private T[]? _managedArray;
private IntPtr _allocatedMemory;
private Span<TUnmanagedElement> _span;
/// <summary>
/// Gets a span that points to the memory where the unmarshalled managed values of the array should be stored.
/// </summary>
/// <param name="length">Length of the array.</param>
/// <returns>Span where managed values of the array should be stored.</returns>
public Span<T> GetManagedValuesDestination(int length) => _allocatedMemory == IntPtr.Zero ? null : _managedArray = new T[length];
/// <summary>
/// Returns a span that points to the memory where the native values of the array are stored after the native call.
/// </summary>
/// <param name="length">Length of the array.</param>
/// <returns>Span over the native values of the array.</returns>
public ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(int length)
{
if (_allocatedMemory == IntPtr.Zero)
return default;
return _span = new Span<TUnmanagedElement>((void*)_allocatedMemory, length);
}
/// <summary>
/// Returns a reference to the marshalled array.
/// </summary>
public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);
/// <summary>
/// Sets the native value representing the array.
/// </summary>
/// <param name="value">The native value.</param>
public void FromUnmanaged(byte* value)
{
_allocatedMemory = (IntPtr)value;
}
/// <summary>
/// Returns the managed array.
/// </summary>
public T[]? ToManaged() => _managedArray;
/// <summary>
/// Frees native resources.
/// </summary>
public void Free()
{
Marshal.FreeCoTaskMem(_allocatedMemory);
}
}
} |
bartonjs
added
api-approved
ghost
added
the
in-pr
ghost
removed
the
in-pr
ghost
locked as resolved and limited conversation to collaborators
Background and motivation
There has been substantial feedback regarding the current
LibraryImportcustom marshaller shapes. The interop team has taken this feedback (internal and external), along with practical usage in the ASP.Net Core and WinForms repositories and decided to revise the current shapes. This revision is primarily designed to improve UX and future use cases with reverse P/Invoke (for example, COM source generation).This API would also apply to all previously approved marshaller APIs.
#66623
#66121
#68248
This will also apply to/supplant outstanding source generator proposals.
#69403
#69281
#69031
API Proposal
Added APIs
Removed APIs
API Usage
Alternative Designs
Instead of specifying the
BufferSizeas a static property on the marshaller type, developers could specify it on an attribute. We determined we did not want to go down this route as we're planning on moving to an interface-based approach as the primary usage scenario in .NET 8 and a static get-only property can be specified on an interface with static abstract members. This also frees us to make the value not a constant at compile time, and part of the API contract, which can be useful if we decide that a particular value needs to be tweaked for performance purposes.Instead of specifying the specific marshallers on the marshaller entry-point type with a single attribute and an enum, we also considered providing separate attributes that required explicitly specifying the marshaller type for each scenario. We decided that requiring the user to specify marshallers for every individual scenario was overly verbose and a bad user experience.
Risks
This is close to an RC and is a non-trivial update to the Marshaller shape. The ASP.Net Core and WinForms repos will need to respond as well. This is introducing complexity in the development slow down following an RC release in several weeks. There is no doubt risk here. A potential escape is to go with the current shape and "make it work". Given the internal/external feedback we have received the risk is worth it from the perspective of the Interop team.
The text was updated successfully, but these errors were encountered: