[API Proposal]: System.Runtime.CompilerServices.Unsafe.AsPtrRef<T, U>(ref T* source) : ref U

[rickbrew]

Background and motivation

I'm told that something similar to this has been suggested before but it was decided that it was "too niche."

Well, I'm finding that I could really use this! Without it, it's impossible to do certain operations on variables or fields that are pointers. No Interlocked.Exchange, no nothing. Generics don't work with pointers, but they do work with pointer-size structs that simply wrap the pointer (e.g. Ptr<T> which is just a struct { T* p; } plus all the constraints and casting operators you'd expect).

From what I can tell, this is the key method that's needed to bridge the gap between pointers and generics. No need for adding many other pieces of support in the language, compiler, runtime, etc. Just let us temporarily reinterpret a T* as a Ptr<T> or IntPtr to enable performing an operation that works fine on that type.

In interop code, it's more idiomatic and less kludgey to just use pointers, and it's unsavory to have to switch everything to pointer-wrapping structs just because pointers aren't compatible with many things in the compiler or the framework. Once you have a field or variable that's a T* you are completely locked out of important operations that are idiomatic in native/interop code. You just can't break the pointer out of its jail without some really weird hacks that the JIT optimizer likely doesn't stand a chance against (someone found a crazy way using function pointers to accomplish this, but because of that it involves a non-inlinable method call).

With this I can do Interlocked operations on pointers, I can do Volatile.Read() and Volatile.Write() on pointers, etc. I can reinterpret an IUnknown* to a ComPtr<IUnknown> (from TerraFX). And as long as the inverse method is available, I can convert back to pointers when needed. Having to sandwich these with Unsafe calls is also unsavory, but par for the course when working heavily with interop code and Unsafe.

With help from others (esp. @jakobbotsch) I was able to get a prototype of this and it does work. Not having this in the runtime is very inconvenient, but not completely blocking, as I could create a nuget package. However, having an assembly and nuget package for the sake of 1 method is a little heavy.

cc @Sergio0694 @jakobbotsch @tannergooding who were part of the discussion in Discord

API Proposal

namespace System.Runtime.CompilerServices
{
    public static class Unsafe
    {
        // This name was chosen in part so it does not sort next to other methods like AsRef, 
        // therefore less probability it could be accidentally used (via auto-complete or etc.).
        // sizeof(U) must equal sizeof(T*)
        public static ref U AsPtrRef<T, U>(ref T* source)
            where T : unmanaged
            where U : unmanaged

        // The inverse operation is needed as well, to convert from ref U back to ref T*
        public static ref U* AsPtrRef<T, U>(ref T source)
            where T : unmanaged
            where U : unmanaged

        // Might want to have `ref T**` versions as well, up to a reasonable arity. `ref T***` perhaps, but `ref T*******` is a bit much. `T***` does _very occasionally_ pop up in native interop code (pointer to 2-dimensional array).
    }
}

The IL for this is pretty straightforward, it's just ldarg.0 and ret, along with attributes to tag T and U as unmanaged. I did manage to get a working version of this with the help of @jakobbotsch

  .method public hidebysig static !!U&  AsPtrRef<valuetype .ctor (class [System.Runtime]System.ValueType modreq([System.Runtime.InteropServices]System.Runtime.InteropServices.UnmanagedType)) T,valuetype .ctor (class [System.Runtime]System.ValueType modreq([System.Runtime.InteropServices]System.Runtime.InteropServices.UnmanagedType)) U>(!!T*& p) cil managed
  {
    .param type T 
      .custom instance void System.Runtime.CompilerServices.IsUnmanagedAttribute::.ctor() = ( 01 00 00 00 ) 
    .param type U 
      .custom instance void System.Runtime.CompilerServices.IsUnmanagedAttribute::.ctor() = ( 01 00 00 00 ) 
    // Code size       2 (0x2)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  ret
  } // end of method AsPtrRef

API Usage

public static unsafe T* InterlockedExchangeHelper<T>(ref T* p, T* newValue) 
    where T : unmanaged
{
    return (T*)Interlocked.Exchange(ref Unsafe.AsPtrRef<T, IntPtr>(ref p), (IntPtr)newValue);
}

public class MyComWrapper 
    : IDisposable
{
    private IUnknown* pObject;

    public MyComWrapper(IUnknown* pObject)
    {
        this.pObject = pObject;
        pObject->AddRef();
    }

    ~MyComWrapper()
    {
        Dispose(false);
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    private void Dispose(bool disposing)
    {
        // can't do this, as the `T` can't be a pointer
        //IUnknown* pObject = Interlocked.Exchange(ref this.pUnknown);

        // but this will work
        IUnknown* pObject = InterlockedExchangeHelper(ref this.pObject, null);

        if (pObject != null)
        {
            pObject->Release();
        }        
    }
}

Alternative Designs

No response

Risks

The naming of the method needs to be chosen very carefully.

If the non-pointer type being converted to/from is not at least pointer-sized, bad things can happen. But, this is Unsafe.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[Sergio0694]

I mean of course I like this proposal, a couple of small questions though 😄

1. Couldn't the first one just be an Unsafe.As overload, given it has a different parameter type?

namespace System.Runtime.CompilerServices
{
    public static class Unsafe
    {
        public static ref U As<T, U>(ref T* source)
             where T : unmanaged
             where U : unmanaged;
    }
}

2. I'd argue the second one should be called AsPointerRef for consistency:

namespace System.Runtime.CompilerServices
{
    public static class Unsafe
    {
        public static ref U* AsPointerRef<T, U>(ref T source)
            where T : unmanaged
            where U : unmanaged;
    }
}

Other than that I agree this would unblock several scenarios while language support isn't there.
That said, it'd still be nice to just have language support for pointers as generics in the long run (#13627).

Additionally, worth mentioning, @AndyAyersMS is looking into letting the JIT inline function pointer calls, which would allow doing this (or, defining these APIs) entirely in C# with no need for new APIs, with the same final codegen). They'd be:

public static ref U As<T, U>(ref T* source)
     where T : unmanaged
     where U : unmanaged
 {
     return ref
         ((delegate*<ref T*, ref U>)
          (delegate*<ref byte, ref byte>)
          &Unsafe.As<byte, byte>)(ref source);
 }

public static ref U* AsPointerRef<T, U>(ref T source)
    where T : unmanaged
    where U : unmanaged
{
    return ref
         ((delegate*<ref T, ref U*>)
          (delegate*<ref byte, ref byte>)
          &Unsafe.As<byte, byte>)(ref source);
}

[rickbrew]

@Sergio0694 I'm fine with whatever the chosen name is :)

[ghost]

Tagging subscribers to this area: @dotnet/area-system-runtime-compilerservices
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

I'm told that something similar to this has been suggested before but it was decided that it was "too niche."

Well, I'm finding that I could really use this! Without it, it's impossible to do certain operations on variables or fields that are pointers. No Interlocked.Exchange, no nothing. Generics don't work with pointers, but they do work with pointer-size structs that simply wrap the pointer (e.g. Ptr<T> which is just a struct { T* p; } plus all the constraints and casting operators you'd expect).

From what I can tell, this is the key method that's needed to bridge the gap between pointers and generics. No need for adding many other pieces of support in the language, compiler, runtime, etc. Just let us temporarily reinterpret a T* as a Ptr<T> or IntPtr to enable performing an operation that works fine on that type.

In interop code, it's more idiomatic and less kludgey to just use pointers, and it's unsavory to have to switch everything to pointer-wrapping structs just because pointers aren't compatible with many things in the compiler or the framework. Once you have a field or variable that's a T* you are completely locked out of important operations that are idiomatic in native/interop code. You just can't break the pointer out of its jail without some really weird hacks that the JIT optimizer likely doesn't stand a chance against (someone found a crazy way using function pointers to accomplish this, but because of that it involves a non-inlinable method call).

With this I can do Interlocked operations on pointers, I can do Volatile.Read() and Volatile.Write() on pointers, etc. I can reinterpret an IUnknown* to a ComPtr<IUnknown> (from TerraFX). And as long as the inverse method is available, I can convert back to pointers when needed. Having to sandwich these with Unsafe calls is also unsavory, but par for the course when working heavily with interop code and Unsafe.

With help from others (esp. @jakobbotsch) I was able to get a prototype of this and it does work. Not having this in the runtime is very inconvenient, but not completely blocking, as I could create a nuget package. However, having an assembly and nuget package for the sake of 1 method is a little heavy.

cc @Sergio0694 @jakobbotsch @tannergooding who were part of the discussion in Discord

API Proposal

namespace System.Runtime.CompilerServices
{
    public static class Unsafe
    {
        // This name was chosen in part so it does not sort next to other methods like AsRef, 
        // therefore less probability it could be accidentally used (via auto-complete or etc.).
        // sizeof(U) must equal sizeof(T*)
        public static ref U AsPtrRef<T, U>(ref T* source)
            where T : unmanaged
            where U : unmanaged

        // The inverse operation is needed as well, to convert from ref U back to ref T*
        public static ref U* AsPtrRef<T, U>(ref T source)
            where T : unmanaged
            where U : unmanaged

        // Might want to have `ref T**` versions as well, up to a reasonable arity. `ref T***` perhaps, but `ref T*******` is a bit much. `T***` does _very occasionally_ pop up in native interop code (pointer to 2-dimensional array).
    }
}

The IL for this is pretty straightforward, it's just ldarg.0 and ret, along with attributes to tag T and U as unmanaged. I did manage to get a working version of this with the help of @jakobbotsch

  .method public hidebysig static !!U&  AsPtrRef<valuetype .ctor (class [System.Runtime]System.ValueType modreq([System.Runtime.InteropServices]System.Runtime.InteropServices.UnmanagedType)) T,valuetype .ctor (class [System.Runtime]System.ValueType modreq([System.Runtime.InteropServices]System.Runtime.InteropServices.UnmanagedType)) U>(!!T*& p) cil managed
  {
    .param type T 
      .custom instance void System.Runtime.CompilerServices.IsUnmanagedAttribute::.ctor() = ( 01 00 00 00 ) 
    .param type U 
      .custom instance void System.Runtime.CompilerServices.IsUnmanagedAttribute::.ctor() = ( 01 00 00 00 ) 
    // Code size       2 (0x2)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  ret
  } // end of method AsPtrRef

API Usage

public static unsafe T* InterlockedExchangeHelper<T>(ref T* p, T* newValue) 
    where T : unmanaged
{
    return (T*)Interlocked.Exchange(ref Unsafe.AsPtrRef<T, IntPtr>(ref p), (IntPtr)newValue);
}

public class MyComWrapper 
    : IDisposable
{
    private IUnknown* pObject;

    public MyComWrapper(IUnknown* pObject)
    {
        this.pObject = pObject;
        pObject->AddRef();
    }

    ~MyComWrapper()
    {
        Dispose(false);
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    private void Dispose(bool disposing)
    {
        // can't do this, as the `T` can't be a pointer
        //IUnknown* pObject = Interlocked.Exchange(ref this.pUnknown);

        // but this will work
        IUnknown* pObject = InterlockedExchangeHelper(ref this.pObject, null);

        if (pObject != null)
        {
            pObject->Release();
        }        
    }
}

Alternative Designs

No response

Risks

The naming of the method needs to be chosen very carefully.

If the non-pointer type being converted to/from is not at least pointer-sized, bad things can happen. But, this is Unsafe.

Author:	rickbrew
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.CompilerServices, untriaged
Milestone:	-

[rickbrew]

cc @DaZombieKiller who was also part of the discussion in Discord

[rickbrew]

For the time being (or longer), I've published a NuGet package that enables this sort of thing, https://github.com/rickbrew/PointerToolkit/ . I use InlineIL.Fody to generate the methods.