ref field support in .NET runtimes

[AaronRobinsonMSFT]

There are several moving parts to support ref fields, this issue represents work in the actual runtime.

The high-level tasks for the runtime are below to address the ref fields work.

• CoreCLR support
  • Port specific work from Hackathon project – https://github.com/dotnet/runtime/tree/feature/lowlevelhackathon
  • Add runtime support for ref fields #63985
  • Fix layout and GCInfo for ByRef fields. #64422
  • Prevent placing ref fields on non-pointer boundaries #64520
  • Validate IntPtr and ByRef field overlap #65246
• Mono support
  • Add runtime support for ref fields #63985
  • Prevent placing ref fields on non-pointer boundaries #64520
  • Validate IntPtr and ByRef field overlap #65246
  • Update tests to help uncover CrossGen2 scenario validation #65477
• API Proposal: Add ref field feature support to RuntimeFeature class #64165
• Audit handling of null byrefs in runtime and libraries Box helpers do not handle null-valued managed pointers #70268
• Tooling support
  • GenAPI needs to account for ref fields  #63751
  • [API Proposal]: Support ref fields in System.Reflection.Metadata.Ecma335.BlobEncoder #68309
  • Validate Linker support. Validate that linker correctly handles ref fields linker#2519
• ECMA update – Proposed changes to ECMA 335 for ref field support #63659
• Delete the ByReference<T> type and replace its use with ref fields.
  • Consume Roslyn with ref fields support #71498
• Diagnostics – Ensure SOS and VS can handle these new semantics.
  • Update SOS to resolve first level byref fields diagnostics#2829
  • Ensure ClrInstrumentationEngine handles ref fields microsoft/CLRInstrumentationEngine#464
  • GetTypeFields should return ELEMENT_TYPE_BYREF. #65880
  • EE doesn't display ref fields as expected. roslyn#62157
• C# test cases – Comprehensive collection of testing for the runtime.
• Update CrossGen
  • Add support for byref fields in NativeAOT/crossgen2 #64366
  • Fix layout and GCInfo for ByRef fields. #64422
  • Update tests to help uncover CrossGen2 scenario validation #65477
• Update NativeAOT – CoreCLR and Mono.
  • Add support for byref fields in NativeAOT/crossgen2 #64366
  • Update tests to help uncover CrossGen2 scenario validation #65477
• Language support - Merge features/ref-fields roslyn#62155
  • Low level struct improvements
  • Span<T> safety
  • Proposal for sunsetting restricted types when targeting net7

This contributes to enabling #45152.

[AaronRobinsonMSFT]

/cc @mangod9 @jaredpar @davidwrighton @steveharter @lambdageek @dotnet/interop-contrib @fanyang-mono

[jkotas]

Remove specialized type handling in runtime – Make ArgIterator, RuntimeArgumentHandle, and TypedReference “just types”.

I am not sure what this is about. The runtime does not have any special handling of these types. (Roslyn does - but you have it under separate check box.)

Also in the list should be: Delete the internal ByReference<T> type and replace its use with ref fields.

[jaredpar]

Remove specialized type handling in runtime – Make ArgIterator, RuntimeArgumentHandle, and TypedReference “just types”.

I believe @davidwrighton believes there is some spec work necessary before we can remove the notion of special types. Likely I think we will also need a feature flag in corelib to identify runtimes where Roslyn is safe to remove the notion of restricted types and depend solely on the type definition being correct. But agree bulk of the work here is in Roslyn.

[AaronRobinsonMSFT]

/cc @tommcdon @noahfalk for Diagnostics impact.

[vitek-karas]

We may possibly need dotnet/linker and mono/cecil work - it needs to be able to roundtrip assembly with ref fields correctly. I know it's not specifically about runtime - but it will be needed in order to use it in the runtime (since we run the code through linker during build).

[AaronRobinsonMSFT]

We may possibly need dotnet/linker and mono/cecil work

This was brought up in the initial planning meeting. If I recall, @davidwrighton wasn't sure there should be any impact.

[vitek-karas]

I created dotnet/linker#2519 to track the validation on the linker side - at the very least we should add some tests for this in the linker.

[John0King]

How big is the size of a ref? if it is fixed like 64bit , then a ref struct contains ref itself should also be legal like class

see
https://sharplab.io/#v2:EYLgZgpghgLgrgJwgZwLRLKsBLCAbAE2QBoCQBqAHwAEAmARgFgAoFl6gZgAIMvkYEcAMYwuAFSQQAcgHsCEFgG8WXVVwAOCbADdYEHhDDjJs+VwAyh0VwDcKtSwC+bVs05c6xiNLkQAwnhQyMhK9qruqhLepv6BwRZWigDmEDA2yKk2zsxqXGEe3JEmvgFByFwASthJABYwyZkZadnZLEA=

[svick]

@John0King That is likely going to be fixed, see dotnet/roslyn#62098.

[hamarb123]

Just a question, if ByReference<T> is removed and in the future we allow things like Span<Span<int>>, would something like Span<ref int> be allowed or would someone need to create their own ByReference type to use like so Span<ByReference<int>>.

[AaronRobinsonMSFT]

would something like Span be allowed or would someone need to create their own ByReference type to use like so Span<ByReference>.

At present, users would need to create Span<ByReference<int>>. The ref in a generic suffers from the same issues as pointer types (for example, int*).

[hamarb123]

At present, users would need to create Span<ByReference<int>>. The ref in a generic suffers from the same issues as pointer types (for example, int*).

I hope this gets implemented in the future. I think a good solution would be to make a new generic constraint that allows these things (e.g. allow pointer, allow ref, allow ref struct, allow any other appropriate types, allow all, which could be expressed through a modopt or attribute), which doesn't allow boxing i.e. no calls to methods on object (ToString, GetHashCode etc., GetType of some form should still exist though). This would still allow many operations to work, just not the ones that wouldn't necessarily make sense for the types specified as allowed.

[jaredpar]

At present, users would need to create Span<ByReference>. The ref in a generic suffers from the same issues as #13627.

I hope this gets implemented in the future. I think a good solution would be to make a new generic constraint that allows these things (e.g. allow pointer, allow ref, allow ref struct, allow any other appropriate types, allow all, which could be expressed through a modopt or attribute), which doesn't allow boxing i.e. no calls to methods on object (ToString, GetHashCode etc., GetType of some form should still exist though)

The issue goes deeper. In order for a generic anti-constraint, like allow T: ref struct, to be useful you also need to support ref fields to ref struct in the span safety rules. That has a number of unintuitive behaviors associated with it as well as forcing us to add yet another layer of ref safety into the ruleset. A few of the challenges are described here.

It's certainly doable but it does come with a significant complexity cost.

At the core I think the following three features are tied together:

• ref fields to ref struct
• ref struct as generic arguments
• ref struct implementing interfaces

Doing one pretty much pulls you into doing the other two very quickly.

[mangod9]

@AaronRobinsonMSFT are there items pending for 7 or does this need to be moved to 8?

[AaronRobinsonMSFT]

Nope, this is done. Tests that can/should be in C# have been converted.

[vitek-karas]

We haven't look at linker yet though... dotnet/linker#2519
@agocke

[AaronRobinsonMSFT]

Shoot. That wasn't a checkbox :(

[mangod9]

is this expected to be complete this week?

[vitek-karas]

Yes - @jtschuster is working on it in the linker repo right now.

[John0King]

ref itself seem's being cut ? eg.

public struct TreeNode
{
    public ref TreeNode Left;
    public ref TreeNode Right;

    ....

}

[hamarb123]

@John0King , ref fields should be in a ref struct, and is only .NET 7+, C# 11+

[John0King]

@hamarb123 so there still no way to write a BineryTree or other Tree in struct ?

[hamarb123]

(As far as I can tell) No, because a ref field must be in ref struct, but a ref field cannot refer to a ref struct. If your value type is a unmanaged struct, then you could stackalloc some memory and manage everything manually (i.e. create a new allocator like malloc for that segment of memory, re-interpretation of memory for the value type, 'pointers' to other values which could just be an offset from the start etc.) then you could possibly create something of use, but it seems like a lot of effort to me and has the issue of the stackalloc'd memory running out, but the ref field wouldn't really help there and this could be done before already.

[John0King]

I think the ref of the field is not the same size of the struct, (I understand the ref is like pointer , it should be 32/64 bit ) so a struct has a ref or another struct of it's type is

(in byte)
----------------
ref  file a           int filed b    ref field c
0 1 2 3 4 5 6 7      0 1 2 3        0  1 2 3 4 5 6 7

[hamarb123]

ref fields are indeed implemented as managed pointers (which are just pointers that move with the GC). The reason that you can't stick them in any type (such as a struct that is not a ref struct) is due to the span safety rules, if you want to read in-depth what they are and why they are what they are see here https://github.com/dotnet/csharplang/blob/main/proposals/low-level-struct-improvements.md. ref fields to ref structs are planned for C# 12 / .NET 8 currently.