Add Reference Assembly support - 'refonly' and 'refout' compiler options #12334

TIHan · 2021-11-04T18:50:24Z

Update (Vlad):
This PR is scoped to compiler changes only (i.e. --refonly and --refout flags) for producing reference assemblies. Changes related to FSharp.Build (sigdata, optdata and MVID generation) will be part of separate PR(s).

Summary:

General rules for emitting reference assemblies now are:
(note, general behaviour follows what Roslyn does)

Types are always emitted (even if they have no members).
Internal/Private fields are emitted if:

a. They are part of a struct type (because of "has default value" and "satisfies unmanaged constraint" semantics).
b. They are part of attribute specification.
Internal/Private properties and their corresponding get_/set_ methods are emitted if:

a. They are part of attribute specification.
Events are emitted if they have corresponding add_/remove_ methods.
We always skip private MethodDefs and skip internal ones if there's no InternalsVisibleTo attribute.
We always emit throw null for any emitted method bodies.

Description:

This PR is part of splitting https://github.com//pull/11521 - as we are trying to reduce the scope of the work. This PR is only meant for compiler output; not IDE support using metadata-only assemblies.

This adds the ability for the compiler to emit reference assemblies. See here for more info about reference assemblies; it has the rules.

Adds the compiler options, --refonly and --refout:<file>.

For the F# compiler, there are some unique quirks about generating reference assemblies:

Reference assembly with optimization data (requires running the optimizer on impl files)
- Used for actual compiler input when building a project.
- This is required because of inline functions that get serialized in F# metadata.
- This is what the compiler option, --refonly and --refout will do.
Static linking:
- You should not be allowed to static link a reference assembly, the compiler should give an error.
FSI:
- Reference assembly generation should not be allowed for FSI; should give an error.

User Benefits

The idea of the compiler emitting reference assemblies is to have faster solution build times when a small change has been made to a source file in a project that doesn't impact the overall signature.

PR Dependencies

Merge Added CI job for deterministic builds #12335
- Necessary for ensuring the compiler remains deterministic.

Acceptance Criteria

…ions and ones without

… the compiler understands Accessibility.

TIHan · 2022-02-21T15:59:58Z

src/fsharp/absil/il.fs

@@ -2122,10 +2122,11 @@ type ILTypeDef(name: string, attributes: TypeAttributes, layout: ILTypeDefLayout
    member _.MethodImpls = methodImpls
    member _.Events = events
    member _.Properties = properties
+    member _.IsAttribute = isAttribute


Storing isAttribute: bool might be a bit heavy weight for ILTypeDef as the question of "is the type an attribute" is answered before construction. Are we not able to determine this by looking at extends?

Thinking about it, looking at extends won't give you all the information of the hierarchy. Seems the only reliable way of determining if it's an attribute is computing it before creating ILTypeDef. So, probably storing isAttribute: bool is fine.

Just note that ILTypeDef tries to store data that is practically 1:1 with assembly metadata, which isAttribute: bool isn't a 1:1.

Because IsAttribute is only determined before writing an assembly, it does feel odd to have it on ILTypeDef as the type is for both reading and writing.

As a possible alternative, we could build some kind of storage in IlxGen that only hold the identities of types that are attributes. Then we can pass that storage to ilwrite in its cenv. That way those checks would look something like this:
if cenv.isAttrStorage.IsAttribute(td) ...

Thinking about it, looking at extends won't give you all the information of the hierarchy.

Yeah, that was a main reason of computing it for ILTypeDef.

Because IsAttribute is only determined before writing an assembly, it does feel odd to have it on ILTypeDef as the type is for both reading and writing.

As a possible alternative, we could build some kind of storage in IlxGen that only hold the identities of types that are attributes. Then we can pass that storage to ilwrite in its cenv. That way those checks would look something like this: if cenv.isAttrStorage.IsAttribute(td) ...

Let me experiment with.

TIHan · 2022-02-21T16:13:17Z

src/fsharp/absil/ilread.fs

@@ -1719,6 +1719,7 @@ and typeDefReader ctxtH: ILTypeDefStored =
                     methodImpls=mimpls,
                     events= events,
                     properties=props,
+                     isAttribute=false,


It looks like IsAttribute could only be set to true in IlxGen and only used for writing an assembly. This means every read ILTypeDef will have IsAttribute return false. We should probably add a comment to ILTypeDef.IsAttribute describing that it's only used for writing an assembly.

This means every read ILTypeDef will have IsAttribute return false.

Using a name like "IsDefinitelyAttribute" or "IsKnownToBeAttribute" may be clarifying

dsyme · 2022-03-01T15:49:10Z

@vzarytovskii Needs a merge :)

vzarytovskii · 2022-03-01T16:21:00Z

@vzarytovskii Needs a merge :)

Yeah, waiting until main is fixed, should be done after #12769

KevinRansom

Generally looks Okay, please take care of Will's and Don's suggestions before merging.

Thanks

KevinRansom · 2022-02-17T23:55:09Z

src/fsharp/FSComp.txt

@@ -1168,6 +1170,8 @@ fscTooManyErrors,"Exiting - too many errors"
 2026,fscDeterministicDebugRequiresPortablePdb,"Deterministic builds only support portable PDBs (--debug:portable or --debug:embedded)"
 2027,fscPathMapDebugRequiresPortablePdb,"--pathmap can only be used with portable PDBs (--debug:portable or --debug:embedded)"
 2028,optsInvalidPathMapFormat,"Invalid path map. Mappings must be comma separated and of the format 'path=sourcePath'"
+2029,optsInvalidRefOut,"Invalid reference assembly path'"
+2030,optsInvalidRefAssembly,"Invalid use of emitting a reference assembly. Check the compiler options to not specify static linking, or using '--refonly' and '--refout' together."


@vzarytovskii
This should read something like:

Invalid use of emitting a reference assembly, do not use --staticlink, or '--refonly and --refout' together.

…-part1

vzarytovskii · 2022-06-01T13:43:49Z

The rest of work is tracked in #13222 and #13223

Generation of reference assemblies for F# was disabled previously (dotnet#13085) because the F# compiler didn't support them. Now both, the F# compiler (dotnet/fsharp#12334) and the FCS task (dotnet/fsharp#13263) support reference assemblies. Also, the bug around the copying of F# ref assemblies was fixed (dotnet/fsharp#13266) so it is time to enable the generation of ref assemblies for F#.

Generation of reference assemblies for F# was disabled previously (dotnet#13085) because the F# compiler didn't support them. Now both, the F# compiler (dotnet/fsharp#12334) and the FCS task (dotnet/fsharp#13263) support reference assemblies. Also, the bug around the copying of F# ref assemblies was fixed (dotnet/fsharp#13266) so it is time to enable the generation of ref assemblies for F#.s

…ons (dotnet#12334) * Adding 'refonly' command line option * Added a simple test, but it needs to fail * We need to emit two kinds of reference assemblies. one with optimizations and ones without * Passing reference assembly flag to IlxGen * Emit ReferenceAssemblyAttribute * Added ref-assembly rules for private and internal methods * use --refonly for now * Use HasFSharpAttribute * Added a failing test * Test passes * Trying to handle anonymous record types * Cleaning up. Using ILMemberAccess instead of Accessibility due to how the compiler understands Accessibility. * Using notlazy * Added another comment * Added mkDummyParameterVal * Using taccessPublic * More cleanup * Minor comment update * more cleanup * Adding FreeAnonRecdTypeInfos set * Adding options * Flowing free anonrecdtypeinfos * Fixing build * Tests pass. Able to emit partial ref assembly with anon recds * Minor rename * Added a failing test * Added failing test * Simpler handling of building lambdas * Trying to figure out default param names * Adding TryEmitReferenceAssembly * Moving some reference assembly generation rules to ilwrite * Fixing build * Added new compiler option '--refout:<file>' * Fixing one of the tests * refonly/refout should only be part of fsc * Updating help baseline * fixed build * Fixing build. Added basic deterministic test * Failing determinism test * Added DeterministicTests * Adding determinism task for CI * moving yml to pipelines * Trying to fix determinism CI * quick fix * removing job * Trying to fix ci * Removing this * Turn on determinism for build * Trying to fix * This works * Determinism * Building * Forgot to run test * Adding job * Trying to fix job * Remove job * Trying to figure out jobs * Updating job * Fixing determinism job * Fixing job * Update test-determinism.ps1 * Update FSharp.Profiles.props quick test to see if determinism CI breaks when deterministic flag is off, it should * Update test-determinism.ps1 * Update FSharpBuild.Directory.Build.props * Trying to fix build * Trying to fix build * fixing build * Fixing build * fixing build * Fixing build * Remove comment as it is not accurate * Removed generating metadata assembly for IDEs * Fixing build * Removing tests * Update ParseAndCheckInputs.fs * Update TypedTree.fs * Fixing build * Update TypedTreeOps.fs * Fixing build * Fixing build * Fixing build * Fixing build * Update baseline for fcs 'help' test * Added a test for '--refout', with outout and IL verification * Added tests to verify that static linking and refassemblies cannot be used together * Add mvid test for refonly + private members. It is failing on purpose, until MVID generation is fixed * WIP: Add some more to the tests * Added more tests for MVID * wip * Added some todos + have more readable canGenMethodDef * Add some more tests * [WIP]: ignore properties if we don't have getter/setter, or we don't have methoddef for them * Don't generate private types, generate nested internal types only if the IVT is set * Merge fix * Another fix after merge + added more internal tests * Fixed test framework after merge (output directory). Add check whether we can generate fields (based on IVT and access). Fixed IVT attribute check (check in manifest). Disabled some tests temporary/ * Emit fields when the type is struct. Always emit types * WIP: added isAttribute to ILTypeRef if type extends Attribute * Fix properties generation, fix generating getter/setter for attributes * Only check properties to generate if we are emitting reference assembly * Fixed surface area tests * Adjusted baselines for IL tests. Fixed events generation. * Cleanup unused yaml files * Fixed docs for ILMemberAccess * Update message + rename property for ILTypeDef to be more clear * Surface area tests * Fixed baseline error message * After-merge fixes * Fix tests Co-authored-by: Vlad Zarytovskii <[email protected]> Co-authored-by: Don Syme <[email protected]>

kerams · 2023-10-26T14:31:33Z

Types are always emitted (even if they have no members).

I know Roslyn does this too, but why? Neither https://blog.paranoidcoding.org/2016/02/15/are-private-members-api-surface.html nor https://github.com/dotnet/roslyn/blob/main/docs/features/refout.md expand on that point.

You can surface non-public types by using them in attributes:

type private Y = interface end

type C (_t: Type) =
    inherit Attribute ()

[<C(typeof<Y>)>]
type X = interface end

While removing Y could make an observable difference as far as the API is concerned, I don't see how this affects the compilation of code that uses X. Could we, at the very least, remove internal compiler-generated types like state machine structs, static initializers and closures? In my quick spike I was able to reduce reference assembly sizes by almost 50% by pruning private and internal IL types (when IVT is not involved), without even touching sig and opt data.

vzarytovskii · 2023-10-26T15:31:26Z

I know Roslyn does this too, but why?

I don't exactly remember why now. I can dig into my notes it if you'd like to (they're on my old laptop though, so might take some time).

vzarytovskii · 2023-10-26T15:42:57Z

So, C#, apparently, doesn't trim any types from the reference assembly, not even closure types which are resulting from trimmed method implementations. Public fields of private types are not removed, including implementation details such as the fields of closure types, or the static field used to save the singleton delegate for a closure such as (new Func<int, int>(X => X + 1)).

@jaredpar do you know the reason for that? Is it for constraint resolutions?

Struct types (and their fields) shouldn't probably be touched, since they are used in constraints (has default value, unmanaged constraints).

Attributes are an interesting case. Consider an attribute like this:

[AttributeUsage(AttributeTargets.All)]
internal class FooAttribute : Attribute
{
   internal int InternalFieldInInternalClass;
}

C# can't use it as an attribute, F# can use the equivalent type as an attribute.

C#, in this case, can skip emitting internal fields of classes from the reference assembly. F#, it wouldn't be valid to do this for attribute types unless we made a change to the language spec.

kerams · 2023-10-26T15:55:38Z

I'm not bothered by always keeping internal/private attribute classes - they are so rare compared to other types that dropping them would make little difference.

TIHan added 30 commits May 3, 2021 18:04

Adding 'refonly' command line option

2411f75

Added a simple test, but it needs to fail

1b6dc26

We need to emit two kinds of reference assemblies. one with optimizat…

ee89986

…ions and ones without

Passing reference assembly flag to IlxGen

9dcdbb0

Emit ReferenceAssemblyAttribute

84cd6a8

Added ref-assembly rules for private and internal methods

3a0a7be

use --refonly for now

5d78b77

Use HasFSharpAttribute

2f1af9b

Added a failing test

512743c

Test passes

4af859b

Trying to handle anonymous record types

6b5c032

Cleaning up. Using ILMemberAccess instead of Accessibility due to how…

84354ba

… the compiler understands Accessibility.

Using notlazy

6d93c92

Added another comment

c31ce72

Added mkDummyParameterVal

0d02203

Using taccessPublic

8f5cbf5

More cleanup

daf1847

Minor comment update

9d0c7f6

more cleanup

47d3b5a

merging

5e4460e

Adding FreeAnonRecdTypeInfos set

3468094

Adding options

23d0da7

Flowing free anonrecdtypeinfos

8a1504f

Fixing build

d7e491e

Tests pass. Able to emit partial ref assembly with anon recds

7ff20d6

Minor rename

a023c38

Added a failing test

6801573

Added failing test

91dbaa0

Simpler handling of building lambdas

4931062

Trying to figure out default param names

e5c1cc8

TIHan commented Feb 21, 2022

View reviewed changes

Fixed docs for ILMemberAccess

f1a7ff4

KevinRansom approved these changes Mar 1, 2022

View reviewed changes

vzarytovskii added 4 commits March 23, 2022 18:45

Merge 'upstream/main'

2b7a114

Update message + rename property for ILTypeDef to be more clear

033a112

Surface area tests

1b4ff9d

Fixed baseline error message

74ff369

vzarytovskii force-pushed the ref-assembly-output-part1 branch from d035444 to 74ff369 Compare March 31, 2022 21:33

vzarytovskii closed this Apr 1, 2022

vzarytovskii reopened this Apr 1, 2022

vzarytovskii added 3 commits April 20, 2022 15:13

Merge remote-tracking branch 'upstream/main' into ref-assembly-output…

7ba8563

…-part1

After-merge fixes

51e9d5a

Fix tests

897d9c1

vzarytovskii merged commit 76432f1 into dotnet:main Apr 20, 2022

nojaf mentioned this pull request May 31, 2022

CopyRefAssembly Could not extract the MVID from reference assembly #13216

Closed

vzarytovskii mentioned this pull request Jun 1, 2022

Generate deterministic MVID for reference assemblies based on contents of the assembly #13222

Closed

marcin-krystianc mentioned this pull request Jun 28, 2022

Removing default ProduceReferenceAssembly value for F# dotnet/sdk#26306

Merged

vzarytovskii mentioned this pull request Oct 12, 2022

F# records in reference assembly (/ref folder) are missing properties except for the last record in file [regression with 7.0.100-rc.2.22477.23 SDK ] #14088

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Reference Assembly support - 'refonly' and 'refout' compiler options #12334

Add Reference Assembly support - 'refonly' and 'refout' compiler options #12334

TIHan commented Nov 4, 2021 •

edited by vzarytovskii

Loading

TIHan Feb 21, 2022 •

edited

Loading

TIHan Feb 21, 2022

TIHan Feb 21, 2022

TIHan Feb 21, 2022

vzarytovskii Feb 28, 2022

TIHan Feb 21, 2022

dsyme Mar 1, 2022

dsyme commented Mar 1, 2022

vzarytovskii commented Mar 1, 2022

KevinRansom left a comment

KevinRansom Feb 17, 2022

vzarytovskii commented Jun 1, 2022

kerams commented Oct 26, 2023 •

edited

Loading

vzarytovskii commented Oct 26, 2023

vzarytovskii commented Oct 26, 2023 •

edited

Loading

kerams commented Oct 26, 2023 •

edited

Loading

Add Reference Assembly support - 'refonly' and 'refout' compiler options #12334

Add Reference Assembly support - 'refonly' and 'refout' compiler options #12334

Conversation

TIHan commented Nov 4, 2021 • edited by vzarytovskii Loading

TIHan Feb 21, 2022 • edited Loading

Choose a reason for hiding this comment

TIHan Feb 21, 2022

Choose a reason for hiding this comment

TIHan Feb 21, 2022

Choose a reason for hiding this comment

TIHan Feb 21, 2022

Choose a reason for hiding this comment

vzarytovskii Feb 28, 2022

Choose a reason for hiding this comment

TIHan Feb 21, 2022

Choose a reason for hiding this comment

dsyme Mar 1, 2022

Choose a reason for hiding this comment

dsyme commented Mar 1, 2022

vzarytovskii commented Mar 1, 2022

KevinRansom left a comment

Choose a reason for hiding this comment

KevinRansom Feb 17, 2022

Choose a reason for hiding this comment

vzarytovskii commented Jun 1, 2022

kerams commented Oct 26, 2023 • edited Loading

vzarytovskii commented Oct 26, 2023

vzarytovskii commented Oct 26, 2023 • edited Loading

kerams commented Oct 26, 2023 • edited Loading

TIHan commented Nov 4, 2021 •

edited by vzarytovskii

Loading

TIHan Feb 21, 2022 •

edited

Loading

kerams commented Oct 26, 2023 •

edited

Loading

vzarytovskii commented Oct 26, 2023 •

edited

Loading

kerams commented Oct 26, 2023 •

edited

Loading