Add .mvid section to PE and remove dependency on MetadataReader (#19133)

docs/compilers/CSharp/CommandLine.md

@@ -4,6 +4,7 @@
 | ---- | ---- |
 | **OUTPUT FILES** |
 | `/out:`*file* | Specify output file name (default: base name of file with main class or first file)
+| `/refout:`*file* | Specify the reference assembly's output file name
 | `/target:exe` |  Build a console executable (default) (Short form: `/t:exe`)
 | `/target:winexe` | Build a Windows executable (Short form: `/t:winexe` )
 | `/target:library` | Build a library (Short form: `/t:library`)
@@ -32,6 +33,7 @@
 | `/debug`:{`full`|`pdbonly`|`portable`} | Specify debugging type (`full` is default, and  enables attaching a debugger to a running program. `portable` is a cross-platform format)
 | `/optimize`{`+`|`-`} | Enable optimizations (Short form: `/o`)
 | `/deterministic` | Produce a deterministic assembly (including module version GUID and timestamp)
+| `/refonly | Produce a reference assembly, instead of a full assembly, as the primary output 
 | **ERRORS AND WARNINGS**
 | `/warnaserror`{`+`|`-`} | Report all warnings as errors
 | `/warnaserror`{`+`|`-`}`:`*warn list* | Report specific warnings as errors

docs/compilers/Visual Basic/CommandLine.md

@@ -4,6 +4,7 @@
 | ---- | ---- |
 | **OUTPUT FILE**
 | `/out:`*file* | Specifies the output file name.
+| `/refout:`*file* | Specify the reference assembly's output file name
 | `/target:exe` | Create a console application (default). (Short form: `/t`)
 | `/target:winexe` | Create a Windows application.
 | `/target:library` | Create a library assembly.
@@ -34,6 +35,7 @@
 | `/debug:portable` | Emit debugging information in the portable format.
 | `/debug:pdbonly` | Emit PDB file only.
 | `/deterministic` | Produce a deterministic assembly (including module version GUID and timestamp)
+| `/refonly | Produce a reference assembly, instead of a full assembly, as the primary output 
 | **ERRORS AND WARNINGS**
 | `/nowarn` | Disable all warnings.
 | `/nowarn:`*number_list* | Disable a list of individual warnings.

docs/features/refout.md

@@ -32,11 +32,12 @@ Two mutually exclusive command-line parameters will be added to `csc.exe` and `v
 - `/refout`
 - `/refonly`
 
-The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below).
+The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below). The filename for the ref assembly should generally match that of the primary assembly, but it can be in a different folder.
 
 The `/refonly` parameter is a flag that indicates that a ref assembly should be output instead of an implementation assembly. 
 The `/refonly` parameter is not allowed together with the `/refout` parameter, as it doesn't make sense to have both the primary and secondary outputs be ref assemblies. Also, the `/refonly` parameter silently disables outputting PDBs, as ref assemblies cannot be executed. 
 The `/refonly` parameter translates to `EmitMetadataOnly` being `true`, and `IncludePrivateMembers` being `false` in the `Emit` API (see details below).
+Neither `/refonly` nor `/refout` are permitted with `/target:module` or `/addmodule` options.
 
 When the compiler produces documentation, the contents produced will match the APIs that go into the primary output. In other words, the documentation will be filtered down when using the `/refonly` parameter.
 
@@ -47,13 +48,14 @@ The `CoreCompile` target will support a new output, called `IntermediateRefAssem
 The `Csc` task will support a new output, called `OutputRefAssembly`, which parallels the existing `OutputAssembly`.
 Both of those basically map to the `/refout` command-line parameter.
 
-An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match, then the destination file is left untouched.
+An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match (by comparing their MVIDs, see details below), then the destination file is left untouched.
 
 ### CodeAnalysis APIs
 It is already possible to produce metadata-only assemblies by using `EmitOptions.EmitMetadataOnly`, which is used in IDE scenarios with cross-language dependencies.
 The compiler will be updated to honour the `EmitOptions.IncludePrivateMembers` flag as well. When combined with `EmitMetadataOnly` or a `metadataPeStream` in `Emit`, a ref assembly will be produced.
-The diagnostic check for emitting methods lacking a body (`void M();`) will be moved from declaration diagnostics to regular diagnostics, so that code will successfully emit with `EmitMetadataOnly`.
+The diagnostic check for emitting methods lacking a body (`void M();`) will be filtered from declaration diagnostics, so that code will successfully emit with `EmitMetadataOnly`.
 Later on, the `EmitOptions.TolerateErrors` flag will allow emitting error types as well.
+`Emit` is also modified to produce a new PE section called ".mvid" containing a copy of the MVID, when producing ref assemblies. This makes it easy for `CopyRefAssembly` to extract and compare MVIDs from ref assemblies.
 
 Going back to the 4 driving scenarios:
 1. For a regular compilation, `EmitMetadataOnly` is left to `false` and no `metadataPeStream` is passed into `Emit`.
@@ -70,7 +72,6 @@ As mentioned above, there may be further refinements after C# 7.1:
 - should explicit method implementations be included in ref assemblies?
 - Non-public attributes on public APIs (emit attribute based on accessibility rule)
 - ref assemblies and NoPia
-- `/refout` and `/addmodule`, should we disallow this combination?
 
 ## Related issues
 - Produce ref assemblies from command-line and msbuild (https://github.com/dotnet/roslyn/issues/2184)

src/Compilers/CSharp/Test/Emit/CSharpCompilerEmitTest.csproj

@@ -55,6 +55,9 @@
   <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|AnyCPU'" />
   <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|AnyCPU'" />
   <ItemGroup>
+    <Compile Include="..\..\..\Core\MSBuildTask\MvidReader.cs">
+      <Link>Emit\MvidReader.cs</Link>
+    </Compile>
     <Compile Include="Attributes\AttributeTests.cs" />
     <Compile Include="Attributes\AttributeTests_Assembly.cs" />
     <Compile Include="Attributes\AttributeTests_CallerInfoAttributes.cs" />

src/Compilers/CSharp/Test/Emit/Emit/CompilationEmitTests.cs

@@ -1,6 +1,7 @@
 // Copyright (c) Microsoft.  All Rights Reserved.  Licensed under the Apache License, Version 2.0.  See License.txt in the project root for license information.
 
 using System;
+using System.Collections.Generic;
 using System.Collections.Immutable;
 using System.IO;
 using System.Linq;
@@ -49,7 +50,6 @@ public void Main()
                 Diagnostic(ErrorCode.ERR_AssgLvalueExpected, "x"));
         }
 
-
         [Fact]
         public void CompilationEmitWithQuotedMainType()
         {
@@ -262,9 +262,11 @@ internal static void Main()
 
                 VerifyEntryPoint(output, expectZero: false);
                 VerifyMethods(output, new[] { "void C.Main()", "C..ctor()" });
+                VerifyMvid(output, hasMvidSection: false);
 
                 VerifyEntryPoint(metadataOutput, expectZero: true);
                 VerifyMethods(metadataOutput, new[] { "C..ctor()" });
+                VerifyMvid(metadataOutput, hasMvidSection: true);
             }
 
             void VerifyEntryPoint(MemoryStream stream, bool expectZero)
@@ -275,6 +277,110 @@ void VerifyEntryPoint(MemoryStream stream, bool expectZero)
             }
         }
 
+        private class TestResourceSectionBuilder : ResourceSectionBuilder
+        {
+            public TestResourceSectionBuilder()
+            {
+            }
+
+            protected override void Serialize(BlobBuilder builder, SectionLocation location)
+            {
+                builder.WriteInt32(0x12345678);
+                builder.WriteInt32(location.PointerToRawData);
+                builder.WriteInt32(location.RelativeVirtualAddress);
+            }
+        }
+
+        private class TestPEBuilder : ManagedPEBuilder
+        {
+            public static readonly Guid s_mvid = Guid.Parse("a78fa2c3-854e-42bf-8b8d-75a450a6dc18");
+
+            public TestPEBuilder(PEHeaderBuilder header,
+                MetadataRootBuilder metadataRootBuilder,
+                BlobBuilder ilStream,
+                ResourceSectionBuilder nativeResources)
+                : base(header, metadataRootBuilder, ilStream, nativeResources: nativeResources)
+            {
+            }
+
+            protected override ImmutableArray<Section> CreateSections()
+            {
+                return base.CreateSections().Add(
+                     new Section(".mvid", SectionCharacteristics.MemRead |
+                        SectionCharacteristics.ContainsInitializedData |
+                        SectionCharacteristics.MemDiscardable));
+            }
+
+            protected override BlobBuilder SerializeSection(string name, SectionLocation location)
+            {
+                if (name.Equals(".mvid", StringComparison.Ordinal))
+                {
+                    var sectionBuilder = new BlobBuilder();
+                    sectionBuilder.WriteGuid(s_mvid);
+                    return sectionBuilder;
+                }
+
+                return base.SerializeSection(name, location);
+            }
+        }
+
+        [Fact]
+        public void MvidSectionNotFirst()
+        {
+            var ilBuilder = new BlobBuilder();
+            var metadataBuilder = new MetadataBuilder();
+
+            var peBuilder = new TestPEBuilder(
+                PEHeaderBuilder.CreateLibraryHeader(),
+                new MetadataRootBuilder(metadataBuilder),
+                ilBuilder,
+                nativeResources: new TestResourceSectionBuilder());
+
+            var peBlob = new BlobBuilder();
+            peBuilder.Serialize(peBlob);
+
+            var peStream = new MemoryStream();
+            peBlob.WriteContentTo(peStream);
+
+            peStream.Position = 0;
+            using (var peReader = new PEReader(peStream))
+            {
+                AssertEx.Equal(new[] { ".text", ".rsrc", ".reloc", ".mvid" },
+                    peReader.PEHeaders.SectionHeaders.Select(h => h.Name));
+
+                peStream.Position = 0;
+                var mvid = BuildTasks.MvidReader.ReadAssemblyMvidOrEmpty(peStream);
+                Assert.Equal(TestPEBuilder.s_mvid, mvid);
+            }
+        }
+
+        /// <summary>
+        /// Extract the MVID using two different methods (PEReader and MvidReader) and compare them. 
+        /// We only expect an .mvid section in ref assemblies.
+        /// </summary>
+        private void VerifyMvid(MemoryStream stream, bool hasMvidSection)
+        {
+            stream.Position = 0;
+            using (var reader = new PEReader(stream))
+            {
+                var metadataReader = reader.GetMetadataReader();
+                Guid mvidFromModuleDefinition = metadataReader.GetGuid(metadataReader.GetModuleDefinition().Mvid);
+
+                stream.Position = 0;
+                var mvidFromMvidReader = BuildTasks.MvidReader.ReadAssemblyMvidOrEmpty(stream);
+
+                Assert.NotEqual(Guid.Empty, mvidFromModuleDefinition);
+                if (hasMvidSection)
+                {
+                    Assert.Equal(mvidFromModuleDefinition, mvidFromMvidReader);
+                }
+                else
+                {
+                    Assert.Equal(Guid.Empty, mvidFromMvidReader);
+                }
+            }
+        }
+
         [Fact]
         public void EmitRefAssembly_PrivatePropertySetter()
         {
@@ -296,6 +402,8 @@ public class C
                 VerifyMethods(output, new[] { "System.Int32 C.<PrivateSetter>k__BackingField", "System.Int32 C.PrivateSetter.get", "void C.PrivateSetter.set",
                     "C..ctor()", "System.Int32 C.PrivateSetter { get; private set; }" });
                 VerifyMethods(metadataOutput, new[] { "System.Int32 C.PrivateSetter.get", "C..ctor()", "System.Int32 C.PrivateSetter { get; }" });
+                VerifyMvid(output, hasMvidSection: false);
+                VerifyMvid(metadataOutput, hasMvidSection: true);
             }
         }
 
@@ -559,6 +667,20 @@ private void CompareAssemblies(string sourceTemplate, string change1, string cha
             {
                 AssertEx.NotEqual(image1, image2, message: $"Expecting difference for includePrivateMembers={includePrivateMembers} case, but they matched.");
             }
+
+            var mvid1 = BuildTasks.MvidReader.ReadAssemblyMvidOrEmpty(new MemoryStream(image1.DangerousGetUnderlyingArray()));
+            var mvid2 = BuildTasks.MvidReader.ReadAssemblyMvidOrEmpty(new MemoryStream(image2.DangerousGetUnderlyingArray()));
+
+            if (!includePrivateMembers)
+            {
+                Assert.NotEqual(Guid.Empty, mvid1);
+                Assert.Equal(expectMatch, mvid1 == mvid2);
+            }
+            else
+            {
+                Assert.Equal(Guid.Empty, mvid1);
+                Assert.Equal(Guid.Empty, mvid2);
+            }
         }
 
         [Fact]

src/Compilers/CSharp/Test/Emit/Emit/DeterministicTests.cs

@@ -18,14 +18,19 @@ namespace Microsoft.CodeAnalysis.CSharp.UnitTests.Emit
     public class DeterministicTests : EmitMetadataTestBase
     {
         private Guid CompiledGuid(string source, string assemblyName, bool debug)
+        {
+            return CompiledGuid(source, assemblyName, options: debug ? TestOptions.DebugExe : TestOptions.ReleaseExe);
+        }
+
+        private Guid CompiledGuid(string source, string assemblyName, CSharpCompilationOptions options, EmitOptions emitOptions = null)
         {
             var compilation = CreateCompilation(source,
                 assemblyName: assemblyName,
                 references: new[] { MscorlibRef },
-                options: (debug ? TestOptions.DebugExe : TestOptions.ReleaseExe).WithDeterministic(true));
+                options: options.WithDeterministic(true));
 
             Guid result = default(Guid);
-            base.CompileAndVerify(compilation, validator: a =>
+            base.CompileAndVerify(compilation, emitOptions: emitOptions, validator: a =>
             {
                 var module = a.Modules[0];
                 result = module.GetModuleVersionIdOrThrow();
@@ -104,6 +109,22 @@ public static void Main(string[] args) {}
             Assert.NotEqual(mvid3, mvid7);
         }
 
+        [Fact]
+        public void RefAssembly()
+        {
+            var source =
+@"class Program
+{
+    public static void Main(string[] args) {}
+    CHANGE
+}";
+            var emitRefAssembly = EmitOptions.Default.WithEmitMetadataOnly(true).WithIncludePrivateMembers(false);
+
+            var mvid1 = CompiledGuid(source.Replace("CHANGE", ""), "X1", TestOptions.DebugDll, emitRefAssembly);
+            var mvid2 = CompiledGuid(source.Replace("CHANGE", "private void M() { }"), "X1", TestOptions.DebugDll, emitRefAssembly);
+            Assert.Equal(mvid1, mvid2);
+        }
+
         const string CompareAllBytesEmitted_Source = @"
 using System;
 using System.Linq;

src/Compilers/Core/MSBuildTask/CopyRefAssembly.cs

@@ -2,8 +2,6 @@
 
 using System;
 using System.IO;
-using System.Reflection.Metadata;
-using System.Reflection.PortableExecutable;
 using Microsoft.Build.Framework;
 using Microsoft.Build.Utilities;
 
@@ -85,10 +83,8 @@ private bool Copy()
         private Guid ExtractMvid(string path)
         {
             using (FileStream source = File.OpenRead(path))
-            using (var reader = new PEReader(source))
             {
-                var metadataReader = reader.GetMetadataReader();
-                return metadataReader.GetGuid(metadataReader.GetModuleDefinition().Mvid);
+                return MvidReader.ReadAssemblyMvidOrEmpty(source);
             }
         }
     }

src/Compilers/Core/MSBuildTask/MSBuildTask.csproj

@@ -55,6 +55,7 @@
     </Compile>
     <Compile Include="AssemblyResolution.cs" />
     <Compile Include="CanonicalError.cs" />
+    <Compile Include="MvidReader.cs" />
     <Compile Include="CopyRefAssembly.cs" />
     <Compile Include="ValidateBootstrap.cs" />
     <Compile Include="CommandLineBuilderExtension.cs" />