[Umbrella] Compilers should be deterministic: same inputs generate same outputs

[gafter]

This is an umbrella issue for making the Roslyn compilers deterministic. See also Open Issues for Determinism.

There are a few issues:

• EditAndContinueTests.AnonymousTypes changes emit order in 64-bit runner #223 Anonymous types should be output in a deterministic order.
• String heap in generated IL should be deterministic #360 The string heap should be output in a deterministic order.
• Decimal parsing ignores significant digits below 1e-49 #375 Decimal parsing reflects bugs in the BCL; fixed in Core 3.
• Order of symbols provided by DataFlowAnalysis should be deterministic #803 The order of symbols produced from the DataFlowAnalysis API should be deterministic.
• Produce PDBs with deterministic GUIDs #926 Produce PDBs with deterministic GUIDs and timestamps by default. This requires a CLR spec change; see https://github.com/dotnet/coreclr/issues/1615. The CLR spec has been agreed internally and implemented in Roslyn, but the new spec has not been published and this is not enabled by default.
• PDB format and caller file name arguments include full path names in compiler output #949 Produce only relative paths in caller file name arguments and in the PDB
• Unable to build assembly using multiple netmodules built from Roslyn #1228 Netmodules should generate per-assembly unique names for PrivateImplementationDetails
• Anonymous types in VB do not include the module name, so cannot be combined into netmodules #1319 VB Anonymous types should generate per-assembly unique names for netmodules
• C# compiler encodes module name into anonymous type, but does not distinguish "." and "_" #1428 C# should mangle module name uniquely to produce anonymous types.
• C# PrivateImplementationDetails class can contain module name with '.', confusing tools which expect to parse into namespaces #1430 C# PrivateImplementationDetails should not have "." in its name
• Ordering of synthesized delegates in metadata should be deterministic #1440 Ordering of synthesized delegates in metadata should be deterministic
• Should the order in which the compiler reports diagnostics be deterministic? #1506 Diagnostics should be returned in a deterministic order
• Make the command-line compilers able to generate reference assemblies #2184 Make the command-line compiler capable of generating a reference assembly
  • [Work items] Refining ref assemblies #17612 Refine what is in reference assemblies and what diagnostics prevent generating one
• Add support for /features:deterministic for msbuild #2303 Add support for /features:deterministic for msbuild (fixes timestamp and MVID)
• Attributes emitted in nondeterministic order #4171 Attributes are sometimes emitted in nondeterministic order
• $STATIC$ fields emitted in nondeterministic order #4172 VB static locals emitted to fields in nondeterministic order
• Roslyn occasionally parses double values incorrectly #4221 Roslyn parses double values differently on x32 vs x64
• Compilation of source with auto-generated version is non-deterministic #4578 Auto-generated assembly version is nondeterministic
• List of compiler inputs that affect deterministic compiler output #5070 Document what compiler inputs affect its deterministic output
• VB interfaces not emitted it a deterministic order across partial types #7111 VB interfaces not emitted it a deterministic order across partial types
• VB shared field initializers should be emitted in source order across partial types #7112 VB shared field initializers should be emitted in source order across partial types
• Constant propagation uses wrong floating-point precision #7262 Constant folding uses wrong precision
• Small method body cache is nondeterministic  #7595 Small method body cache nondeterministic (see also Small method body cache disabled for deterministic builds #17052)
• Review usage of random guid in Emit/ErrorType.cs #8703 Review usage of random guid in Emit/ErrorType.cs
• Deterministic builds need a mechanism to control PDB path in PE #9813 Need a mechanism to cause file-relative PDB path to be included in the assembly.
• Document compiler /determinism effect on generated assembly #10321 Document compiler /determinism effect on generated assembly
• Enumerate files deterministically when compiling *.cs or *.vb #10858 Enumerate files deterministically when compiling *.cs or *.vb
• Symbol.Locations has indeterministic ordering of locations between compilations with same inputs #11015 Symbol.Locations has indeterministic ordering of locations between compilations with same inputs
• The order in which type forwarders are emitted to ExportedType table is non-deterministic #11990 The order in which type forwarders are emitted to ExportedType table is non-deterministic
• Compiler does not take all inputs into account for determinism #17121 Document compiler inputs for determinism (for cacheability)
• Generation of the GetHashCode method of anonymous types is not deterministic #23020 Generation of the GetHashCode method of anonymous types is not deterministic
• Compilers should normalize NaNs produced for 0.0/0.0 for determinism #30439 Produce the same NaN bits on different platforms
• Constant folding produces different IL depending on host architecture #37527 Constant folding produces different IL depending on host architecture
• Report error/warning if /pathmap doesn't cover all emitted source paths #23969 Diagnose incomplete pathmap

[ghost]

@jcouv, the still-unchecked items in the first post can be revisited. Some of them are closed issues (some are discussion-going-nowhere open issues).

[martinsuchan]

Wondering, are there any plans to make UWP app builds deterministic by default? Timestamps and wildcards in assembly versions are not really important when publishing apps to Store.
Last time I was testing deterministic builds for UWP apps it was not possible to produce identical packages because WinMDExp and MakePri.exe tools from the Windows SDK build pipeline don't support producing deterministic outputs yet.
Also since Microsoft Store uses differential updates for downloading new packages, having deterministic builds by default could save tons of Store traffic basically for free.

[jcouv]

Tagging @MichalStrehovsky @sergiy-k for UWP question.

[MichalStrehovsky]

This came up in the context of .NET Native in the past (#23456) but I couldn't find the owners of the tools in question. Maybe @tarekgh would know for MakePri?

[tarekgh]

@axelandrejs can help answering MakePri question.

[axelandrejs]

MakePri is expected to produce identical output for identical input. Now, it is possible that some of the inputs change unexpectedly, since MakePri parses the file system. If you have two files that were produced over the same set of inputs but are not identical, can you share the files?

Note that if you are implementing your own build pipeline, we now have a programmatic equivalent to MakePri. See https://msdn.microsoft.com/en-us/library/windows/desktop/mt845690(v=vs.85).aspx . It avoids dependencies on the file system and as such takes out some of the potential non-determinism. Also, it's faster. We are developing a tool that supports specifying all the data needed to build a UWP resources file via a relatively simple XML, in case that would help.

[MrSapps]

Does anyone know if resgen.exe is deterministic?

[martinsuchan]

@axelandrejs @tarekgh Reviving the discussion, MakePri does not produce deterministic outputs, nor WinMDExp, just checked again in VS 15.7.4. There is a simple repro solution available in #23456. Just build the solution twice in the same clean folder and extract all appxupload/appx files in it.
The expected result is identical content of both folders, the reality is differences in resources.pri and Winrt.Component.winmd.
Any chance to find the product owners of those tools and make them deterministic by default?

Files with different checksum:

\AppxBlockMap.xml
\AppxSignature.p7x
\ReproTest_1.0.0.0_x86.appx
\AppxMetadata\AppxBundleManifest.xml
\ReproTest_1.0.0.0_x86\AppxBlockMap.xml
\ReproTest_1.0.0.0_x86\AppxSignature.p7x
\ReproTest_1.0.0.0_x86\resources.pri
\ReproTest_1.0.0.0_x86\Winrt.Component.winmd
\ReproTest_1.0.0.0_x86\AppxMetadata\CodeIntegrity.cat

[tarekgh]

@axelandrejs may be able to help better here.

[axelandrejs]

The problem is not actually makepri. The problem are the XBF files, specifically App.xbf and MainPage.xbf. They are different between runs.

By default XAML files are embedded as BLOBs into the PRI files to improve performance. This means that if the XBF files change, you will see the difference in the resources.pri file. You can see for yourself. You can dump the contents of the resoures.pri file via the following command.

makepri dump /dt detailed /if [the resoures.pri file] /of [some .xml file name]. I did this for the two files you had in ReproTest.build1 and ReproTest.build2. You can see the the App.xbf contents below.

@LarryOsterman can speak to the .winmd differences.
@jevansaks can speak to the .xbf differences.

WEJGAJYBAABaAAAAAgAAAAEAAAB4AAAAAAAAAHYBAAAAAAAAegEAAAAAAAB+AQAAAAAAAIIBAAAAAAAAhgEAAAAAAAAzNjZCNDlDODVFRjdEOTIyQkRERTQ4RTA2MzY3MUJCMQAICAAzAGEAEiodUrgWAIzg/iADAQAAAAAAAAAAAAAAAwAAADkAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwALwBwAHIAZQBzAGUAbgB0AGEAdABpAG8AbgAAACwAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwAAAAPAAAAdQBzAGkAbgBnADoAUgBlAHAAcgBvAFQAZQBzAHQAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAABAAAAAgAAAAEAAAAAAAAATgAAABIAAAAAAAADAQABAAAAeAADAgAFAAAAbABvAGMAYQBsAAsNAAAAUgBlAHAAcgBvAFQAZQBzAHQALgBBAHAAcAAXH4AaSIALQgIAAAAAIQ==

VERSUS

WEJGAJYBAABaAAAAAgAAAAEAAAB4AAAAAAAAAHYBAAAAAAAAegEAAAAAAAB+AQAAAAAAAIIBAAAAAAAAhgEAAAAAAAAzNjZCNDlDODVFRjdEOTIyQkRERTQ4RTA2MzY3MUJCMQAICAAAAAAAHQT6JAASAIpTAHkAcwB0AGUAbQAuAE8AAwAAADkAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwALwBwAHIAZQBzAGUAbgB0AGEAdABpAG8AbgAAACwAAABoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZgB4AC8AMgAwADAANgAvAHgAYQBtAGwAAAAPAAAAdQBzAGkAbgBnADoAUgBlAHAAcgBvAFQAZQBzAHQAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAABAAAAAgAAAAEAAAAAAAAATgAAABIAAAAAAAADAQABAAAAeAADAgAFAAAAbABvAGMAYQBsAAsNAAAAUgBlAHAAcgBvAFQAZQBzAHQALgBBAHAAcAAXH4AaSIALQgIAAAAAIQ==

[gafter]

Added #375 to the list of issues.

[tannergooding]

@gafter, just as an FYI. decimal parsing should now respect all digits given in netcoreapp3.0 and forward.

[gafter]

@tannergooding Thank you. That just confirms that different runtimes do it differently from each other.

Is there decimal parsing code that we could copy or adapt into Roslyn?

[tannergooding]

@gafter, yes. The logic lives here: https://source.dot.net/#System.Private.CoreLib/shared/System/Number.Parsing.cs,f156a872d71c54fd,references

Most of this is similar to the floating-point parsing code Roslyn already has. That is TryStringToNumber converts the string into a digit buffer and scale (CoreFX also tracks the sign, but I believe Roslyn handles that separately).

It then converts that into the actual decimal metadata in TryNumberToDecimal, which is also where the rounding and additional digit considerations occur.

[jevansaks]

When i was first added to this issue I wasn't on GitHub. But I am now and GitHub started notifying me of new comments. :) For the XBF issue, are you still seeing it? I can file an issue for the XAML compiler team to take a look.

[martinsuchan]

@jevansaks I've added repro for deterministic UWP app building here #23456
I'm quite sure it wasn't fixed yet.

[CyrusNajmabadi]

@jaredpar do we need this issue anymore? Or is the work here basically considered 'done', with only bugs to mop up?

[jaredpar]

No, we can close this out.