Enumerate files deterministically when compiling *.cs or *.vb

[gafter]

When you compile *.cs or *.vb, the compiler enumerates files to determine what to compile. It should sort the result so that the compilation is more deterministic.

The relevant code has a bottleneck in Roslyn.Utilities.PortableShim.Directory.EnumerateFiles where the sorting could take place.

[jaredpar]

I'm currently leaning towards won't fix on this. My logic is that, within reason, adding the /deterministic argument to the compiler should only observably change the inherently non-deterministic parts of the produced PE: MVID, PDB ID and time stamp.

Fixing this issue would introduce a new observable behavior: partial class ordering. That's not a big difference but is observable.

Instead of changing this behavior, I'd instead push on MSBuild / SDK to ensure they give us files in deterministic order as they normally drive our build.

Thoughts @gafter?

[tannergooding]

I think we should preserve the inputs as given to us and push on other tools to give them to us deterministically.

[gafter]

I believe the compiler may expand wildcards on input file specifications. If that is so, then the compiler is the only one that has control of the order in which they are enumerated.

[jaredpar]

I believe the compiler may expand wildcards on input file specifications. If that is so, then the compiler is the only one that has control of the order in which they are enumerated.

Hmm, I'm not sure if we do or don't do this. Will investigate. If we do manual expansion of *.cs though ourselves then yeah we should probably do it deterministically.

[CyrusNajmabadi]

I'm nearly certain we do wildcard expansion as that's not a property of the core cmd shell (like it would be on unix).

[jaredpar]

I'm nearly certain we do wildcard expansion as that's not a property of the core cmd shell (like it would be on unix).

Given that it's the option that would cause more work for me, I'm certain we do it too 😄

[CyrusNajmabadi]

Expansions are done in ExpandFileNamePattern

[tannergooding]

@jaredpar, would this track only expanding wildcards deterministically or sorting all code file inputs deterministically?

I would think we want to expand wildcards deterministically (since that is done by us) but would preserve any non-expanding inputs the user specified in the order they specified them.

I would also think that we would preserve ordering of our expansions and ordering of user inputs relative to each other. That is, if the user passed in SomeFile.cs *.cs SomeOtherFile.cs, where *.cs expands to A.cs and B.cs, then we would have a set of inputs of: SomeFile.cs A.cs B.cs SomeOtherFile.cs. Likewise, if the user reordered the way things were specified to be *.cs SomeOtherFile.cs SomeFile.cs, we would expand it to be exactly A.cs B.cs SomeOtherFile.cs SomeFile.cs.

I think doing this is important since, as you mentioned, it does make some difference in some cases (such as partial class ordering).

[jaredpar]

I would think we want to expand wildcards deterministically (since that is done by us) but would preserve any non-expanding inputs the user specified in the order they specified them

Yep.