Race condition writing Microsoft.Interop.DllImportGenerator.dll during library product build (Windows x64 Release)

[trylek]

Example run: https://dev.azure.com/dnceng/public/_build/results?buildId=1472395&view=logs&j=3ee9f22f-fdeb-5ba9-4d07-86b7b8a7437b&t=796c5bee-4d73-5a57-e6ab-2038dda597c4

Observed on platform: Windows x64 Release (the failure is apparently non-deterministic so it may also repro on other platforms)

Diagnostic output:

  System.Private.CoreLib.Generators -> D:\a\_work\1\s\artifacts\bin\System.Private.CoreLib.Generators\netstandard2.0-Release\System.Private.CoreLib.Generators.dll
  Microsoft.Interop.SourceGeneration -> D:\a\_work\1\s\artifacts\bin\Microsoft.Interop.SourceGeneration\netstandard2.0-Release\Microsoft.Interop.SourceGeneration.dll
CSC : error CS2012: Cannot open 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' for writing -- 'The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.' [D:\a\_work\1\s\src\libraries\System.Runtime.InteropServices\gen\DllImportGenerator\DllImportGenerator.csproj]
##[error]CSC(0,0): error CS2012: (NETCORE_ENGINEERING_TELEMETRY=Build) Cannot open 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' for writing -- 'The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.'
  DllImportGenerator -> D:\a\_work\1\s\artifacts\bin\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll

Based on my past experience I would expect this to be caused by interlocking holes that cause two projects to try building the DllImportGenerator project in parallel but I'm not a library build expert. @jkoritzinsky, based on your experience with Roslyn generators, could you please suggest how to continue the investigation?

Thanks

Tomas

[ghost]

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Example run: https://dev.azure.com/dnceng/public/_build/results?buildId=1472395&view=logs&j=3ee9f22f-fdeb-5ba9-4d07-86b7b8a7437b&t=796c5bee-4d73-5a57-e6ab-2038dda597c4

Observed on platform: Windows x64 Release (the failure is apparently non-deterministic so it may also repro on other platforms)

Diagnostic output:

  System.Private.CoreLib.Generators -> D:\a\_work\1\s\artifacts\bin\System.Private.CoreLib.Generators\netstandard2.0-Release\System.Private.CoreLib.Generators.dll
  Microsoft.Interop.SourceGeneration -> D:\a\_work\1\s\artifacts\bin\Microsoft.Interop.SourceGeneration\netstandard2.0-Release\Microsoft.Interop.SourceGeneration.dll
CSC : error CS2012: Cannot open 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' for writing -- 'The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.' [D:\a\_work\1\s\src\libraries\System.Runtime.InteropServices\gen\DllImportGenerator\DllImportGenerator.csproj]
##[error]CSC(0,0): error CS2012: (NETCORE_ENGINEERING_TELEMETRY=Build) Cannot open 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' for writing -- 'The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.'
  DllImportGenerator -> D:\a\_work\1\s\artifacts\bin\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll

Based on my past experience I would expect this to be caused by interlocking holes that cause two projects to try building the DllImportGenerator project in parallel but I'm not a library build expert. @jkoritzinsky, based on your experience with Roslyn generators, could you please suggest how to continue the investigation?

Thanks

Tomas

Author:	trylek
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[jkoritzinsky]

I'll take this one

[safern]

I just saw this on all configurations build as well: https://dev.azure.com/dnceng/public/_build/results?buildId=1472394&view=logs&j=81a0f8c2-d4db-561f-9b8e-3f045280d7fe&t=2a104984-c43b-5b93-1361-6bd8a1e96b04&l=1181

[am11]

Still failing: https://dev.azure.com/dnceng/public/_build/results?buildId=1481003&view=logs&jobId=79ebbb2e-07f2-5457-c5e5-132dd7ee4ba8

CSC(0,0): error CS2012: (NETCORE_ENGINEERING_TELEMETRY=Build) Cannot open 'D:\a_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' for writing -- 'The process cannot access the file 'D:\a_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.'

[jkoritzinsky]

😢 If my fix didn't work, then I'm out of ideas. We'll get a permanent fix once we update to the RTM sdk.

[trylek]

For the sake of completeness I have just hit a somewhat different bug and at the first glance it's not clear to me whether it's just a different manifestation of the same problem or something slightly different:

https://dev.azure.com/dnceng/public/_build/results?buildId=1486386&view=logs&j=a21a5f2c-b5cf-50e3-b899-59e91a62c520&t=9276a34f-ae0d-54e7-242f-96090118551b

  System.Private.CoreLib.Generators -> /__w/1/s/artifacts/bin/System.Private.CoreLib.Generators/netstandard2.0-Release/System.Private.CoreLib.Generators.dll
/__w/1/s/.dotnet/sdk/6.0.100-rc.2.21505.57/Microsoft.Common.CurrentVersion.targets(4635,5): error MSB3026: Could not copy "/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll" to "/__w/1/s/artifacts/bin/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll' because it is being used by another process.  [/__w/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj]
##[error].dotnet/sdk/6.0.100-rc.2.21505.57/Microsoft.Common.CurrentVersion.targets(4635,5): error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll" to "/__w/1/s/artifacts/bin/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll' because it is being used by another process. 
/__w/1/s/.dotnet/sdk/6.0.100-rc.2.21505.57/Microsoft.Common.CurrentVersion.targets(4635,5): error MSB3026: Could not copy "/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll" to "/__w/1/s/artifacts/bin/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll' because it is being used by another process.  [/__w/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj]
##[error].dotnet/sdk/6.0.100-rc.2.21505.57/Microsoft.Common.CurrentVersion.targets(4635,5): error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll" to "/__w/1/s/artifacts/bin/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/__w/1/s/artifacts/obj/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll' because it is being used by another process. 
  Microsoft.Interop.SourceGeneration -> /__w/1/s/artifacts/bin/Microsoft.Interop.SourceGeneration/netstandard2.0-Release/Microsoft.Interop.SourceGeneration.dll

Reopening the bug for tracking purposes, my current understanding is that we'll close it once we update the SDK to RTM.

Thanks

Tomas

[danmoseley]

Hit this myself

runtime 20211126.41 / Libraries Build Linux_musl arm Release / Restore and Build Product
❌[Log] .dotnet/sdk/6.0.100/Microsoft.Common.CurrentVersion.targets(4635,5): error MSB3026: Could not copy "/__w/1/s/artifacts/obj/DllImportGenerator/netstandard2.0-Release/Microsoft.Interop.DllImportGenerator.dll" to "/__w/1/s/artifacts/bin/DllImportGenerator/netstandard2.0-Release/Microsoft.Interop.DllImportGenerator.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/__w/1/s/artifacts/bin/DllImportGenerator/netstandard2.0-Release/Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.
❌[Log] .dotnet/sdk/6.0.100/Microsoft.Common.CurrentVersion.targets(4635,5): error MSB3026: Could not copy "/__w/1/s/artifacts/obj/DllImportGenerator/netstandard2.0-Release/Microsoft.Interop.DllImportGenerator.dll" to "/__w/1/s/artifacts/bin/DllImportGenerator/netstandard2.0-Release/Microsoft.Interop.DllImportGenerator.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/__w/1/s/artifacts/bin/DllImportGenerator/netstandard2.0-Release/Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.**

https://dev.azure.com/dnceng/public/_build/results?buildId=1487466&view=results

[eerhardt]

I hit this as well:

CSC(0,0): error CS2012: (NETCORE_ENGINEERING_TELEMETRY=Build) Cannot open 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' for writing -- 'The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\DllImportGenerator\netstandard2.0-Release\Microsoft.Interop.DllImportGenerator.dll' because it is being used by another process.'

https://github.com/dotnet/runtime/pull/61928/checks?check_run_id=4372017369
https://dev.azure.com/dnceng/public/_build/results?buildId=1487100&view=logs&j=bcd0b573-6223-5b97-9574-b5bb8414cd58&t=a7a8c57b-a5a4-5fc1-9ee7-d8ad6c4ef580&l=1000