Solution Rebuild might fail when a shared output directory is used

[MichaeIDietrich]

Steps to Reproduce

1. Clone https://github.com/MichaeIDietrich/TestRebuildError
2. dotnet build
3. dotnet msbuild -t:rebuild -m:1

msbuild command is used to disable parallel compilation, so the projects are built one after the other to easily reproduce this issue. On large solutions there is no need to limit parallel compilation.

Actual Behavior

Compilation fails:

TestRebuildError -> \TestRebuildError\bin\Debug\net5.0\TestRebuildError.dll
C:\Program Files\dotnet\sdk\5.0.301\Microsoft.Common.CurrentVersion.targets(4964,5): error MSB3030: Could not copy the file "\TestRebuildError\bin\Debug\net5.0\TestRebuildError.runtimeconfig.dev.json" because it was not found. [\TestRebuildError\TestRebuildError.Tests\TestRebuildError.Tests.csproj]
C:\Program Files\dotnet\sdk\5.0.301\Microsoft.Common.CurrentVersion.targets(4964,5): error MSB3030: Could not copy the file "\TestRebuildError\bin\Debug\net5.0\TestRebuildError.deps.json" because it was not found. [\TestRebuildError\TestRebuildError.Tests\TestRebuildError.Tests.csproj]
C:\Program Files\dotnet\sdk\5.0.301\Microsoft.Common.CurrentVersion.targets(4964,5): error MSB3030: Could not copy the file "\TestRebuildError\bin\Debug\net5.0\TestRebuildError.runtimeconfig.json" because it was not found. [\TestRebuildError\TestRebuildError.Tests\TestRebuildError.Tests.csproj]

Expected Behavior

Both projects compile successfully.

Analyzation

The Clean target of the test project will also clear the *.deps.json of the referenced project,
which then leads to the issue that the test project can no longer copy the *.deps.json file when the CopyFilesToOutputDirectory target is invoked to copy transtively referenced items.

There is a FileWritesShareable item group that is used to filter out shared items so the Clean target does not delete shared files. But for some reason it does not work for *.runtimeconfig.dev.json, *.runtimeconfig.json and *.deps.json.

Workaround

Setting UseCommonOutputDirectory to true will prevent any copying and cleaning of transitively referenced items and might be the recommended solution here.

Edit:
As it turns out the workaround does not work as expected, since it also prevents any referenced item to be copied.
Unfortunately there is a condition on the Copy task within the _CopyFilesMarkedCopyLocal target.

Question

While there is a workaround, I still think that not using UseCommonOutputDirectory should compile nevertheless.
There is no information UseCommonOutputDirectory. Maybe this property should be officially documented somewhere?

[MichaeIDietrich]

Related issue: #17645

[marcpopMSFT]

#17645 tracks the issues related to building to a common output folder. This is not recommended though as it's safer to compile and then copy after. We do not have plans at the moment to improve this experience beyond the UseCommonOutputDirectory flag already mentioned which has its drawbacks.

@tdykstra to investigate whether we should have a document covering UseCommonOutputDirectory and common output directory issues as the comment here covers the details: #17645 (comment)

[tdykstra]

@ghogen should we add UseCommonOutputDirectory to https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-reserved-and-well-known-properties

[ghogen]

@tdykstra Yes, thanks. Created issue as linked above.

[MichaeIDietrich]

I just would like to add some info why a shared output directory is useful and might be kind of needed for some solutions.

On the one hand project-scoped output directories do not really scale well with large solutions because each executable project will copy all of its (transitive) NuGet DLLs into its output directory, same goes for test projects. (For our solution this ended up in several GB of duplicated DLLs in those directories which is especially problematic for the potentially limited space on build agents.)

And on the other hand it is far more closer to how it usually would end up on the customer's PCs where all binaries will be installed into one directory, e.g. like MS Office. (At least I would have no good idea how to finally copy all resulting files of all top level, executable projects into a final common directory while maintaining the default output locations for each project.)

Any recommendations would be pretty welcome, if there are any.
And thank you for taking the time to look at this.

[KalleOlaviNiemitalo]

@MichaeIDietrich, Microsoft.Build.Artifacts looks like it could be used to "finally copy all resulting files". That would consume even more disk space, though.

[SebFerraro]

Regardless of whether UseCommonOutputDirectory is documented or not, I still believe it should be possible to easily build several projects to one output directory. This will most likely cause issues if people (like ourselves) are starting to migrate some older projects to SDK format - that is what seemingly broke our builds

[Gav-Brown]

If it helps, I posted a possible solution here which allows you to not set UseCommonOutputDirectory but just the OutputDir and then projects, dlls, and NuGet dlls all get copied to a single "bin" folder without missing deps.json files etc.