[Bug]: "An item with the same key has already been added" when migrating to CPM with ProjectDependencies in solution file

[YoshiRulz]

NuGet Product Used

dotnet.exe

Product Version

.NET SDK 6.0.400 from AUR

Worked before?

unknown

Impact

It bothers me. A fix would be nice

Repro Steps & Context

1. take sample project with YoshiRulz/Samples@9a0e83a, observe build success
2. take additional commit YoshiRulz/Samples@f66b488, observe build failure

In my real codebase, I use PrivateAssets="all" to give every project w/ the .props import certain packages without those packages leaking to dependent projects. Specifically, I add Microsoft.NETFramework.ReferenceAssemblies, Nullable, System.Resources.Extensions, and various Analyzers.

Verbose Logs

output of dotnet build --verbosity diag at second, broken commit edit: I've since updated the .NET SDK but still can't build my main project w/ CPM

[jeffkl]

@YoshiRulz I'm unable to reproduce the issue.

D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample>git checkout f66b488896c237db80c36a8960ef95ef8329e133
HEAD is now at f66b488 broken
M       CentralPackageManagementExample/Directory.Build.props

D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample>git clean -xfd && dotnet restore
Removing src/ClassLibrary1/obj/
Removing src/ClassLibraryWithTransitivePinning/obj/
Removing src/ClassLibraryWithVersionOverride/obj/
  Determining projects to restore...
  The project D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibrary1\ClassLibrary1.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
  The project D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithTransitivePinning\ClassLibraryWithTransitivePinning.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
  The project D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithVersionOverride\ClassLibraryWithVersionOverride.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
  Restored D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithTransitivePinning\ClassLibraryWithTransitivePinning.csproj (in 179 ms).
  Restored D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibrary1\ClassLibrary1.csproj (in 179 ms).
  Restored D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithVersionOverride\ClassLibraryWithVersionOverride.csproj (in 257 ms).

D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample>git checkout 9a0e83a418d1e18d7cf754e2b8df3e612b51dc09
Previous HEAD position was f66b488 broken
HEAD is now at 9a0e83a works
M       CentralPackageManagementExample/Directory.Build.props

D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample>git clean -xfd && dotnet restore
Removing src/ClassLibrary1/obj/
Removing src/ClassLibraryWithTransitivePinning/obj/
Removing src/ClassLibraryWithVersionOverride/obj/
  Determining projects to restore...
  The project D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibrary1\ClassLibrary1.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
  The project D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithTransitivePinning\ClassLibraryWithTransitivePinning.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
  The project D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithVersionOverride\ClassLibraryWithVersionOverride.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
  Restored D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithTransitivePinning\ClassLibraryWithTransitivePinning.csproj (in 196 ms).
  Restored D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibrary1\ClassLibrary1.csproj (in 197 ms).
  Restored D:\Repros\YoshiRulz-Samples\CentralPackageManagementExample\src\ClassLibraryWithVersionOverride\ClassLibraryWithVersionOverride.csproj (in 275 ms).

Can you supply a binlog by specifying the /bl command-line argument and attaching msbuild.binlog for the broken build case please?

[YoshiRulz]

No because it's building correctly after a .NET SDK update. Hopefully msbuild.binlog from my real project is insightful, but I'll try to make a new repro project too.

[YoshiRulz]

repro project
I wasn't able to adapt our Nix-based CI to this new project, as it needs a special dotnet restore wrapper to work with Nix' network sandboxing. This msbuild.binlog is from my own machine.

[jeffkl]

@YoshiRulz Thanks, unfortunately I'm not able to open the binlog for some reason? Does it open in the Structured Log Viewer for you?

[YoshiRulz]

Webapp can load it

[jeffkl]

Ah sorry I figured out that the attachment is msbuild.binlog.gz so I was unzipping it. I renamed the download file and now I can open it.

I believe the problem is that you're using the Update guesture on the <PackageVersion /> items but you need to use Include:

- <PackageVersion Update="DotNetAnalyzers.DocumentationAnalyzers" Version="1.0.0-beta.59" />
+ <PackageVersion Include="DotNetAnalyzers.DocumentationAnalyzers" Version="1.0.0-beta.59" />

Other implementations of central package management use an Update to specify versions in a common import. But in NuGet's central package management, you declare both <PackageReference /> and <PackageVersion /> items with Include and NuGet stitches them together.

[YoshiRulz]

That gives /usr/share/dotnet/sdk/6.0.400/NuGet.targets(132,5): error : An item with the same key has already been added. Key: DotNetAnalyzers.DocumentationAnalyzers [/BizHawk.sln], even if I also remove PrivateAssets="all" from the <PackageReference/>s. msbuild.binlog

(I was confused by the one occurrence of Update in the docs here, is that only for overriding a higher file, or simply a typo?)

[jeffkl]

@YoshiRulz you seem to have uncovered a corner case with the way NuGet gathers package references. Your solution file is declare dependencies:

Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "BizHawk.Version", "src\BizHawk.Version\BizHawk.Version.csproj", "{0CE8B337-08E3-4602-BF10-C4D4C75D2F13}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "BizHawk.Common", "src\BizHawk.Common\BizHawk.Common.csproj", "{866F8D13-0678-4FF9-80A4-A3993FD4D8A3}"
+	ProjectSection(ProjectDependencies) = postProject
+		{0CE8B337-08E3-4602-BF10-C4D4C75D2F13} = {0CE8B337-08E3-4602-BF10-C4D4C75D2F13}
+	EndProjectSection
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "BizHawk.Client.EmuHawk", "src\BizHawk.Client.EmuHawk\BizHawk.Client.EmuHawk.csproj", "{DD448B37-BA3F-4544-9754-5406E8094723}"
+	ProjectSection(ProjectDependencies) = postProject
+		{0CE8B337-08E3-4602-BF10-C4D4C75D2F13} = {0CE8B337-08E3-4602-BF10-C4D4C75D2F13}
+	EndProjectSection
EndProject

Under normal conditions, MSBuild generates a "metaproj" project in memory from the contents of a solution file and builds that. But when your solution file declares project build ordering, it creates the metaproj in a very unique way.

This is the section of the main metaproj that tells MSBuild what projects to build:

  <ItemGroup>
    <ProjectReference Include="/home/yoshi/Downloads/hawk-cpm-repro/src/BizHawk.Version/BizHawk.Version.csproj">
      <AdditionalProperties>Configuration=Debug; Platform=AnyCPU</AdditionalProperties>
      <Platform>AnyCPU</Platform>
      <Configuration>Debug</Configuration>
      <ToolsVersion>
      </ToolsVersion>
      <SkipNonexistentProjects>False</SkipNonexistentProjects>
    </ProjectReference>
    <ProjectReference Include="/home/yoshi/Downloads/hawk-cpm-repro/src/BizHawk.Common/BizHawk.Common.csproj.metaproj">
      <AdditionalProperties>Configuration=Debug; Platform=Any CPU</AdditionalProperties>
      <Platform>AnyCPU</Platform>
      <Configuration>Debug</Configuration>
      <ToolsVersion>Current</ToolsVersion>
      <SkipNonexistentProjects>Build</SkipNonexistentProjects>
    </ProjectReference>
    <ProjectReference Include="/home/yoshi/Downloads/hawk-cpm-repro/src/BizHawk.Client.EmuHawk/BizHawk.Client.EmuHawk.csproj.metaproj">
      <AdditionalProperties>Configuration=Debug; Platform=Any CPU</AdditionalProperties>
      <Platform>AnyCPU</Platform>
      <Configuration>Debug</Configuration>
      <ToolsVersion>Current</ToolsVersion>
      <SkipNonexistentProjects>Build</SkipNonexistentProjects>
    </ProjectReference>
  </ItemGroup>

For BizHawk.Version.csproj, MSBuild builds it directly. But for BizHawk.Common.csproj and BizHank.Client.EmuHawk, MSBuild generates another metaproj for each of those which has this:

BizHawk.Client.EmuHawk.csproj.metaproj

  <ItemGroup>
    <ProjectReference Include="/home/yoshi/Downloads/hawk-cpm-repro/src/BizHawk.Version/BizHawk.Version.csproj">
      <AdditionalProperties>Configuration=Debug; Platform=AnyCPU</AdditionalProperties>
      <Platform>AnyCPU</Platform>
      <Configuration>Debug</Configuration>
      <ToolsVersion>
      </ToolsVersion>
      <SkipNonexistentProjects>False</SkipNonexistentProjects>
    </ProjectReference>
  </ItemGroup>

BizHawk.Common.csproj.metaproj

  <ItemGroup>
    <ProjectReference Include="/home/yoshi/Downloads/hawk-cpm-repro/src/BizHawk.Version/BizHawk.Version.csproj">
      <AdditionalProperties>Configuration=Debug; Platform=AnyCPU</AdditionalProperties>
      <Platform>AnyCPU</Platform>
      <Configuration>Debug</Configuration>
      <ToolsVersion>
      </ToolsVersion>
      <SkipNonexistentProjects>False</SkipNonexistentProjects>
    </ProjectReference>
  </ItemGroup>

However, BizHawk.Common already has a ProjectReference to BizHawk.Version but is setting ReferenceAssemblyOutput="false". This causes NuGet to ask BizHawk.Common for its references and BizHawk.Version gets called a second time to add its packages to the graph, resulting in duplicates. This manifests as an error since this line of code is not checking if a package version has already been added:

https://github.com/NuGet/NuGet.Client/blob/cf6d15e81f43e96f56b314a3706d8ed9195352f6/src/NuGet.Core/NuGet.Commands/RestoreCommand/Utility/MSBuildRestoreUtility.cs#L1005

I believe if you remove the project ordering/dependency information from the solution file, things will work. You can also technically remove the project references in the projects themselves since MSBuild is injecting the references on-the-fly when building the solution. But this will break all builds that aren't for the solution file.

Thanks for reporting this issue!