Support publishing for Multiple RIDs in parallel

[JustArchi]

dotnet publish command takes a while, especially with self-contained applications and several different platforms to build for. Until now I accelerated that process by executing dotnet restore manually, then calling dotnet publish --no-restore with different runtimes in parallel, all of them with the output to a different folder. This continues to work perfectly, I've never encountered any issues related to that.

My question is, can it be used this way? If yes, how about dotnet publish without --no-restore, can it guarantee proper restore for independent builds in the process?

Another question is if perhaps dotnet publish could natively support this setup by something like -rl - --runtimes-list where one could specify win-x64,linux-x64,osx-x64 and dotnet would run all builds possible to run in parallel automatically. The lack of such option is the reason why I execute dotnet publish command for all my runtimes in the first place - I'm trying to do something that should be happening already.

I know that roslyn is already compiling things in parallel, but sadly many compilation steps (such as generating resources) are still run synchronously. Even if everything would be accelerated, it'd still make sense to run things in parallel, since those build steps are independent and there is no reason why one would need to execute 5 different dotnet publish commands one after another. I did some testing and the entire build was indeed much faster when I compiled with all runtimes in parallel, this is especially important in CI environments that can benefit greatly from reduced compile time, example.

Thank you for answering.

[livarcocc]

@nguerrera or @dsplaisted can you answer this question?

[dsplaisted]

I've changed the title of this issue to track adding built-in support for this. I imagine we could do it by supporting multiple -r arguments to dotnet publish, ie dotnet publish -r win7-x64 -r linux-x64, and then running those publishes in parallel as possible.

As far as whether you can do this yourself today, it is probably a better idea to let MSBuild handle the parallelism, which will let it take advantage of its knowledge about what can be parallelized and what doesn't need to be re-run (for example referenced projects shouldn't need to be rebuilt for each RuntimeIdentifier).

I've written an MSBuild target that allows you to run multiple publishes in parallel. In some quick testing, it worked for me.

I put most the logic in a separate .targets file, so you can apply it to multiple projects. I put in in a file called PublishAllRids.targets, with the following contents:

<Project DefaultTargets="Build">

  <PropertyGroup>
    <MSBuildAllProjects>$(MSBuildAllProjects);$(MSBuildThisFileFullPath)</MSBuildAllProjects>

    <!-- Enable roll-forward to latest patch.  This allows one restore operation
         to apply to all of the self-contained publish operations. -->
    <TargetLatestRuntimePatch>true</TargetLatestRuntimePatch>
  </PropertyGroup>

  <Target Name="PublishAllRids">
    <ItemGroup>
      <!-- Transform RuntimeIdentifiers property to item -->
      <RuntimeIdentifierForPublish Include="$(RuntimeIdentifiers)" />

      <!-- Transform RuntimeIdentifierForPublish items to project items to pass to MSBuild task -->
      <ProjectToPublish Include="@(RuntimeIdentifierForPublish->'$(MSBuildProjectFullPath)')">
        <AdditionalProperties>RuntimeIdentifier=%(RuntimeIdentifierForPublish.Identity)</AdditionalProperties>
      </ProjectToPublish>
    </ItemGroup>

    <MSBuild Projects="@(ProjectToPublish)"
             Targets="Publish"
             BuildInParallel="true"
             />
  </Target>

</Project>

Then in the project file itself, you just need to set the RuntimeIdentifiers property to a semicolon-separated list of RIDs to publish, and import the .targets file. Here's what my test .csproj file looks like with these changes:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <!-- The RuntimeIdentifiers to restore and to publish for -->
    <RuntimeIdentifiers>win-x64;win-x86;linux-x64</RuntimeIdentifiers>
  </PropertyGroup>

  <Import Project="PublishAllRids.targets" />

</Project>

One thing to note is that in order for a single restore to work for multiple publishes, I've set the TargetLatestRuntimePatch to true. This means that the app will roll forward to the latest patch of .NET Core. If you also want to publish the app as framework-dependent, you might want to turn this property off in that case.

With a project set up like this, you can publish for multiple RIDs with the following command:

dotnet msbuild -restore -t:PublishAllRids

You could also run the restore as a separate command:

dotnet restore
dotnet msbuild -t:PublishAllRids

[simonachmueller]

Can I publish different RuntimeIdentifiers in different output folder using the method mentioned above? I want to run the dotnet publish with different -o values.

Then in the project file itself, you just need to set the RuntimeIdentifiers property to a semicolon-separated list of RIDs to publish, and import the .targets file. Here's what my test .csproj file looks like with these changes:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <!-- The RuntimeIdentifiers to restore and to publish for -->
    <RuntimeIdentifiers>win-x64;win-x86;linux-x64</RuntimeIdentifiers>
  </PropertyGroup>

  <Import Project="PublishAllRids.targets" />

</Project>

[dsplaisted]

@semenmiroshnichenko You wouldn't be able to specify separate output folders via the command line in this case. By default, each RuntimeIdentifier will be published to a separate folder anyway. If you want to customize that, you would need to do so via logic in the project file (or imports), rather than passing in on the command line.

[hrumhurum]

@dsplaisted, thank you for sharing that impressive snippet.

Any plans to include cross-RID publishing into official release? If we already have TargetFrameworks then why not to have the official support for RuntimeIdentifiers out of the box?

[dsplaisted]

Any plans to include cross-RID publishing into official release? If we already have TargetFrameworks then why not to have the official support for RuntimeIdentifiers out of the box?

We probably do want to support this better, but it's quite complicated so it will probably take a long time.

I don't think we want to by default just build for multiple RuntimeIdentifiers the way we do for TargetFrameworks, because if you were to combine them it would get messy. Rather, what we are thinking is that TargetFrameworks will be the way to build the same project multiple times, but allowing you to define your own TargetFramework values that may include a RuntimeIdentifier. So you could set the TargetFrameworks to something like net7.0-win-x64;net7.0-linux-x64.

The next step to support this better is probably NuGet/Home#5154.