Switch repo to use native AzDO container support (#14541)

* Switch repo to use native AzDO container support

This is a major refactor of the YAML used by the installer repo. The goal is to utilize the native container support that AzDO provides, rather than building containers on the fly and issuing commands using custom infrastructure. To do this, the YAML requires a bit of a refactor. The matrix strategy approach used by the repo to build a ton of different OS's does not work with containers, because a matrix strategy only changes the variables available to each build command. It cannot change the AzDO host environment. In order to resolve this, I refactored build.yml to take and use optional container names. In the process of doing this, I discovered a number of other things about the old YAML that just happened to 'accidentally' work and fixed them or did general cleanup. Including:

- This construct, used in build.yml, does not work as you might expect. If the parameter is not declared as a boolean (or not declared at all), this evaluates to "does this parameter exist", not "is it true":

    ```
    ${{ if parameters.pgoInstrument }}:
    ```
- I fully specified all the parameters and their types to avoid issues in the future.
- Build pool selection was moved to build.yml
- Removed some unused parameters.
- There was a **very** subtle indentation change here: https://github.com/dotnet/installer/blob/762d2966ee649e9f8174747d528aa067eaa7c4f0/.vsts-ci.yml#L275-L281. This meant that this leg ran in both PR and official builds. I have no idea whether this was the intention or not, but I kept it this way and reorganized the file.

**One change of note:** One of the upsides of the original matrix based approach is that job dependencies are simple. The jobs generated by the matrix are referred to in `dependsOn` lists only by the original job that contains the matrix. That keeps the dependsOn list small even if the number of jobs generated is large. Installer has a large set of independently addressable jobs now. Normally, we would solve this by using the arcade jobs template. The jobs template takes a set of jobs and automatically adds dependsOn for the publishing jobs. BUT, AzDO does not allow templates to be passed as parameters to other templates. This would mean that you couldn't use the build.yml template in conjunction with the jobs template and would have to list all the installer jobs as dependencies. This list would be hard to keep up to date. To avoid this, I used a new feature of publishing (`publishAssetsImmediately`) which uses the Publish To Build Asset Registry job to do the actual publishing call, and put it in a separate phase, then eliminated the post-build.yml call. This means that the publishing stage depends on all jobs in the build stage, and does not need to address them individually. Eliminating the post-build.yml stage may seem odd, but this is what actually happens when `publishAssetsImmediately` is set to true anyway.

build.sh

@@ -29,11 +29,6 @@ args=
 while [[ $# > 0 ]]; do
     lowerI="$(echo $1 | awk '{print tolower($0)}')"
     case $lowerI in
-        --docker)
-            export BUILD_IN_DOCKER=1
-            export DOCKER_IMAGENAME=$2
-            shift
-            ;;
         --noprettyprint)
             export DOTNET_CORESDK_NOPRETTYPRINT=1
             ;;
@@ -44,14 +39,4 @@ while [[ $# > 0 ]]; do
     shift
 done
 
-dockerbuild()
-{
-    BUILD_COMMAND=$DIR/run-build.sh $DIR/eng/dockerrun.sh --non-interactive "$@"
-}
-
-# Check if we need to build in docker
-if [ ! -z "$BUILD_IN_DOCKER" ]; then
-    dockerbuild $args
-else
-    $DIR/run-build.sh $args
-fi
+$DIR/run-build.sh $args

eng/build.yml

@@ -1,72 +1,153 @@
 parameters:
   # Agent OS identifier and used as job name
-  agentOs: ''
+- name: agentOs
+  type: string
 
-  # Agent pool
-  pool: {}
+  # Job name
+- name: jobName
+  type: string
 
-  # Additional variables
-  variables: {}
-
-  # Build strategy - matrix
-  strategy: {}
+# Container to run the build in, if any
+- name: container
+  type: string
+  default: ''
 
   # Job timeout
-  timeoutInMinutes: 180
-
-  # Publish using pipelines
-  enablePublishUsingPipelines: true
-
-phases:
+- name: timeoutInMinutes
+  type: number
+  default: 180
+
+# Build configuration (Debug, Release)
+- name: buildConfiguration
+  type: string
+  values:
+  - Debug
+  - Release
+
+# Build architecture
+- name: buildArchitecture
+  type: string
+  values:
+  - arm
+  - arm64
+  - x64
+  - x86
+
+# Linux portable. If true, passes portable switch to build
+- name: linuxPortable
+  type: boolean
+  default: false
+
+# Runtime Identifier
+- name: runtimeIdentifier
+  type: string
+  default: ''
+
+# UI lang
+- name: dotnetCLIUILanguage
+  type: string
+  default: ''
+
+# Additional parameters
+- name: additionalBuildParameters
+  type: string
+  default: ''
+
+# Run tests
+- name: runTests
+  type: boolean
+  default: true
+
+# PGO instrumentation jobs
+- name: pgoInstrument
+  type: boolean
+  default: false
+
+jobs:
 - template: /eng/common/templates/job/job.yml
   parameters:
+    # Set up the name of the job.
     ${{ if parameters.pgoInstrument }}:
-      name: PGO_${{ parameters.agentOs }}
+      name: PGO_${{ parameters.agentOs }}_${{ parameters.jobName }}
     ${{ if not(parameters.pgoInstrument) }}:
-      name: ${{ parameters.agentOs }}
-    timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
+      name: ${{ parameters.agentOs }}_${{ parameters.jobName }}
+
+    ## Set up the pool/machine info to be used based on the Agent OS
     ${{ if eq(parameters.agentOs, 'Windows_NT') }}:
       enableMicrobuild: true
+      pool:
+        ${{ if eq(variables['System.TeamProject'], 'public') }}:
+          name: NetCore-Public
+          demands: ImageOverride -equals build.windows.10.amd64.vs2019.open
+        ${{ if eq(variables['System.TeamProject'], 'internal') }}:
+          name: NetCore1ESPool-Internal
+          demands: ImageOverride -equals build.windows.10.amd64.vs2019
+    ${{ if eq(parameters.agentOs, 'Linux') }}:
+      pool:
+        ${{ if eq(variables['System.TeamProject'], 'public') }}:
+          name: NetCore-Public
+          demands: ImageOverride -equals Build.Ubuntu.1804.Amd64.Open
+        ${{ if eq(variables['System.TeamProject'], 'internal') }}:
+          name: NetCore1ESPool-Internal
+          demands: ImageOverride -equals Build.Ubuntu.1804.Amd64
+      container: ${{ parameters.container }}
+    ${{ if eq(parameters.agentOs, 'Darwin') }}:
+      pool:
+        vmImage: 'macOS-latest'
+
+    timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
     enablePublishBuildAssets: true
-    # Don't use built-in arcade logic for publishing artifacts and test results, as we manually do this and give better names
-    #enablePublishBuildArtifacts: true
-    #enablePublishTestResults: true
-    enablePublishUsingPipelines: ${{parameters.enablePublishUsingPipelines}}
+    enablePublishUsingPipelines: true
     enableTelemetry: true
     helixRepo: dotnet/installer
-    pool: ${{ parameters.pool }}
-    ${{ if ne(parameters.strategy, '') }}:
-      strategy: ${{ parameters.strategy }}
     workspace:
       clean: all
 
+# Test parameters
     variables:
+    - ${{ if eq(parameters.agentOs, 'Windows_NT') }}:
+      - _PackArg: '-pack'
+      - ${{ if parameters.runTests }}:
+        - _TestArg: '-test'
+      - ${{ else }}:
+        - _TestArg: ''
+    - ${{ if ne(parameters.agentOs, 'Windows_NT') }}:
+      - _PackArg: '--pack'
+      - ${{ if parameters.runTests }}:
+        - _TestArg: '--test'
+      - ${{ else }}:
+        - _TestArg: ''
+
+    - ${{ if parameters.pgoInstrument }}:
+      - _PgoInstrument: '/p:PgoInstrument=true'
+      - _PackArg: ''
+    - ${{ else }}:
       - _PgoInstrument: ''
-      - ${{ if eq(parameters.agentOs, 'Windows_NT') }}:
-        - _PackArg: '-pack'
-      - ${{ if ne(parameters.agentOs, 'Windows_NT') }}:
-        - _PackArg: '--pack'
-      - ${{ if parameters.pgoInstrument }}:
-        - _PgoInstrument: '/p:PgoInstrument=true'
-        - _PackArg: ''
-
-      - _AgentOSName: ${{ parameters.agentOs }}
-      - _TeamName: Roslyn-Project-System
-      - _SignType: test
-      - _BuildArgs: '/p:DotNetSignType=$(_SignType) $(_PgoInstrument)'
-
-      - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
-        - group: DotNet-Symbol-Server-PATs
-        - group: DotNet-HelixApi-Access
-        - group: DotNet-Blob-Feed
-        - _DotNetPublishToBlobFeed: true
-        - _PushToVSFeed: true
-        - _SignType: real
-        - _BuildArgs: /p:OfficialBuildId=$(BUILD.BUILDNUMBER)
-                  /p:DotNetSignType=$(_SignType)
-                  /p:TeamName=$(_TeamName)
-                  /p:DotNetPublishUsingPipelines=$(_PublishUsingPipelines)
-                  $(_PgoInstrument)
+
+    - ${{ if parameters.linuxPortable }}:
+      - _LinuxPortable: '--linux-portable'
+    - ${{ else }}:
+      - _LinuxPortable: ''
+
+    - ${{ if ne(parameters.runtimeIdentifier, '') }}:
+      - _RuntimeIdentifier: '--runtime-id ${{ parameters.runtimeIdentifier }}'
+    - ${{ else }}:
+      - _RuntimeIdentifier: ''
+
+    - _AgentOSName: ${{ parameters.agentOs }}
+    - _TeamName: Roslyn-Project-System
+    - _SignType: test
+    - _BuildArgs: '/p:DotNetSignType=$(_SignType) $(_PgoInstrument)'
+
+    - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
+      - group: DotNet-HelixApi-Access
+      - _PushToVSFeed: true
+      - _SignType: real
+      - _BuildArgs: /p:OfficialBuildId=$(BUILD.BUILDNUMBER)
+                /p:DotNetSignType=$(_SignType)
+                /p:TeamName=$(_TeamName)
+                /p:DotNetPublishUsingPipelines=$(_PublishUsingPipelines)
+                $(_PgoInstrument)
 
     steps:
     - checkout: self
@@ -83,14 +164,14 @@ phases:
       - script: build.cmd
                   $(_TestArg) $(_PackArg)
                   -publish -ci -sign
-                  -Configuration $(_BuildConfig)
-                  -Architecture $(_BuildArchitecture)
+                  -Configuration ${{ parameters.buildConfiguration }}
+                  -Architecture ${{ parameters.buildArchitecture }}
                   $(_BuildArgs)
-                  $(_AdditionalBuildParameters)
+                  ${{ parameters.additionalBuildParameters }}
                   $(_InternalRuntimeDownloadArgs)
         displayName: Build
         env:
-          DOTNET_CLI_UI_LANGUAGE: $(_DOTNET_CLI_UI_LANGUAGE)
+          DOTNET_CLI_UI_LANGUAGE: ${{ parameters.dotnetCLIUILanguage }}
 
     - ${{ if ne(parameters.agentOs, 'Windows_NT') }}:
       - ${{ if ne(variables['System.TeamProject'], 'public') }}:
@@ -106,13 +187,12 @@ phases:
                   $(_TestArg) $(_PackArg)
                   --publish --ci
                   --noprettyprint
-                  --configuration $(_BuildConfig)
-                  $(_DockerParameter)
-                  --architecture $(_BuildArchitecture)
+                  --configuration ${{ parameters.buildConfiguration }}
+                  --architecture ${{ parameters.buildArchitecture }}
                   $(_LinuxPortable)
                   $(_RuntimeIdentifier)
                   $(_BuildArgs)
-                  $(_AdditionalBuildParameters)
+                  ${{ parameters.additionalBuildParameters }}
                   $(_InternalRuntimeDownloadArgs)
         displayName: Build
 
@@ -121,41 +201,43 @@ phases:
                   $(_TestArg)
                   --pack --publish --ci
                   --noprettyprint
-                  --configuration $(_BuildConfig)
-                  --architecture $(_BuildArchitecture)
+                  --configuration ${{ parameters.buildConfiguration }}
+                  --architecture ${{ parameters.buildArchitecture }}
                   $(_RuntimeIdentifier)
                   $(_BuildArgs)
-                  $(_AdditionalBuildParameters)
+                  ${{ parameters.additionalBuildParameters }}
                   $(_InternalRuntimeDownloadArgs)
         displayName: Build
 
-    - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest'), eq(parameters.agentOs, 'Windows_NT'), ne(variables['PostBuildSign'], 'true')) }}:
-      - task: NuGetCommand@2
-        displayName: Push Visual Studio NuPkgs
-        inputs:
-          command: push
-          packagesToPush: '$(Build.SourcesDirectory)/artifacts/packages/$(_BuildConfig)/NonShipping/VS.*.nupkg'
-          nuGetFeedType: external
-          publishFeedCredentials: 'DevDiv - VS package feed'
-        condition: and(succeeded(), ne(variables['PostBuildSign'], true), eq(variables['_PushToVSFeed'], 'true'), eq(variables['_DotNetPublishToBlobFeed'], 'true'), or(eq(variables['_BuildArchitecture'], 'x64'), eq(variables['_BuildArchitecture'], 'x86')))
+    # In general, only push VS nupkgs if post-build signing is off, and this is not a PR
+    - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest'), ne(variables['PostBuildSign'], 'true')) }}:
+      # In addition, restrict to just Windows x86 or x64 builds (other windows builds do not produce VS redist packages.)
+      - ${{ if and(eq(parameters.agentOs, 'Windows_NT'), in(parameters.buildArchitecture, 'x64', 'x86')) }}:
+        - task: NuGetCommand@2
+          displayName: Push Visual Studio NuPkgs
+          inputs:
+            command: push
+            packagesToPush: '$(Build.SourcesDirectory)/artifacts/packages/${{ parameters.buildConfiguration }}/NonShipping/VS.*.nupkg'
+            nuGetFeedType: external
+            publishFeedCredentials: 'DevDiv - VS package feed'
 
     - task: PublishTestResults@2
       displayName: Publish Test Results
       inputs:
         testRunner: XUnit
-        testResultsFiles: 'artifacts/TestResults/$(_BuildConfig)/*.xml'
+        testResultsFiles: 'artifacts/TestResults/${{ parameters.buildConfiguration }}/*.xml'
         testRunTitle: '$(_AgentOSName)_$(Agent.JobName)'
         platform: '$(BuildPlatform)'
-        configuration: '$(_BuildConfig)'
+        configuration: '${{ parameters.buildConfiguration }}'
       condition: ne(variables['_TestArg'], '')
 
     - task: CopyFiles@2
       displayName: Gather Logs
       inputs:
         SourceFolder: '$(Build.SourcesDirectory)/artifacts'
         Contents: |
-         log/$(_BuildConfig)/**/*
-         TestResults/$(_BuildConfig)/**/*
+         log/${{ parameters.buildConfiguration }}/**/*
+         TestResults/${{ parameters.buildConfiguration }}/**/*
         TargetFolder: '$(Build.ArtifactStagingDirectory)'
       continueOnError: true
       condition: always()