Introduce "job" version of prepare-pipelines.yml, with "pool" defined…

… to use Ubuntu 20.04 (#4930) This PR, in tandem with PRs #4915 and #4916, addresses #4888. This PR subsumes the abandoned PR of #4911. Specifically, this PR ensures that `prepare-pipelines.yml` is configured to use specific `pool`, set to use Ubuntu 20.04: ``` pool: name: azsdk-pool-mms-ubuntu-2004-general vmImage: MMSUbuntu20.04 ``` Without the [pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines) definition pointing to Ubuntu 20.04, the steps within the file used an implicit default, which resolves to `ubuntu-latest`. This caused breakage as explained in #4888. Setting the pool to Ubuntu 20.04 will allow us to: - Immediately unblock the language repos `prepare-pipelines.yml` pipelines [1], as Ubuntu 20.04 has the .NET Core version used by it [2]; - Allow us to migrate our tooling to .NET 6 as both Ubuntu 20.04 and `ubuntu-latest`, which is Ubuntu 22.04 [3], have .NET on them [4]. This step is accomplished by PRs #4916 and #4915; - Allow us to migrate to Ubuntu 22.04 once we migrate to .NET 6. ## Technical considerations of the changes - The current `prepare-pipelines.yml` has only [steps](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps?view=azure-pipelines) definition, which does not support `pool` definition. One needs to wrap the steps in [job](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job?view=azure-pipelines) definition to support `pool` [5]. - Because the `.yml` file is located in `eng/common/pipelines/templates/steps/` directory (note the `steps`), we cannot wrap the steps in job and leave it there; the introduction of `job` necessitates introducing the changes in `eng/common/pipelines/templates/jobs/` directory instead (note the `jobs`), which is done in this PR. - Because this PR introduces a new file, once this PR is merged, the language-specific `prepare-pipelines.yml` language pipelines need to be rewired to point to the newly introduced file. This approach was proposed in #4911 (comment). - Language repos PRs doing the rewiring: Azure/azure-sdk-for-net#32999 ... more to come! ## Testing done I have confirmed the proposed modification to the `.yml` file will work [by modifying the existing `prepare-pipelines.yml` directly](Azure/azure-sdk-for-java@f8cef4c) to have exactly the same contents as the `/jobs/prepare-pipelines.yml` introduced in this PR, and observing the [relevant build succeeds on azure-sdk-for-java](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2044120&view=logs&j=7f42699e-eeb0-56c8-40c2-c88ae4093e4f&t=a02502cb-238e-5603-14ee-8bb7bd07f0c6): ![image](https://user-images.githubusercontent.com/4429827/206805076-b01baf33-871a-4ff2-816e-fa8458fa0063.png) while it failed before this change: ![image](https://user-images.githubusercontent.com/4429827/206805129-b3bd8ebb-b97b-444b-8785-e6edf68c9720.png) # Footnotes [1] for example, [internal / java / prepare-pipelines](https://dev.azure.com/azure-sdk/internal/_build?definitionId=2158&_a=summary) pipeline, or [internal / net / prepare-pipelines](https://dev.azure.com/azure-sdk/internal/_build?definitionId=2179) pipeline. [2] as evidenced by the [Ubuntu 20.04 image software page](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#net-core-sdk), it has .NET Core 3.1. The pool definition value points to a 1ES hosted image which has the software listed on that page, as explained by this comment: #4888 (comment). [3] See this comment: #4888 (comment). [4] as evidenced by the [Ubuntu 22.04 image software page](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#net-core-sdk). [5] Observe that on [pool definition](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines) YAML reference page, we can read: > Properties that use this definition: pipeline.pool, stages.stage.pool, jobs.job.pool, jobs.deployment.pool. i.e. `steps.pool` is not listed.
Azure · Dec 12, 2022 · 1e7a3a9 · 1e7a3a9
1 parent 7dbcb86
commit 1e7a3a9
Show file tree

Hide file tree

Showing 2 changed files with 197 additions and 0 deletions.
diff --git a/eng/common/pipelines/templates/jobs/prepare-pipelines.yml b/eng/common/pipelines/templates/jobs/prepare-pipelines.yml
@@ -0,0 +1,192 @@
+parameters:
+  - name: Repository
+    type: string
+    default: $(Build.Repository.Name)
+  - name: Prefix
+    type: string
+  - name: CIConventionOptions
+    type: string
+    default: ''
+  - name: UPConventionOptions
+    type: string
+    default: ''
+  - name: TestsConventionOptions
+    type: string
+    default: ''
+  - name: GenerateUnifiedWeekly
+    type: boolean
+    default: false
+
+jobs:
+- job: PreparePipelines
+  pool:
+    name: azsdk-pool-mms-ubuntu-2004-general
+    vmImage: MMSUbuntu20.04
+  steps:
+    - template: install-pipeline-generation.yml
+    - template: /eng/common/pipelines/templates/steps/set-default-branch.yml
+    # This covers our public repos.
+    - ${{ if not(endsWith(parameters.Repository, '-pr'))}}:
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project public
+          --prefix ${{parameters.Prefix}}
+          --devopspath "\${{parameters.Prefix}}"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention ci
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          ${{parameters.CIConventionOptions}}
+        displayName: Create CI Pipelines for Public Repository
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}
+          --devopspath "\${{parameters.Prefix}}"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention up
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          ${{parameters.UPConventionOptions}}
+        displayName: Create UP Pipelines for Public Repository
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}
+          --devopspath "\${{parameters.Prefix}}"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention tests
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          ${{parameters.TestsConventionOptions}}
+        displayName: Create Live Test Pipelines for Public Repository
+        condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}
+          --devopspath "\${{parameters.Prefix}}"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention testsweekly
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          ${{parameters.TestsConventionOptions}}
+        displayName: Create Weekly (Multi-Cloud) Live Test Pipelines for Public Repository
+        condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}
+          --devopspath "\${{parameters.Prefix}}"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention upweekly
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          ${{parameters.UPConventionOptions}}
+        displayName: Create Weekly (Multi-Cloud) Unified Test Pipelines for Public Repository
+        condition: and(succeeded(), eq(${{parameters.GenerateUnifiedWeekly}},true))
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+
+
+    # This covers our -pr repositories.
+    - ${{ if endsWith(parameters.Repository, '-pr')}}:
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}-pr
+          --devopspath "\${{parameters.Prefix}}\pr"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention ci
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          --no-schedule
+          ${{parameters.CIConventionOptions}}
+        displayName: Create CI Pipelines for Private Repository
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}-pr
+          --devopspath "\${{parameters.Prefix}}\pr"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention up
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          --no-schedule
+          ${{parameters.UPConventionOptions}}
+        displayName: Create UP Pipelines for Private Repository
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+      - script: >
+          $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+          --organization https://dev.azure.com/azure-sdk
+          --project internal
+          --prefix ${{parameters.Prefix}}-pr
+          --devopspath "\${{parameters.Prefix}}\pr"
+          --path $(System.DefaultWorkingDirectory)/sdk
+          --endpoint Azure
+          --repository ${{parameters.Repository}}
+          --convention tests
+          --agentpool Hosted
+          --branch refs/heads/$(DefaultBranch)
+          --patvar PATVAR
+          --set-managed-variables
+          --debug
+          --no-schedule
+          ${{parameters.TestsConventionOptions}}
+        displayName: Create Live Test Pipelines for Private Repository
+        condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
+        env:
+          PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
diff --git a/eng/common/pipelines/templates/steps/prepare-pipelines.yml b/eng/common/pipelines/templates/steps/prepare-pipelines.yml
@@ -1,3 +1,8 @@
+# As of 12/9/2022 this file is going to become obsolete and then deleted soon,
+# superseded by ../jobs/prepare-pipelines.yml, which includes all of the
+# logic in this file.
+# For more, please see: https://github.com/Azure/azure-sdk-tools/issues/4888
+
 parameters:
   - name: Repository
     type: string