From 1e7a3a9e6d7acc537c49cde2410760b5ab854afc Mon Sep 17 00:00:00 2001 From: Konrad Jamrozik Date: Mon, 12 Dec 2022 15:48:26 -0800 Subject: [PATCH] Introduce "job" version of prepare-pipelines.yml, with "pool" defined to use Ubuntu 20.04 (#4930) This PR, in tandem with PRs #4915 and #4916, addresses https://github.com/Azure/azure-sdk-tools/issues/4888. This PR subsumes the abandoned PR of #4911. Specifically, this PR ensures that `prepare-pipelines.yml` is configured to use specific `pool`, set to use Ubuntu 20.04: ``` pool: name: azsdk-pool-mms-ubuntu-2004-general vmImage: MMSUbuntu20.04 ``` Without the [pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines) definition pointing to Ubuntu 20.04, the steps within the file used an implicit default, which resolves to `ubuntu-latest`. This caused breakage as explained in #4888. Setting the pool to Ubuntu 20.04 will allow us to: - Immediately unblock the language repos `prepare-pipelines.yml` pipelines [1], as Ubuntu 20.04 has the .NET Core version used by it [2]; - Allow us to migrate our tooling to .NET 6 as both Ubuntu 20.04 and `ubuntu-latest`, which is Ubuntu 22.04 [3], have .NET on them [4]. This step is accomplished by PRs #4916 and #4915; - Allow us to migrate to Ubuntu 22.04 once we migrate to .NET 6. ## Technical considerations of the changes - The current `prepare-pipelines.yml` has only [steps](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps?view=azure-pipelines) definition, which does not support `pool` definition. One needs to wrap the steps in [job](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job?view=azure-pipelines) definition to support `pool` [5]. - Because the `.yml` file is located in `eng/common/pipelines/templates/steps/` directory (note the `steps`), we cannot wrap the steps in job and leave it there; the introduction of `job` necessitates introducing the changes in `eng/common/pipelines/templates/jobs/` directory instead (note the `jobs`), which is done in this PR. - Because this PR introduces a new file, once this PR is merged, the language-specific `prepare-pipelines.yml` language pipelines need to be rewired to point to the newly introduced file. This approach was proposed in https://github.com/Azure/azure-sdk-tools/pull/4911#discussion_r1043940090. - Language repos PRs doing the rewiring: https://github.com/Azure/azure-sdk-for-net/pull/32999 ... more to come! ## Testing done I have confirmed the proposed modification to the `.yml` file will work [by modifying the existing `prepare-pipelines.yml` directly](https://github.com/Azure/azure-sdk-for-java/commit/f8cef4c924fa112afc22c53cdb633ba256873e57) to have exactly the same contents as the `/jobs/prepare-pipelines.yml` introduced in this PR, and observing the [relevant build succeeds on azure-sdk-for-java](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2044120&view=logs&j=7f42699e-eeb0-56c8-40c2-c88ae4093e4f&t=a02502cb-238e-5603-14ee-8bb7bd07f0c6): ![image](https://user-images.githubusercontent.com/4429827/206805076-b01baf33-871a-4ff2-816e-fa8458fa0063.png) while it failed before this change: ![image](https://user-images.githubusercontent.com/4429827/206805129-b3bd8ebb-b97b-444b-8785-e6edf68c9720.png) # Footnotes [1] for example, [internal / java / prepare-pipelines](https://dev.azure.com/azure-sdk/internal/_build?definitionId=2158&_a=summary) pipeline, or [internal / net / prepare-pipelines](https://dev.azure.com/azure-sdk/internal/_build?definitionId=2179) pipeline. [2] as evidenced by the [Ubuntu 20.04 image software page](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#net-core-sdk), it has .NET Core 3.1. The pool definition value points to a 1ES hosted image which has the software listed on that page, as explained by this comment: https://github.com/Azure/azure-sdk-tools/issues/4888#issuecomment-1342033218. [3] See this comment: https://github.com/Azure/azure-sdk-tools/issues/4888#issuecomment-1342033218. [4] as evidenced by the [Ubuntu 22.04 image software page](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#net-core-sdk). [5] Observe that on [pool definition](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines) YAML reference page, we can read: > Properties that use this definition: pipeline.pool, stages.stage.pool, jobs.job.pool, jobs.deployment.pool. i.e. `steps.pool` is not listed. --- .../templates/jobs/prepare-pipelines.yml | 192 ++++++++++++++++++ .../templates/steps/prepare-pipelines.yml | 5 + 2 files changed, 197 insertions(+) create mode 100644 eng/common/pipelines/templates/jobs/prepare-pipelines.yml diff --git a/eng/common/pipelines/templates/jobs/prepare-pipelines.yml b/eng/common/pipelines/templates/jobs/prepare-pipelines.yml new file mode 100644 index 00000000000..e776ade7c50 --- /dev/null +++ b/eng/common/pipelines/templates/jobs/prepare-pipelines.yml @@ -0,0 +1,192 @@ +parameters: + - name: Repository + type: string + default: $(Build.Repository.Name) + - name: Prefix + type: string + - name: CIConventionOptions + type: string + default: '' + - name: UPConventionOptions + type: string + default: '' + - name: TestsConventionOptions + type: string + default: '' + - name: GenerateUnifiedWeekly + type: boolean + default: false + +jobs: +- job: PreparePipelines + pool: + name: azsdk-pool-mms-ubuntu-2004-general + vmImage: MMSUbuntu20.04 + steps: + - template: install-pipeline-generation.yml + - template: /eng/common/pipelines/templates/steps/set-default-branch.yml + # This covers our public repos. + - ${{ if not(endsWith(parameters.Repository, '-pr'))}}: + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project public + --prefix ${{parameters.Prefix}} + --devopspath "\${{parameters.Prefix}}" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention ci + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + ${{parameters.CIConventionOptions}} + displayName: Create CI Pipelines for Public Repository + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}} + --devopspath "\${{parameters.Prefix}}" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention up + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + ${{parameters.UPConventionOptions}} + displayName: Create UP Pipelines for Public Repository + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}} + --devopspath "\${{parameters.Prefix}}" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention tests + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + ${{parameters.TestsConventionOptions}} + displayName: Create Live Test Pipelines for Public Repository + condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}','')) + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}} + --devopspath "\${{parameters.Prefix}}" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention testsweekly + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + ${{parameters.TestsConventionOptions}} + displayName: Create Weekly (Multi-Cloud) Live Test Pipelines for Public Repository + condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}','')) + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}} + --devopspath "\${{parameters.Prefix}}" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention upweekly + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + ${{parameters.UPConventionOptions}} + displayName: Create Weekly (Multi-Cloud) Unified Test Pipelines for Public Repository + condition: and(succeeded(), eq(${{parameters.GenerateUnifiedWeekly}},true)) + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + + + # This covers our -pr repositories. + - ${{ if endsWith(parameters.Repository, '-pr')}}: + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}}-pr + --devopspath "\${{parameters.Prefix}}\pr" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention ci + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + --no-schedule + ${{parameters.CIConventionOptions}} + displayName: Create CI Pipelines for Private Repository + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}}-pr + --devopspath "\${{parameters.Prefix}}\pr" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention up + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + --no-schedule + ${{parameters.UPConventionOptions}} + displayName: Create UP Pipelines for Private Repository + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) + - script: > + $(Pipeline.Workspace)/pipeline-generator/pipeline-generator + --organization https://dev.azure.com/azure-sdk + --project internal + --prefix ${{parameters.Prefix}}-pr + --devopspath "\${{parameters.Prefix}}\pr" + --path $(System.DefaultWorkingDirectory)/sdk + --endpoint Azure + --repository ${{parameters.Repository}} + --convention tests + --agentpool Hosted + --branch refs/heads/$(DefaultBranch) + --patvar PATVAR + --set-managed-variables + --debug + --no-schedule + ${{parameters.TestsConventionOptions}} + displayName: Create Live Test Pipelines for Private Repository + condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}','')) + env: + PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat) \ No newline at end of file diff --git a/eng/common/pipelines/templates/steps/prepare-pipelines.yml b/eng/common/pipelines/templates/steps/prepare-pipelines.yml index ff40670f4be..ca98ff9cb6b 100644 --- a/eng/common/pipelines/templates/steps/prepare-pipelines.yml +++ b/eng/common/pipelines/templates/steps/prepare-pipelines.yml @@ -1,3 +1,8 @@ +# As of 12/9/2022 this file is going to become obsolete and then deleted soon, +# superseded by ../jobs/prepare-pipelines.yml, which includes all of the +# logic in this file. +# For more, please see: https://github.com/Azure/azure-sdk-tools/issues/4888 + parameters: - name: Repository type: string