Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify build yml #36567

Merged
merged 25 commits into from
Jul 25, 2024
Merged

Simplify build yml #36567

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
38aeb2c
update venv reference methodology
scbedd Jul 19, 2024
8a2b75c
remove platform from build step
scbedd Jul 19, 2024
217d394
Merge branch 'update-venv-reference' into always-build
scbedd Jul 22, 2024
1c8ddf8
a _bunch_ of build changes. this is to get our builds all running the…
scbedd Jul 22, 2024
354213e
ok we aren't honoring the conditionals, but we can test the actual bu…
scbedd Jul 22, 2024
d84253d
update indentation
scbedd Jul 22, 2024
0442b24
update condition usage in use-venv
scbedd Jul 22, 2024
2cf398e
remove wonky condition
scbedd Jul 22, 2024
c0ffa9c
adjust packages usage in build-extended-artifacts
scbedd Jul 22, 2024
a2ea84b
Merge remote-tracking branch 'upstream/main' into simplify-build-agen…
scbedd Jul 22, 2024
1a7dcc1
why is this thing having such a hard time processing getting corehttp…
scbedd Jul 23, 2024
10a6127
Merge remote-tracking branch 'upstream/main' into simplify-build-agen…
scbedd Jul 23, 2024
425b744
waiting for confirmation, but pretty certain this will work
scbedd Jul 23, 2024
871e662
Update eng/pipelines/templates/steps/build.yml
scbedd Jul 23, 2024
0c0f06d
update the conditions in build-package-artifacts. the intent is to sa…
scbedd Jul 23, 2024
b358429
correct all the variable references
scbedd Jul 23, 2024
79f95f7
correct condition in set dev build
scbedd Jul 23, 2024
9753eb8
simplify the conditional
scbedd Jul 23, 2024
7ff3b89
cant remove a directory we're operating in
scbedd Jul 23, 2024
105edfe
leave behind the credscan suppression file though
scbedd Jul 23, 2024
2ba740d
clean up all the files in the repo
scbedd Jul 23, 2024
c205cb6
buncha updates!
scbedd Jul 24, 2024
873ffd6
clean up the outputs in build-package-artifacts
scbedd Jul 24, 2024
56ec683
fix the artifact download call
scbedd Jul 24, 2024
387d3d0
restore the python release stages to what they were. my changes were …
scbedd Jul 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions eng/pipelines/templates/jobs/ci.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,7 @@ jobs:
InjectedPackages: $(InjectedPackages)
TestProxy: ${{ parameters.TestProxy }}
BeforeTestSteps:
- task: DownloadPipelineArtifact@2
inputs:
artifactName: 'packages'
targetPath: $(Build.ArtifactStagingDirectory)
- template: /eng/pipelines/templates/steps/download-package-artifacts.yml

- template: ../steps/set-dev-build.yml
parameters:
Expand Down
185 changes: 56 additions & 129 deletions eng/pipelines/templates/jobs/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,141 +61,68 @@ parameters:
- name: GenerateApiReviewForManualOnly
type: boolean
default: false
- name: AdvancedBuild
type: boolean
default: false

jobs:
- job: 'Build_Linux'
timeoutInMinutes: 90

- ${{ if eq(parameters['AdvancedBuild'], false) }}:
- job: 'Build'
timeoutInMinutes: 90

pool:
name: $(LINUXPOOL)
image: $(LINUXVMIMAGE)
os: linux

steps:
- template: /eng/pipelines/templates/steps/targeting-string-resolve.yml
parameters:
BuildTargetingString: ${{ parameters.BuildTargetingString }}

- template: ../steps/build-package-artifacts.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
Artifacts: ${{ parameters.Artifacts }}

- ${{ if eq(parameters['AdvancedBuild'], true) }}:
- job: 'Build_Linux'
timeoutInMinutes: 90

pool:
name: $(LINUXPOOL)
image: $(LINUXVMIMAGE)
os: linux

steps:
- template: /eng/pipelines/templates/steps/targeting-string-resolve.yml
parameters:
BuildTargetingString: ${{ parameters.BuildTargetingString }}

- template: ../steps/build-package-artifacts.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
Artifacts: ${{ parameters.Artifacts }}
ArtifactSuffix: linux
UseVirtualEnvironment: false

- job: 'Build_Windows'
timeoutInMinutes: 90

pool:
name: $(WINDOWSPOOL)
image: $(WINDOWSVMIMAGE)
os: windows

steps:
- template: /eng/pipelines/templates/steps/targeting-string-resolve.yml
parameters:
BuildTargetingString: ${{ parameters.BuildTargetingString }}

- template: ../steps/build-package-artifacts.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
Artifacts: ${{ parameters.Artifacts }}
ArtifactSuffix: windows
UseVirtualEnvironment: false

- job: 'Build_MacOS'
timeoutInMinutes: 90

pool:
name: $(MACPOOL)
vmImage: $(MACVMIMAGE)
os: macOS

steps:
- template: /eng/pipelines/templates/steps/targeting-string-resolve.yml
parameters:
BuildTargetingString: ${{ parameters.BuildTargetingString }}
pool:
name: $(LINUXPOOL)
image: $(LINUXVMIMAGE)
os: linux

- template: ../steps/build-package-artifacts.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
Artifacts: ${{ parameters.Artifacts }}
ArtifactSuffix: mac
UseVirtualEnvironment: false
steps:
- template: /eng/pipelines/templates/steps/build.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BuildTargetingString: ${{ parameters.BuildTargetingString }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
ArtifactSuffix: linux
Artifacts: ${{ parameters.Artifacts }}

- job: 'CoalesceBuildArtifacts'
displayName: Combine Built Artifacts
dependsOn:
- 'Build_Linux'
- 'Build_Windows'
- 'Build_MacOS'
timeoutInMinutes: 90
- job: 'Build_Windows'
timeoutInMinutes: 90

pool:
name: $(LINUXPOOL)
image: $(LINUXVMIMAGE)
os: linux
pool:
name: $(WINDOWSPOOL)
image: $(WINDOWSVMIMAGE)
os: windows

steps:
- task: DownloadPipelineArtifact@2
inputs:
artifactName: 'packages_windows'
targetPath: $(Build.ArtifactStagingDirectory)/packages
steps:
- template: /eng/pipelines/templates/steps/build.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BuildTargetingString: ${{ parameters.BuildTargetingString }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
ArtifactSuffix: windows
Artifacts: ${{ parameters.Artifacts }}

- task: DownloadPipelineArtifact@2
inputs:
artifactName: 'packages_mac'
targetPath: $(Build.ArtifactStagingDirectory)/packages
- job: 'Build_MacOS'
timeoutInMinutes: 90

- task: DownloadPipelineArtifact@2
inputs:
artifactName: 'packages_linux'
targetPath: $(Build.ArtifactStagingDirectory)/packages
pool:
name: $(MACPOOL)
vmImage: $(MACVMIMAGE)
os: macOS

- template: /eng/common/pipelines/templates/steps/publish-1es-artifact.yml
parameters:
ArtifactPath: '$(Build.ArtifactStagingDirectory)/packages'
ArtifactName: 'packages'
steps:
- template: /eng/pipelines/templates/steps/build.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
BuildTargetingString: ${{ parameters.BuildTargetingString }}
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
TestPipeline: ${{ parameters.TestPipeline }}
ArtifactSuffix: mac
Artifacts: ${{ parameters.Artifacts }}

- job: 'Build_Extended'
displayName: Build Extended
dependsOn:
- ${{ if eq(parameters['AdvancedBuild'], true) }}:
- 'CoalesceBuildArtifacts'
- ${{ else }}:
- 'Build'
- 'Build_Linux'
- 'Build_Windows'
- 'Build_MacOS'

timeoutInMinutes: 90

Expand All @@ -205,6 +132,8 @@ jobs:
os: linux

steps:
- template: /eng/pipelines/templates/steps/download-package-artifacts.yml

- template: /eng/pipelines/templates/steps/targeting-string-resolve.yml
parameters:
BuildTargetingString: ${{ parameters.BuildTargetingString }}
Expand Down Expand Up @@ -272,10 +201,9 @@ jobs:
OsVmImage: azsdk-pool-mms-ubuntu-2004-1espt
Pool: azsdk-pool-mms-ubuntu-2004-general
DependsOn:
- ${{ if eq(parameters['AdvancedBuild'], true) }}:
- 'CoalesceBuildArtifacts'
- ${{ else }}:
- 'Build'
- 'Build_Linux'
- 'Build_Windows'
- 'Build_MacOS'
MatrixConfigs: ${{ parameters.MatrixConfigs }}
MatrixFilters: ${{ parameters.MatrixFilters }}
MatrixReplace: ${{ parameters.MatrixReplace }}
Expand Down Expand Up @@ -327,10 +255,9 @@ jobs:
CloudConfig:
Cloud: Public
DependsOn:
- ${{ if eq(parameters['AdvancedBuild'], true) }}:
- 'CoalesceBuildArtifacts'
- ${{ else }}:
- 'Build'
- 'Build_Linux'
- 'Build_Windows'
- 'Build_MacOS'
AdditionalParameters:
BuildTargetingString: ${{ parameters.BuildTargetingString }}
ServiceDirectory: ${{ parameters.ServiceDirectory }}
Expand Down
7 changes: 2 additions & 5 deletions eng/pipelines/templates/jobs/regression.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,13 +79,10 @@ jobs:
displayName: 'Use Python 3.9'
inputs:
versionSpec: '3.9'

- template: /eng/pipelines/templates/steps/use-venv.yml

- task: DownloadPipelineArtifact@2
inputs:
artifactName: 'packages'
targetPath: $(Build.ArtifactStagingDirectory)
- template: /eng/pipelines/templates/steps/download-package-artifacts.yml

- pwsh: |
mkdir -p $(TEST_PROXY_FOLDER)
Expand Down
2 changes: 1 addition & 1 deletion eng/pipelines/templates/stages/archetype-python-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ stages:
deploy:
steps:
- checkout: self

- download: current
artifact: ${{parameters.ArtifactName}}
timeoutInMinutes: 5
Expand Down Expand Up @@ -294,7 +295,6 @@ stages:
name: azsdk-pool-mms-ubuntu-2004-general
os: linux
steps:
- checkout: none
- download: current
artifact: ${{parameters.ArtifactName}}
timeoutInMinutes: 5
Expand Down
4 changes: 0 additions & 4 deletions eng/pipelines/templates/stages/archetype-sdk-client.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,6 @@ parameters:
- name: GenerateApiReviewForManualOnly
type: boolean
default: false
- name: AdvancedBuild
type: boolean
default: false
- name: oneESTemplateTag
type: string
default: release
Expand Down Expand Up @@ -113,7 +110,6 @@ extends:
ValidateFormatting: ${{ parameters.ValidateFormatting }}
TestProxy: ${{ parameters.TestProxy }}
GenerateApiReviewForManualOnly: ${{ parameters.GenerateApiReviewForManualOnly }}
AdvancedBuild: ${{ parameters.AdvancedBuild }}

variables:
- template: /eng/pipelines/templates/variables/globals.yml
Expand Down
5 changes: 0 additions & 5 deletions eng/pipelines/templates/steps/build-extended-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,6 @@ parameters:

# Please use `$(TargetingString)` to refer to the python packages glob string. This was previously `${{ parameters.BuildTargetingString }}`.
steps:
- task: DownloadPipelineArtifact@2
inputs:
artifactName: 'packages'
targetPath: $(Build.ArtifactStagingDirectory)

- task: UsePythonVersion@0
displayName: 'Use Python 3.11'
inputs:
Expand Down
Loading