Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add prepare artifacts job; fixup AzDO fetch script #1679

Merged
merged 3 commits into from
Jul 24, 2020
Merged

Add prepare artifacts job; fixup AzDO fetch script #1679

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dc2cb06
Add prepare artifacts job; fixup AzDO fetch script
dagood Jul 3, 2020
274cca0
Skip in public PR validation
dagood Jul 22, 2020
99fc59d
Add right reason for gunzip workaround
dagood Jul 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .vsts.pipelines/builds/matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,14 @@ jobs:
scriptPrefix: ./
scriptSuffix: .sh
setupMac: true

- template: ../jobs/prepare-artifacts.yml
dagood marked this conversation as resolved.
Show resolved Hide resolved
parameters:
# Gather from all jobs that have offline in the matrix. This also defines dependencies.
gatherJobs:
- job: centos71
- job: centos8
- job: debian9
artifactName: Tarball debian9 Online
- job: fedora30
gatherPortableJob: centos71
84 changes: 84 additions & 0 deletions .vsts.pipelines/jobs/prepare-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
parameters:
gatherJobs: []
gatherPortableJob: ''

# downloadBuildConfig by default uses the current build, but can be changed to point at an
# existing/previous build by passing properties like these, instead:
#
# buildType: specific
# buildVersionToDownload: specific
# project: 'public'
# definition: 'source-build-CI'
# buildId: 741400
#
# By defining this in one place, it's easier to set this up if necessary. This may be useful
# because the build takes a long time relative to prepare-artifacts, and we may want to tweak
# prepare-artifacts in case it fails due to a bug in the future. ...It's also nice to have the
# properties listed in this comment even if we don't use them here, because it's hard to figure
# them out from the docs alone--it'll be a nice reference.
downloadBuildConfig:
buildType: current

jobs:
- job: PrepareArtifacts
pool:
vmImage: 'ubuntu-18.04'
dependsOn:
- ${{ each gather in parameters.gatherJobs }}:
- ${{ gather.job }}
timeoutInMinutes: 300
workspace:
clean: all
variables:
artifactStageDir: '$(Build.SourcesDirectory)/artifacts/stage'
nonportableSourceBuiltStageDir: '$(artifactStageDir)/nonportableSourceBuilt'
portableSourceBuiltStageDir: '$(artifactStageDir)/portableSourceBuilt'
allSourceBuiltStageDir: '$(artifactStageDir)/allSourceBuilt'
outputTarballFile: '$(artifactStageDir)/Private.SourceBuilt.Artifacts.$(Build.BuildId).tar.gz'
steps:
- ${{ each gather in parameters.gatherJobs }}:
- task: DownloadBuildArtifacts@0
displayName: 'Download ${{ gather.job }} nonportable'
inputs:
${{ insert }}: ${{ parameters.downloadBuildConfig }}
downloadType: single
artifactName: ${{ coalesce(gather.artifactName, format('Tarball {0} Offline', gather.job)) }}
downloadPath: $(nonportableSourceBuiltStageDir)
allowPartiallySucceededBuilds: true

- task: DownloadBuildArtifacts@0
displayName: 'Download ${{ parameters.gatherPortableJob }} portable'
inputs:
${{ insert }}: ${{ parameters.downloadBuildConfig }}
downloadType: single
artifactName: 'Tarball ${{ parameters.gatherPortableJob }} Offline Portable'
downloadPath: $(portableSourceBuiltStageDir)
allowPartiallySucceededBuilds: true

- script: |
find "$(artifactStageDir)" -type f -exec du -h {} \;
displayName: Show downloaded artifacts

- script: |
set -xeuo pipefail
mkdir -p "$(allSourceBuiltStageDir)"

# Extract all source-built assets into a single place. Overlap and overwrites are expected.
# What matters is the portable ones are copied last and ultimately win.
find \
"$(nonportableSourceBuiltStageDir)" \
"$(portableSourceBuiltStageDir)" \
-iname 'Private.SourceBuilt.Artifacts.*.tar.gz' \
-exec tar -xf {} -C "$(allSourceBuiltStageDir)" \;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about PackageVersions.props file? If last one in wins, it will not reflect the combined files. It seems like this needs to be coupled with #1638

Copy link
Member Author

@dagood dagood Jul 22, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is (the general process of) how @crummel made the previously-source-built tarball we're using now, and I don't see any PVPs, so I assume it worked out... I'm not sure.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's not strictly correct, but it works out because we don't use runtime package versions for anything right now. If we're productizing it, it might be worth looking into a bit. There's probably two things we care about:

  • getting all package versions into PackageVersions.props: I think something like adding unique lines from each individual PackageVersions.props to the output PackageVersions.props would work decently well here and probably shouldn't screw up too much. This currently doesn't matter but would cover us if we ever needed to depend on, e.g., a runtime package's version.
  • not having any conflicting package versions: in theory, this shouldn't happen. Package names and versions should be consistent across RIDs, and I don't think we've seen any real-life cases where this has gone wrong. I can't think offhand of a cheap way to detect this other than a bunch of sed nonsense, but I think it would be worth filing an issue for a future improvement.

I don't think either of these are blockers and #1638 should obviate them anyway.


# Intentionally don't create a top-level directory. Matches earlier versions of this artifact.
cd "$(allSourceBuiltStageDir)"
tar --numeric-owner -czf "$(outputTarballFile)" *
displayName: Create source-built artifacts tar.gz

- publish: '$(outputTarballFile)'
artifact: Private.SourceBuilt.Artifacts

- script: |
tar -tf "$(outputTarballFile)" | sort
displayName: Show tarball contents
16 changes: 9 additions & 7 deletions scripts/fetch-azdo-artifact-tarball.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,21 @@
#!/usr/bin/env bash

set -euo pipefail
# Library. Source this script or copy-paste it into a shell to import its functions.

# Logic is encapsulated to make this function easy to copy-paste to a machine that doesn't have a
# clone of this repository. This may be common when downloading a tarball.
# Set up AzDO authentication if not already done in this environment, download a file, and extract
# it to the current dir. Logic is encapsulated to make this function easy to copy-paste to a machine
# that doesn't have a clone of this repository, which is common when downloading a tarball onto a
# fresh machine to validate it.
#
# Usage: download_tarball <url>
# url: the URL to download. Wrap it in single quotes to avoid issues with special chars.
download_tarball() {
[ "${azdo_build_pat:-}" ] || read -p 'AzDO dnceng PAT with build read permission: ' -s azdo_build_pat
(
set -euo pipefail
url=$1
temp_name=${temp_name:-tarball.tar.gz}

[ "$azdo_build_pat" ] || read -p 'AzDO dnceng PAT with build read permission: ' -s azdo_build_pat

echo; echo 'Starting download...'

curl -o "$temp_name" -u "intentionally_blank:$azdo_build_pat" "$url"
Expand All @@ -30,5 +34,3 @@ download_tarball() {
echo 'Complete!'
)
}

download_tarball "$@"