[Packaging] Deploy build artifacts to github releases

[asfimport]

Reporter: Krisztian Szucs / @kszucs
Assignee: Krisztian Szucs / @kszucs

Subtasks:

• [Packaging] Postfix conda artifacts with target arch
• [Packaging] GPG sign downloaded artifacts

PRs and other links:

• GitHub Pull Request #2109

_{Note: This issue was originally created as ARROW-2676. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Wanted to repeat my comments from GitHub (#1869 (comment)) here to explain the desired workflow:

For each run of the job creation tool (i.e. the current crossbow.py)

• Create a separate tagged branch that contains a YAML file indicating the information about each task created as part of the run. So there should be one entry for each job that was created – the git hash for the task, the CI service used to run the task, etc. It should also indicate if one or more artifacts are expected to be uploaded

• Tasks should be configured with the tag name, and artifacts should be uploaded to GitHub under the tag which should appear as a release on the repo

• We write a status tool which can query the status of a particular run and determine if the run is complete

• We write a tool to download all artifacts associated with a particular run and sign them with the release manager's GPG key

  So here's what I'm expecting:

  $ python crossbow.py ...
  ...
  Job run tag: build-1234
  
  $ python check-build-status.py build-1234
  Task 1 on Travis CI completed
  ...
  Task N on AppVeyor still running
  
  $ python check-build-status.py build-1234
  ...
  All tasks completed
  
  $ python download-build-artifacts.py build-1234 --sign

  Here are the boxes we need to check:

• Can we run each desired task in a particular CI service? (the MVP for this is ARROW-2430)

• We can determine the list of created tasks associated with a particular run

• Each task can upload its artifacts to a deterministic central location (e.g. GitHub), where the artifacts are not commingled with any other run (that's this issue ARROW-2676)

• We can determine whether all the expected artifacts from a particular run have been successfully uploaded (i.e. to GitHub)

• We can, in an automated way and without clicking on a webpage, download all the artifacts from a successful run and GPG sign them for purposes of a release vote

[asfimport]

Phillip Cloud / @cpcloud:
Update on this: @kszucs is making progress.

Let's address each bullet individually:

• Can we run each desired task in a particular CI service?: This is currently possible as of ARROW-2430, as noted.
• We can determine the list of created tasks associated with a particular run: Crossbow is now creating a new branch per job like build-<number>. The individual tasks are prefixed with the job name, and that is followed by the task name. A concrete example is: build-30-conda-linux. This contains the YAML file that is run on some service, in this case Travis CI.
• Each task can upload its artifacts to a deterministic central location (e.g. GitHub), where the artifacts are not commingled with any other run: Because of the way branches are named, this is by definition how the tool works. Here are the artifacts for build-30-conda-linux.
• We can determine whether all the expected artifacts from a particular run have been successfully uploaded (i.e. to GitHub): This is in progress.
• We can, in an automated way and without clicking on a webpage, download all the artifacts from a successful run and GPG sign them for purposes of a release vote: Not yet started, but next up in the queue.

[asfimport]

Wes McKinney / @wesm:
OK, great, thank you for the update. I understand this is a complex endeavor (made more so by the build-up of technical debt in arrow-dist) so wanted to make sure we're breaking down the hard problem into tractable pieces (in the Lao Tzu spirit of "accomplish the great task by a series of small acts")

[asfimport]

Phillip Cloud / @cpcloud:
Issue resolved by pull request 2109
#2109