ARROW-2676: [Packaging] Deploy build artifacts to github releases #2109

kszucs · 2018-06-06T08:19:24Z

~~Added a new task which trigger crossbow builds on master@crossbow.~~
~~See travis output https://travis-ci.org/kszucs/crossbow/builds/388667590~~

Here are the boxes we need to check:

Create a separate tagged branch that contains a YAML file indicating the information about each task created as part of the run. So there should be one entry for each job that was created -- the git hash for the task, the CI service used to run the task, etc. It should also indicate if one or more artifacts are expected to be uploaded
Write a status tool which can query the status of a particular run and determine if the run is complete (needs cleanup)
Can we run each desired task in a particular CI service
We can determine the list of created tasks associated with a particular run
Tasks should be configured with the tag name, and artifacts should be uploaded to GitHub under the tag which should appear as a release on the repo
Each task can upload its artifacts to a deterministic central location (e.g. GitHub), where the artifacts are not commingled with any other run
-> only linux packages are failing, I suggest resolving it in a subsequent PR (issue https://issues.apache.org/jira/browse/ARROW-2713)
~~We can determine whether all the expected artifacts from a particular run have been successfully uploaded (i.e. to GitHub)~~ to be done in https://issues.apache.org/jira/browse/ARROW-2724
We can download all the artifacts from a successful run and GPG sign them for purposes of a release vote

Example of artifacts available here https://github.com/kszucs/crossbow/releases
Jobs and tasks here https://github.com/kszucs/crossbow/branches
Job definition here https://github.com/kszucs/crossbow/blob/build-36/job.yml

codecov-io · 2018-06-10T15:23:47Z

Codecov Report

Merging #2109 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2109      +/-   ##
==========================================
- Coverage   86.39%   86.39%   -0.01%     
==========================================
  Files         230      230              
  Lines       40706    40706              
==========================================
- Hits        35167    35166       -1     
- Misses       5539     5540       +1

Impacted Files	Coverage Δ
cpp/src/arrow/util/thread-pool-test.cc	`98.91% <0%> (-0.55%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ef9888...87860d0. Read the comment docs.

wesm · 2018-06-12T14:46:09Z

dev/tasks/config-nightlies/travis.linux.yml

+      wheel-linux \
+      wheel-win \
+      wheel-osx \
+      linux-packages


I'm not sure about this. I was thinking we would want to schedule nightlies using a cronjob of some kind. Or at least the manifest of nightly jobs would be specified in a YAML file someplace

This would be just for convenience, to use travis' cron instead of a self hosted one.
BTW I'm just refactoring the branching logic to support tagging / deploying (github releases).

This seems like a fine place to keep the nightly jobs list for now. There's nothing preventing us from moving this to a non-travis cron in the future if there's some reason we would need to do that.

wesm · 2018-06-12T14:52:11Z

dev/tasks/tasks.yml

+  #   branch: defaults to name
+  #   platform: osx|linux|win
+  #   template: path of jinja2 templated yml
+  #   params: optional extra parameters


Having the template be a .travis.yml or .appveyor.yml file only in need of template processing seems potentially inflexible to me. Here is a possible alternative:

tasks.yml provides the path to a task definition file (which could be YAML)

The task definition file contains the platform, parameters, and additional information needed to process the task. The idea here would be to make the task definition file more modular; so simple tasks could just be a templated .travis.yml, but we wouldn't be constraining ourselves to that model

My guess is that as time goes on, more of the logic for each task will move from templates to Python code, as a means of improving code reuse.

I've already lost in the YAML forest :)
I'm trying to update this PR do deploy the artifacts - hopefully until the sync, but these ci-test-loops are insanely time consuming.

kszucs · 2018-06-13T14:16:19Z

@wesm Rigth now I'm struggling with passing appveyor auth_tokens, but a couple of artifacts are already uploaded as release assets: https://github.com/kszucs/crossbow/releases

Here https://github.com/kszucs/crossbow/blob/build-30/job.yml is an example yml containing the submitted job's state.

There is draft for querying tasks statuses too.

kszucs · 2018-06-13T14:36:27Z

@wesm @cpcloud Do we want to support other git hosting services or just github? Because we might consider to drop pygit2 dependency and use only github's API.

cpcloud · 2018-06-14T19:09:53Z

@kszucs Let's stick with github for now and we can extend to other if the need arises.

cpcloud

Looking mostly good! Couple of questions/comments.

cpcloud · 2018-06-18T18:55:49Z

cpp/CMakeLists.txt

@@ -343,6 +343,9 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${ARROW_CXXFLAGS}")
 # For any C code, use the same flags.
 set(CMAKE_C_FLAGS "${CMAKE_CXX_FLAGS}")

+# Remove --std=c++11 to avoid errors from C compilers
+string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})


FYI, this is a temporary fix until we can address ARROW-2720. This is fragile if (with CMAKE_CXX_STANDARD set to 11) -std=gnu++11 ends up in CMAKE_C_FLAGS.

This is removed in master (I removed it in a patch recently). I think this is a rebase artifact and should be removed

Yep, that removal broke osx builds though. That's why we added it back.

It broke osx wheel and/or conda builds, not CI builds (obviously) though.

That's super weird, how did that flag end up in CMAKE_C_FLAGS? What kind of error did it cause?

I don't exactly what's happening, but I know that we are setting CMAKE_CXX_STANDARD to 11 and then settings CMAKE_C_FLAGS to CMAKE_CXX_FLAGS, which might explain how the std flag is ending up there. However, this doesn't show up on any NIX systems except OSX which is why I'm not 100% sure what's happening.

The error you get is error: invalid argument '-std=c++11' not allowed with 'C/ObjC' when the ae.c file is being compiled during the plasma compilation. Without a mac to figure out what's going on, it's pretty hard for me to tell what the heck is happening.

Oh I see. I think gcc and clang just ignore the flag when compiling C code. This suggests we should be setting up the CMAKE_C_FLAGS separate from CXX. I'm opening a JIRA

https://issues.apache.org/jira/browse/ARROW-2730

Great! Thanks.

cpcloud · 2018-06-18T18:56:20Z

dev/tasks/README.md

@@ -106,7 +106,7 @@ The script does the following:
   $ git clone https://github.com/kszucs/crossbow

   $ cd arrow/dev/tasks
-   $ python crossbow.py
+   $ python crossbow.py submit <task names>


Can you give an example here instead of a placeholder?

cpcloud · 2018-06-18T18:56:28Z

dev/tasks/README.md

@@ -115,7 +115,7 @@ The script does the following:

   ```bash
   git checkout ARROW-<ticket number>
-   python dev/tasks/crossbow.py --dry-run
+   python dev/tasks/crossbow.py submit --dry-run <task names>


Same as above.

cpcloud · 2018-06-18T18:58:07Z

dev/tasks/README.md

@@ -124,85 +124,56 @@ The script does the following:
 3. Reads and renders the required build configurations with the parameters
   substituted.
 2. Create a commit per build configuration to its own branch. For example


Is this still correct? Isn't it now "A branch per task, prefixed with the job id"?

cpcloud · 2018-06-18T19:00:32Z

dev/tasks/config-nightlies/travis.linux.yml

+
+    # Configure conda.
+    - |
+      echo ""


Any particular reason why we're echoing a newline here?

Copy paste from conda forge. This newline probably makes the output more readable.

cpcloud · 2018-06-18T19:04:17Z

dev/tasks/config-nightlies/travis.linux.yml

+      wheel-linux \
+      wheel-win \
+      wheel-osx \
+      linux-packages


This seems like a fine place to keep the nightly jobs list for now. There's nothing preventing us from moving this to a non-travis cron in the future if there's some reason we would need to do that.

cpcloud · 2018-06-18T19:04:37Z

dev/tasks/crossbow.py

+#     format="[%(asctime)s] %(levelname)s Crossbow %(message)s",
+#     datefmt="%H:%M:%S",
+#     stream=click.get_text_stream('stdout')
+# )


Any reason not to just delete this code instead of leaving it here commented out?

Well, the previous solution used logging instead of click.echo. It was Wes' request, but generated a lot of noise, so I've switched to directly printing to stdout.
Honestly I'd remove it :)

cpcloud · 2018-06-18T19:05:01Z

dev/tasks/crossbow.py

-
-
-class Target(object):
+class Repo(object):


Are we really trying to be python 2 compatible?

cpcloud · 2018-06-18T19:07:20Z

dev/tasks/crossbow.py

+    def next_job_id(self, prefix):
+        """Auto increments the branch's identifier based on the prefix"""
+        pattern = re.compile(prefix + '-(\d+)')
+        matches = list(filter(None, map(pattern.match, self.repo.branches)))


Is there away to get the most recently created branch, from the server, so that we don't have to look at every branch that exists every time we create a new one?

I'm not aware of any, but I don't think we should worry about that - with libgit2 traversing is really fast.
If it's going to be a problem, we can address it later.

Sounds good.

cpcloud · 2018-06-18T19:12:03Z

dev/tasks/linux-packages/travis.linux.yml

+  provider: releases
+  api_key: $CROSSBOW_GITHUB_TOKEN
+  file_glob: true
+  file: /path/to/pachages/*.tar.gz


Does this need to be changed?

After the build passes.

cpcloud · 2018-06-18T19:23:04Z

dev/tasks/README.md

@@ -106,7 +106,7 @@ The script does the following:
   $ git clone https://github.com/kszucs/crossbow

   $ cd arrow/dev/tasks
-   $ python crossbow.py
+   $ python crossbow.py submit <task names>


Is there a way that we can make the submit subcommand show a better error message if the task isn't valid?

Right now I get this:

$ python crossbow.py submit --dry-run linux-packges Traceback (most recent call last): File "crossbow.py", line 409, in <module> crossbow(auto_envvar_prefix='CROSSBOW') File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "crossbow.py", line 338, in submit tasks = {name: tasks[name] for name in task_names} File "crossbow.py", line 338, in <dictcomp> tasks = {name: tasks[name] for name in task_names} KeyError: 'linux-packges'

Addressed in kszucs#3

cpcloud · 2018-06-18T19:26:49Z

dev/tasks/crossbow.py

+        return Job(tasks=tasks, branch=data.get('branch'))
+
+    def files(self):
+        return {'job.yml': yaml.dump(self.to_dict(), default_flow_style=False)}


 # this should be the mailing list
 MESSAGE_EMAIL = '[email protected]'


Can we instead 1) make this configurable via an environment variable, and 2) default to the email of the git user triggering the builds?

I can submit a patch to your fork for these changes.

changes here https://github.com/kszucs/arrow/pull/2/files

cpcloud · 2018-06-18T19:41:18Z

dev/tasks/README.md

@@ -80,12 +80,12 @@ submission. The tasks are defined in `tasks.yml`
 6. Install the python dependencies for the script:



One thing that is worth nothing in this README.md is that you need at least one commit in your queue repo or else this happens:

Traceback (most recent call last): File "crossbow.py", line 413, in <module> crossbow(auto_envvar_prefix='CROSSBOW') File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "crossbow.py", line 356, in submit queue.put(job) File "crossbow.py", line 186, in put branch = self._create_branch(task.branch, files=task.files()) File "crossbow.py", line 158, in _create_branch parents = parents or [self.head] File "crossbow.py", line 96, in head return self.repo.head.target _pygit2.GitError: reference 'refs/heads/master' not found

Is there any way that we can automate this away? If not, make sure to add a note about it in the README.

We don't need parent commits, it just makes the graph nicer.

kszucs · 2018-06-18T21:00:33Z

@cpcloud https://github.com/kszucs/crossbow/releases/tag/build-95 26 artifacts, is that correct?

(not counting linux packages)

cpcloud · 2018-06-18T21:18:51Z

@kszucs There should be two each of pyarrow and arrow-cpp windows packages: one for python 3.5 and one for python 3.6 for a total of 4 windows conda packages + parquet-cpp

cpcloud · 2018-06-19T20:25:33Z

Restarted the travis build

kszucs · 2018-06-20T16:32:54Z

@cpcloud All of the conda and whl artifacts are uploaded now https://github.com/kszucs/crossbow/releases
Linux pkgs afe still failing, here is the tracking issue https://issues.apache.org/jira/browse/ARROW-2713

cpcloud · 2018-06-20T17:21:40Z

@wesm @xhochy @kszucs has artifacts for all platforms and pythons here if you're interested: https://github.com/kszucs/crossbow/releases/tag/build-115

cpcloud

This looks good to me. @kszucs Thanks for all of your work on this, I know it's been an uphill battle with continuous integration services.

This is really going to streamline the release process going forward.

@wesm @xhochy If you have any more comments, I'd like to merge this today or tomorrow.

cpcloud · 2018-06-20T17:30:31Z

@kszucs Do we have a JIRA for this?

We can determine whether all the expected artifacts from a particular run have been successfully uploaded (i.e. to GitHub)

cpcloud · 2018-06-20T18:41:02Z

dev/tasks/conda-recipes/appveyor.yml

-  - for %%f in (*.tar.bz2) do ren "%%f" "%%~nf-win-64.tar.bz2"
+  - for %%f in (*.tar.bz2) do (
+      set %%g=%%~nf
+      ren "%%f" "%%~ng-win-64.tar.bz2"


Nice. Batch is truly horrifying sometimes.

kszucs · 2018-06-20T19:01:52Z

@cpcloud Created JIRA for that https://issues.apache.org/jira/browse/ARROW-2724

cpcloud · 2018-06-22T20:47:40Z

@wesm I'm ready to merge if you are.

There are a couple of follow up tasks:

Setup nightlies on travis (which will trigger runs on both appveyor and travis). https://issues.apache.org/jira/browse/ARROW-1581 and https://issues.apache.org/jira/browse/ARROW-1582.
https://issues.apache.org/jira/browse/ARROW-2724: Determine if all artifacts are successfully uploaded.

wesm

+1 to merge and move on to the next stage of work. We need update dev/release/RELEASE_MANAGEMENT.md -- maybe when this is done you should advise the mailing list so some others can take this for a test run and see what issues they run into

…linux-packages

cpcloud · 2018-06-24T16:59:44Z

@pcmoritz @robertnishihara Have either of you encountered this error before: https://travis-ci.org/apache/arrow/jobs/395810684

I'm guessing that we happened to get a machine with very little memory on that particular run. Wondering if this is transient or an honest-to-goodness bug.

cpcloud · 2018-06-24T17:00:42Z

Arg, restarting the build appears to overwrite the previous build's log.

cpcloud · 2018-06-24T18:27:42Z

Ok, merging.

cpcloud · 2018-06-24T18:34:44Z

Thanks @kszucs! Onwards!

kszucs force-pushed the nightly branch 2 times, most recently from c3a68f1 to dd747c2 Compare June 10, 2018 14:46

wesm reviewed Jun 12, 2018

View reviewed changes

kszucs changed the title ~~ARROW-2674: [Packaging] Start building nightlies~~ ARROW-2676: [Packaging] Deploy build artifacts to github releases Jun 12, 2018

kszucs force-pushed the nightly branch from 708bbc4 to 0d9d89b Compare June 13, 2018 19:56

cpcloud requested changes Jun 18, 2018

View reviewed changes

cpcloud reviewed Jun 18, 2018

View reviewed changes

cpcloud self-requested a review June 18, 2018 20:00

kszucs force-pushed the nightly branch from 44ff0ff to 266265c Compare June 20, 2018 14:38

cpcloud approved these changes Jun 20, 2018

View reviewed changes

cpcloud reviewed Jun 20, 2018

View reviewed changes

wesm approved these changes Jun 23, 2018

View reviewed changes

kszucs added 4 commits June 23, 2018 14:21

don't create tree based on master

0ab0567

track remote branches

b65cb3f

trigger build task; explicit task names during submission

8a5fcb0

tigger builds template

a0a8127

kszucs and others added 23 commits June 23, 2018 14:21

print build id

fd6bd23

build prefix

14da2d5

little more status context

ccbc150

gry to remove osx wheel flag

167a393

remove flag

af6dd48

call rake version update

0773f3a

retrieve commit's combined status

95bc3fa

fix osx wheel builkds

bfb5eae

postfix conda pkgs with arch

71b6943

force update conda win assets

0561e62

update tasks

9b22238

Add email property and use target repo user.email by default

4c64db6

Code formatting cleanups

1b3d5fb

Validate github token

12efaba

Validate submit tasks

337234d

outdated readme section

f317a06

remove logging config

2e9fcf9

don't depend on any previous commit

e4e971c

explicit task names instead of placeholder in readme; foxme note for …

b892471

…linux-packages

conda win renames

062da7c

fixed conda-win deployments

0701d64

don't use dirty flag in version number

83f333b

batch correctly rename double extensions

87860d0

kszucs force-pushed the nightly branch from e7c7e38 to 87860d0 Compare June 23, 2018 12:21

cpcloud closed this in 58a2366 Jun 24, 2018

asfimport mentioned this pull request Jun 24, 2018

[Packaging] Deploy build artifacts to github releases #19068

Closed

2 tasks

		@@ -80,12 +80,12 @@ submission. The tasks are defined in `tasks.yml`
		6. Install the python dependencies for the script:



		class Target(object):
		class Repo(object):

ARROW-2676: [Packaging] Deploy build artifacts to github releases #2109

ARROW-2676: [Packaging] Deploy build artifacts to github releases #2109

Conversation

kszucs commented Jun 6, 2018 • edited by cpcloud Loading

codecov-io commented Jun 10, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

kszucs Jun 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Jun 13, 2018 • edited Loading

kszucs commented Jun 13, 2018

cpcloud commented Jun 14, 2018

cpcloud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Jun 18, 2018 • edited Loading

cpcloud commented Jun 18, 2018

cpcloud commented Jun 19, 2018

kszucs commented Jun 20, 2018

cpcloud commented Jun 20, 2018

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud commented Jun 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Jun 20, 2018

cpcloud commented Jun 22, 2018 • edited Loading

wesm left a comment

Choose a reason for hiding this comment

cpcloud commented Jun 24, 2018

cpcloud commented Jun 24, 2018

cpcloud commented Jun 24, 2018

cpcloud commented Jun 24, 2018

kszucs commented Jun 6, 2018 •

edited by cpcloud

Loading

codecov-io commented Jun 10, 2018 •

edited

Loading

kszucs Jun 12, 2018 •

edited

Loading

kszucs commented Jun 13, 2018 •

edited

Loading

kszucs commented Jun 18, 2018 •

edited

Loading

cpcloud commented Jun 20, 2018 •

edited

Loading

cpcloud commented Jun 22, 2018 •

edited

Loading