ci-cygwin.yml: Multi-stage build #29403

mkoeppe · 2020-03-25T01:55:07Z

The GitHub workflow for cygwin does not fit into the 6-hour time limit.
We do a multi-stage build, saving and loading local/ to/from build artifacts.

Depends on #29417
Depends on #29392

CC: @dimpase @embray @slel @jpflori @tscrim @vbraun @kcrisman

Component: porting: Cygwin

Author: Matthias Koeppe

Branch/Commit: 27ddea3

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29403

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-03-25T02:09:32Z

Branch: u/mkoeppe/ci_cygwin_yml__multi_stage_build

sagetrac-git · 2020-03-25T03:13:09Z

Commit: 79bee2f

sagetrac-git · 2020-03-25T03:13:09Z

Branch pushed to git repo; I updated commit sha1. New commits:

`79bee2f`	`ci-cygwin.yml: Remove sage-spkg -y -o`

sagetrac-git · 2020-03-26T19:50:32Z

Changed commit from 79bee2f to cef0d73

sagetrac-git · 2020-03-26T19:50:32Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`cd0a0ec`	`handle symlinks`
`5a387d0`	`Use winsymlinks:native`
`122f972`	`Full targets`
`424d86e`	`prepare the final logs artifact`
`f646274`	`store local/var/tmp/sage/build in logs artifact`
`6924328`	`stage 1`
`97d3f86`	`touch installation records to get same mtime`
`5939bc7`	`Upload log artifact first`
`59d20e0`	`stage 4`
`cef0d73`	`more stages`

mkoeppe · 2020-03-26T19:52:29Z

comment:4

Sample run https://github.com/mkoeppe/sage/actions/runs/63708828

mkoeppe · 2020-03-26T20:07:25Z

Dependencies: #29397

mkoeppe · 2020-03-29T03:03:39Z

comment:6

Something is still triggering rebuilds of targets that have been built in a previous stage.

dimpase · 2020-03-29T03:14:59Z

comment:7

I'd suspect it's the toolchain crud.

mkoeppe · 2020-03-29T04:29:02Z

Changed dependencies from #29397 to none

mkoeppe · 2020-03-29T04:29:35Z

comment:9

I'm running one test with MAKE="make --debug=verbose" to see what's happening there

sagetrac-git · 2020-03-30T22:38:17Z

Changed commit from cef0d73 to de50cf8

sagetrac-git · 2020-03-30T22:38:17Z

Branch pushed to git repo; I updated commit sha1. New commits:

`de50cf8`	`better touch`

mkoeppe · 2020-03-30T22:48:32Z

Author: Matthias Koeppe

mkoeppe · 2020-03-31T16:03:26Z

comment:13

This works pretty well now.

sagetrac-git · 2020-04-04T14:53:38Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9db2311`	`.github/workflows/ci-cygwin.yml: Edit targets`
`9f38852`	`cygwin: Add minimal and standard-python2 builds`

sagetrac-git · 2020-04-04T14:53:38Z

Changed commit from de50cf8 to 9f38852

mkoeppe · 2020-04-05T02:43:38Z

comment:15

New failure mode: https://github.com/mkoeppe/sage/runs/560884908

"/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 775, in __init__
  [backports_abc-0.5]       restore_signals, start_new_session)
  [backports_abc-0.5]     File "/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 1453, in _execute_child
  [backports_abc-0.5]       restore_signals, start_new_session, preexec_fn)
  [backports_abc-0.5]   BlockingIOError: [Errno 11] Resource temporarily unavailable
  [backports_abc-0.5]   Cleaning up...

mkoeppe · 2020-04-05T02:45:28Z

comment:16

https://github.com/mkoeppe/sage/runs/560884917

Installing backports_abc-0.5
      0 [main] python3.7 16471 child_info_fork::abort: address space needed by 'cyglzma-5.dll' (0x400000) is already occupied
Installing package backports_abc using pip3
      0 [main] python3.7 16554 child_info_fork::abort: address space needed by 'cyglzma-5.dll' (0x400000) is already occupied
Ignoring indexes: https://pypi.org/simple

mkoeppe · 2020-04-05T02:47:31Z

comment:17

https://github.com/mkoeppe/sage/runs/560884917

 [r-3.6.2.p0]   gcc -I../../src/extra -I../../src/extra/xdr -I. -I../../src/include -I../../src/include -I/cygdrive/d/a/sage/sage/local/include  -I../../src/nmath -DHAVE_CONFIG_H   -fopenmp   -g -O2  -fPIC  -c version.c -o version.o
  [r-3.6.2.p0]   sysutils.c:554:10: fatal error: iconv.h: No such file or directory
  [r-3.6.2.p0]     554 | #include <iconv.h>
  [r-3.6.2.p0]         |          ^~~~~~~~~
  [r-3.6.2.p0]   compilation terminated.

dimpase · 2020-04-05T02:53:02Z

comment:18

Replying to @mkoeppe:

New failure mode: https://github.com/mkoeppe/sage/runs/560884908

"/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 775, in __init__
  [backports_abc-0.5]       restore_signals, start_new_session)
  [backports_abc-0.5]     File "/cygdrive/d/a/sage/sage/local/lib/python3.7/subprocess.py", line 1453, in _execute_child
  [backports_abc-0.5]       restore_signals, start_new_session, preexec_fn)
  [backports_abc-0.5]   BlockingIOError: [Errno 11] Resource temporarily unavailable
  [backports_abc-0.5]   Cleaning up...

welcome to Cygwin rebase Hell.
One might need to call one of these src/bin/sage-rebase* (sacrificing a black chicken might be a pre-req).

sagetrac-git · 2020-04-08T14:41:14Z

Branch pushed to git repo; I updated commit sha1. New commits:

`62f4aa5`	`build/pkgs/cygwin.txt: Add libiconv-devel`
`0a53429`	`.github/workflows/ci-cygwin.yml: Call rebaseall after extracting SAGE_LOCAL archive`

sagetrac-git · 2020-04-08T14:41:14Z

Changed commit from 9f38852 to 0a53429

mkoeppe · 2020-04-09T16:37:48Z

comment:21

In https://github.com/mkoeppe/sage/runs/572730280 (cygwin-stage-ii-a (minimal)), despite rebasing in the "Extract sage-local artifact" step, I am still getting _fork::abort: address space needed by 'cyglzma-5.dll' (0x400000) is already occupied when building packages

mkoeppe · 2020-04-09T16:51:26Z

comment:23

#29486 cygwin-standard: rpy2 fails to build

mkoeppe · 2020-04-09T16:55:56Z

comment:24

cygwin-stage-ii-b (minimal) times out - https://github.com/mkoeppe/sage/runs/572730298 - may need to rebalance stages

dimpase · 2020-04-17T07:27:03Z

comment:37

with these multi-stage builds, where *.so get built at one place, and moved then somewhere esle, it could be that more rebasing is needed. In particular, I recall seeing and fixing rebase locations for Python modules being built.
(Basically it was the issue that totally broke building Sage on Cygwin32, simply due to insufficient address space).

mkoeppe · 2020-04-17T07:30:11Z

comment:38

Testing at https://github.com/mkoeppe/sage/actions/runs/80516745

mkoeppe · 2020-04-17T07:31:10Z

comment:39

Yes, I'm rebasing after unpacking local from the previous stages.

mkoeppe · 2020-04-17T21:19:22Z

comment:40

I think it's working now, and we can see the real build errors again in cygwin-standard: matplotlib, rpy2, scipy (likely BLAS-related) https://github.com/mkoeppe/sage/runs/596536457

sagetrac-git · 2020-04-17T23:45:24Z

Changed commit from ffdc97e to fc61002

sagetrac-git · 2020-04-17T23:45:24Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fc61002`	`ci-cygwin.yml: Package up SAGE_LOCAL/var/tmp/sage/build to avoid error with actions/upload-artifact`

dimpase · 2020-04-18T04:14:09Z

comment:43

testing on https://github.com/dimpase/sage/actions/runs/81294070

mkoeppe · 2020-04-18T04:35:37Z

comment:44

I cannot reproduce the matplotlib build error locally - even when I download and unpack stage-i-a and stage-i-b (standard) and configure against that.

sagetrac-git · 2020-04-18T05:37:05Z

Branch pushed to git repo; I updated commit sha1. New commits:

`59ec460`	`./configure --enable-experimental-packages --enable-download-from-upstream-url`
`c9cf3cc`	`Merge branch 't/29392/__configure___enable_experimental_packages___enable_download_from_upstream_url' into t/29403/ci_cygwin_yml__multi_stage_build`
`037f508`	`tox.yml: Pass CONFIGURE_ARGS: --enable-experimental-packages --enable-download-from-upstream-url`

sagetrac-git · 2020-04-18T05:37:05Z

Changed commit from fc61002 to 037f508

mkoeppe · 2020-04-18T05:37:23Z

Changed dependencies from #29417 to #29417, #29392

sagetrac-git · 2020-04-18T05:40:35Z

Changed commit from 037f508 to 998f02d

sagetrac-git · 2020-04-18T05:40:35Z

Branch pushed to git repo; I updated commit sha1. New commits:

`998f02d`	`ci-cygwin.yml: Fixup for: Package up SAGE_LOCAL/var/tmp/sage/build`

sagetrac-git · 2020-04-18T07:08:46Z

Changed commit from 998f02d to d6bf4b1

sagetrac-git · 2020-04-18T07:08:46Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d6bf4b1`	`ci-cygwin.yml: Install standard system packages also for 'stndard-python2'`

dimpase · 2020-04-18T09:37:09Z

comment:49

how does one need to modify ~/.github/ to make GH Actions only run cygwin tests?

mkoeppe · 2020-04-18T15:57:49Z

comment:50

Replying to @dimpase:

how does one need to modify ~/.github/ to make GH Actions only run cygwin tests?

Just delete the file tox.yml

sagetrac-git · 2020-04-18T16:36:16Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2779e24`	`ci-cygwin.yml: Another fixup for: Package up SAGE_LOCAL/var/tmp/sage/build`

sagetrac-git · 2020-04-18T16:36:16Z

Changed commit from d6bf4b1 to 2779e24

sagetrac-git · 2020-04-19T03:30:26Z

Changed commit from 2779e24 to 27ddea3

sagetrac-git · 2020-04-19T03:30:26Z

Branch pushed to git repo; I updated commit sha1. New commits:

`27ddea3`	`build/bin/sage-logger: Back out an experimental change`

dimpase · 2020-04-19T23:47:28Z

Reviewer: Dima Pasechnik

mkoeppe · 2020-04-20T00:30:39Z

comment:54

Thanks!

vbraun · 2020-04-23T22:32:47Z

Changed branch from u/mkoeppe/ci_cygwin_yml__multi_stage_build to 27ddea3

mkoeppe added this to the sage-9.1 milestone Mar 25, 2020

mkoeppe added c: porting labels Mar 25, 2020

mkoeppe added c: porting: cygwin and removed c: porting labels Apr 17, 2020

dimpase added s: positive review and removed s: needs review labels Apr 19, 2020

vbraun removed the s: positive review label Apr 23, 2020

vbraun closed this as completed in 5cba647 Apr 23, 2020

ci-cygwin.yml: Multi-stage build #29403

ci-cygwin.yml: Multi-stage build #29403

Comments

mkoeppe commented Mar 25, 2020

mkoeppe commented Mar 25, 2020

sagetrac-git mannequin commented Mar 25, 2020

sagetrac-git mannequin commented Mar 25, 2020

sagetrac-git mannequin commented Mar 26, 2020

sagetrac-git mannequin commented Mar 26, 2020

mkoeppe commented Mar 26, 2020

mkoeppe commented Mar 26, 2020

mkoeppe commented Mar 29, 2020

dimpase commented Mar 29, 2020

mkoeppe commented Mar 29, 2020

mkoeppe commented Mar 29, 2020

sagetrac-git mannequin commented Mar 30, 2020

sagetrac-git mannequin commented Mar 30, 2020

mkoeppe commented Mar 30, 2020

mkoeppe commented Mar 31, 2020

sagetrac-git mannequin commented Apr 4, 2020

sagetrac-git mannequin commented Apr 4, 2020

mkoeppe commented Apr 5, 2020

mkoeppe commented Apr 5, 2020

mkoeppe commented Apr 5, 2020

dimpase commented Apr 5, 2020

sagetrac-git mannequin commented Apr 8, 2020

sagetrac-git mannequin commented Apr 8, 2020

mkoeppe commented Apr 9, 2020

mkoeppe commented Apr 9, 2020

mkoeppe commented Apr 9, 2020

dimpase commented Apr 17, 2020

mkoeppe commented Apr 17, 2020

mkoeppe commented Apr 17, 2020

mkoeppe commented Apr 17, 2020

sagetrac-git mannequin commented Apr 17, 2020

sagetrac-git mannequin commented Apr 17, 2020

dimpase commented Apr 18, 2020

mkoeppe commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

mkoeppe commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

dimpase commented Apr 18, 2020

mkoeppe commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

sagetrac-git mannequin commented Apr 18, 2020

sagetrac-git mannequin commented Apr 19, 2020

sagetrac-git mannequin commented Apr 19, 2020

dimpase commented Apr 19, 2020

mkoeppe commented Apr 20, 2020

vbraun commented Apr 23, 2020