CI: try Github Actions as CI provider #688

breznak · 2019-09-24T12:35:43Z

GH Actions initial setup, building on all platforms

use GH Actions as CI for all platforms (OSX, Linux, WIn)
- keep other (Travis, Circle, AppV) CI around, runnig for some time. But make them not mandatory
ARM64 build in docker
Debug build in CI for c++ tests (and NOT building python in Debug)
- but this segfaults on a test -- known Yaml-cpp bug in debug mode #218, skipping the test for (Debug & Linux)
Releases to GH -- must be tested with a tag
publish to PYPI -- ditto, needs tag

Other:

parallel builds (for UNIXes) with -j4 in setup.py to speedup build times
setup.py provides configure target to build deps

Fixes #612
Fixes #19
Fixes #361

GH Actions initial setup, building on all platforms

Win does not handle #TODO at end of pip install -r requirements.txt #TODO

which is needed for OSX

as it's the common location and easier to find for new developers.

python setup.py test runs both c++ and python tests

on Linux

as in CI, other platforms run slow.

uses ubuntu-latest as platform

breznak · 2019-09-24T17:08:33Z

@dkeeney some more things we need to do for this PR:

get the docker ARM build running again (I plan to use the same 'hack' as for Debug, use a separate platform (say ubuntu-16.04) and run docker on that image)
the Debug build is not running correctly:
CMAKE_BUILD_TYPE=Debug python setup.py install does not export the Debug as I expected.
- build_type exists in setup.py, but is never changed, I can easily set it to Debug if the env.CMAKE_BUILD_TYPE exists, but I'm not sure if that would break the python/bindings build?

For next PR will be:

PYPI, should be easy
GH Releases

breznak

@dkeeney please review, when you have time. I'd like to merge to test the new Release & PYPI functionality.

Some overview from porting to Github Actions:
Some things are much easier now: 👍

publishing to PYPI
binary Releases to Github
multi-platform build using a matrix of all OSs (Linux, OSX, Win)

More problematic parts:

setting custom c++ compiler (eg gcc-8 for Ubuntu)
- this is due to each step being "blank new", not inheriting env from other steps (bcs steps can run in parallel)
passing env variables is not possible

Overall, GitHub Actions is quite great CI! I had the multiplatform build running within an hour. (then spent hours on debugging the Debug and gcc-8 issues). Highlights: multiplatform support, integration with services and GH (native).

Other changes:

python setup.py test now runs also the c++ tests (for convenience)
split requirements/extras dependencies

breznak · 2019-09-25T22:27:25Z

.github/workflows/htmcore.yml

+    - name: Install dependencies (gcc-8)
+      if: matrix.os == 'ubuntu-18.04'
+      env:
+        CC: gcc-8


setting custom gcc (gcc-8, so we use c++17 and not :boost) is quite challenging here

breznak · 2019-09-25T22:27:56Z

.github/workflows/htmcore.yml

+        python -m pip install --upgrade pip setuptools wheel
+
+    - name: build htmcore with setup.py
+      run: python setup.py install --user


multi-platform Release build is darn easy now!

breznak · 2019-09-25T22:28:31Z

.github/workflows/htmcore.yml

+        cmake --build . --config Release --target package # make package
+
+    - name: Release (deploy)
+      # from https://github.com/marketplace/actions/gh-release


GH Release is easy now

breznak · 2019-09-25T22:28:46Z

.github/workflows/htmcore.yml

+        ls dist
+
+    - name: Publish to PyPI
+      if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')


PYPI is easy

breznak · 2019-09-25T22:29:04Z

.github/workflows/htmcore.yml

+
+
+  build-arm64-docker:
+    name: Build for ARM64 on Docker


ARM64 build

.github/workflows/htmcore.yml

breznak · 2019-09-25T22:31:05Z

Dockerfile

-    python-dev \
-    python-numpy \
-    python-pip
+    python3-minimal \


updated our Dockerfile to python3

breznak · 2019-09-25T22:32:03Z

README.md

@@ -64,7 +64,7 @@ Fork or download the HTM-Community htm.core repository from https://github.com/h
 ```
 cd to-repository-root
 python -m pip install --user --upgrade pip setuptools setuptools-scm wheel
-python -m pip install --no-cache-dir --user -r bindings/py/packaging/requirements.txt
+python -m pip install --no-cache-dir --user -r requirements.txt


moved requirements here, TODO might need move back (for pypi dist/ build)

Yes, as I mentioned before, in order to get the requirements.txt file into the wheel it needs to somehow get into the Release/distr/dist/

fixed by copying /requirements.txt to build/Release/distr/dist during make install step

breznak · 2019-09-25T22:34:01Z

bindings/py/packaging/setup.py

@@ -340,7 +347,13 @@ def generateExtensions(platform, build_type):
        "htm.bindings": ["*.so", "*.pyd"],
        "htm.examples": ["*.csv"],
    },
-    extras_require = {},
+    #install extras by `pip install htm.core[examples]`
+    extras_require={'scikit-image>0.15.0':'examples',


split requirements.txt to hard-core dependencies (requirements.txt) , and optional (in setup.py extras_required)
This is due to Arm64 which failed on some of the new pip dependencies (scikit ?)

breznak · 2019-09-25T22:35:54Z

src/examples/hotgym/HelloSPTP.cpp

-#ifndef _MSC_VER
-        const size_t CI_avg_time = (size_t)floor(20*Timer::getSpeed()); //sec
+#ifdef NTA_OS_LINUX
+        const size_t CI_avg_time = (size_t)floor(99*Timer::getSpeed()); //sec //FIXME the CI speed broken for docker linux


TODO re-enable the time check? Time was agan different on the new CI

dkeeney

Great job. I am glad you were able to pull this together. 🥇

had problem with ../../requirements.txt on Windows, so now call it first (=from ./requirements.txt)

as it's quite time consuming, and needed only for release

on Windows

as ARM build takes too long (4hrs, due to docker emulation), make it build only nightly (not on each PR)

copy it there during install, needed for pypi

can call python setup.py configure which installs c++ deps and runs cmake

known issue in 3rd party dependency.

This reverts commit 17b9d5d.

because Debug & Linux is broken due to YAML parser bug: issue #218

uses hack by passing -- -j 4 to cmake's make, Since cmake 3.12 crossplatform -j4 is supported, but cmake 3.12 is quite new to require as a dependency (eg not in ubuntu 18.04 LTS)

This reverts commit 2b0abc5.

breznak

I will now merge, all CI are now passing green 💚

Next steps:

check PYPI
check GH Releases
make GH Actions CI mandatory and other CI only optional

breznak · 2019-09-26T10:27:48Z

.github/workflows/arm64-build.yml

+
+jobs:
+  build-arm64-docker:
+    name: Build for ARM64 on Docker


ARM build takes 4h, so separated to a standalone file and runs on schedure (nightly at 6am)

breznak · 2019-09-26T10:31:45Z

bindings/py/packaging/setup.py

+
+    # build: make && make install
+    if platform != "windows": #TODO since cmake 3.12 "-j4" is directly supported (=crossplatform), for now -- passes other options to make
+      subprocess.check_call(["cmake", "--build", ".", "--target", "install", "--config", build_type, "--", "-j", "4"]) 


we now call cmake/make with implicit -j4, this really speeds up builds using python setup.py for users and developers.

on CI the build time is 2x faster (down to 6-10min, from 20+)

crossplatform approach since cmake 3.12, but I don't find it reasonable to bump dependency, it's too new yet.

Windows slows down the green PRs now, I tried building with /MT instead of /MD (which I found is equivalent of using make -j, but then unittests complain to be compiled with /MD, so reverted for now. FYI @dkeeney

CI: try Github Actions for CI

7ed3e8e

GH Actions initial setup, building on all platforms

breznak added tooling build platform labels Sep 24, 2019

breznak self-assigned this Sep 24, 2019

GH Actions: CC not set as env variable

6e9433e

breznak force-pushed the ci_gh_actions branch from 395b314 to 6e9433e Compare September 24, 2019 12:50

GH Actions: fix pip requirements for Windows

385966d

Win does not handle #TODO at end of pip install -r requirements.txt #TODO

breznak force-pushed the ci_gh_actions branch from 7886ca3 to 21da78c Compare September 24, 2019 13:08

GH Actions: export ARCH_FLAGS

cbe73a4

which is needed for OSX

breznak force-pushed the ci_gh_actions branch from 21da78c to cbe73a4 Compare September 24, 2019 13:12

breznak added 2 commits September 24, 2019 15:23

setup.py: set ARCHFLAGS for OSX as required

c100cfe

setup.py: move requirements to top-level dir

46514b2

as it's the common location and easier to find for new developers.

breznak force-pushed the ci_gh_actions branch from 6b60eb4 to 46514b2 Compare September 24, 2019 13:35

breznak added 2 commits September 24, 2019 16:00

GH Actions: fix running c++ tests

1e1ee2a

setup.py: python runs also the c++ tests

9851bd9

python setup.py test runs both c++ and python tests

breznak mentioned this pull request Sep 24, 2019

Publish bindings on pypi #19

Closed

4 tasks

breznak added 4 commits September 24, 2019 17:08

Merge branch 'master_community' into ci_gh_actions

cb93666

Hotgym: bump CI time for GH Actions

b311854

GH Actions: run valgrind memcheck

839dff8

on Linux

Hotgym: check for time only on Linux

f3ca3d9

as in CI, other platforms run slow.

breznak force-pushed the ci_gh_actions branch from 4e54cbb to 346cbdd Compare September 24, 2019 15:54

fix syntax for if for CI valgrind

7c8b594

breznak force-pushed the ci_gh_actions branch 4 times, most recently from e2cd0a1 to 94e482a Compare September 24, 2019 16:35

CI Actions: run one Debug build

4cf5087

uses ubuntu-latest as platform

breznak force-pushed the ci_gh_actions branch from 94e482a to 4cf5087 Compare September 24, 2019 16:49

breznak added the ready label Sep 25, 2019

breznak requested a review from dkeeney September 25, 2019 22:25

breznak commented Sep 25, 2019

View reviewed changes

dkeeney previously approved these changes Sep 26, 2019

View reviewed changes

breznak added 7 commits September 26, 2019 07:11

Merge branch 'master_community' into ci_gh_actions

dd9a64a

GH Actions: fix version tag filter

2159d9d

GH Actions: configure installs requirements.txt

47b5af9

had problem with ../../requirements.txt on Windows, so now call it first (=from ./requirements.txt)

GH Actions: gate "make package" to run on tagged only

3f7a8bc

as it's quite time consuming, and needed only for release

GH Actions: fix comment interpreted during pre-pypi

9b33317

on Windows

GH Actions: separate ARM64 build into scheduled build

d09a87d

as ARM build takes too long (4hrs, due to docker emulation), make it build only nightly (not on each PR)

requirements.txt included in dist/

4c28afe

copy it there during install, needed for pypi

breznak dismissed dkeeney’s stale review via 4c28afe September 26, 2019 06:05

breznak force-pushed the ci_gh_actions branch from d9292f9 to 85b4e23 Compare September 26, 2019 06:23

debug Windows

5a8f08e

breznak force-pushed the ci_gh_actions branch from 85b4e23 to 5a8f08e Compare September 26, 2019 06:30

breznak added 2 commits September 26, 2019 09:36

setup.py: add configure command

5803bd5

can call python setup.py configure which installs c++ deps and runs cmake

fix broken YAML test in Debug on Linux

17b9d5d

known issue in 3rd party dependency.

breznak requested a review from dkeeney September 26, 2019 07:54

breznak added 5 commits September 26, 2019 10:03

GH Actions: Debug build cleanup

5513580

Revert "fix broken YAML test in Debug on Linux"

341b09a

This reverts commit 17b9d5d.

GH Actions: run Debug build on OSX

86250e7

because Debug & Linux is broken due to YAML parser bug: issue #218

fix debug build

39058ea

setup.py: parallel builds with -j4

7166ffd

uses hack by passing -- -j 4 to cmake's make, Since cmake 3.12 crossplatform -j4 is supported, but cmake 3.12 is quite new to require as a dependency (eg not in ubuntu 18.04 LTS)

breznak force-pushed the ci_gh_actions branch from dd9d062 to 7166ffd Compare September 26, 2019 09:39

breznak added 2 commits September 26, 2019 12:10

MSVC: build with /MT instead of /MD for faster builds

2b0abc5

Revert "MSVC: build with /MT instead of /MD for faster builds"

257267f

This reverts commit 2b0abc5.

breznak force-pushed the ci_gh_actions branch from 8ec3db0 to 257267f Compare September 26, 2019 10:26

breznak commented Sep 26, 2019

View reviewed changes

breznak merged commit 5048692 into master Sep 26, 2019

breznak deleted the ci_gh_actions branch September 26, 2019 10:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: try Github Actions as CI provider #688

CI: try Github Actions as CI provider #688

breznak commented Sep 24, 2019 •

edited

Loading

breznak commented Sep 24, 2019 •

edited

Loading

breznak left a comment

breznak Sep 25, 2019

breznak Sep 25, 2019

breznak Sep 25, 2019

breznak Sep 25, 2019

breznak Sep 25, 2019

breznak Sep 25, 2019

breznak Sep 25, 2019

dkeeney Sep 26, 2019

breznak Sep 26, 2019 •

edited

Loading

breznak Sep 25, 2019

breznak Sep 25, 2019

dkeeney left a comment

breznak left a comment

breznak Sep 26, 2019

breznak Sep 26, 2019

CI: try Github Actions as CI provider #688

CI: try Github Actions as CI provider #688

Conversation

breznak commented Sep 24, 2019 • edited Loading

breznak commented Sep 24, 2019 • edited Loading

breznak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breznak Sep 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkeeney left a comment

Choose a reason for hiding this comment

breznak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breznak commented Sep 24, 2019 •

edited

Loading

breznak commented Sep 24, 2019 •

edited

Loading

breznak Sep 26, 2019 •

edited

Loading