Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: try Github Actions as CI provider #688

Merged
merged 69 commits into from
Sep 26, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
7ed3e8e
CI: try Github Actions for CI
breznak Sep 24, 2019
6e9433e
GH Actions: CC not set as env variable
breznak Sep 24, 2019
385966d
GH Actions: fix pip requirements for Windows
breznak Sep 24, 2019
cbe73a4
GH Actions: export ARCH_FLAGS
breznak Sep 24, 2019
c100cfe
setup.py: set ARCHFLAGS for OSX as required
breznak Sep 24, 2019
46514b2
setup.py: move requirements to top-level dir
breznak Sep 24, 2019
1e1ee2a
GH Actions: fix running c++ tests
breznak Sep 24, 2019
9851bd9
setup.py: python runs also the c++ tests
breznak Sep 24, 2019
cb93666
Merge branch 'master_community' into ci_gh_actions
breznak Sep 24, 2019
b311854
Hotgym: bump CI time for GH Actions
breznak Sep 24, 2019
839dff8
GH Actions: run valgrind memcheck
breznak Sep 24, 2019
f3ca3d9
Hotgym: check for time only on Linux
breznak Sep 24, 2019
7c8b594
fix syntax for if for CI valgrind
breznak Sep 24, 2019
4cf5087
CI Actions: run one Debug build
breznak Sep 24, 2019
ae3ec97
fix valgrind commandline
breznak Sep 24, 2019
0a87372
Docker: update to python3
breznak Sep 25, 2019
7ae9903
GH Actions: docker ARM64 build WIP
breznak Sep 25, 2019
750db07
Actions: fix order
breznak Sep 25, 2019
7102d56
Revert "Actions: fix order"
breznak Sep 25, 2019
2fbd707
Docker: try sudo
breznak Sep 25, 2019
63f01c6
GH Actions: try get docker running
breznak Sep 25, 2019
6879513
try docker on ubuntu 16.04
breznak Sep 25, 2019
f717aaa
GH Actions: ARM as a separate job
breznak Sep 25, 2019
0a2536d
setup.py: honor CMAKE_BUILD_TYPE
breznak Sep 25, 2019
d663d19
typo
breznak Sep 25, 2019
530343c
setup: dont fail if var not set
breznak Sep 25, 2019
e725466
GH Actions: set gcc-8 for ubuntu 18.04
breznak Sep 25, 2019
d19f0c8
setup.py: split requirements to install, extras
breznak Sep 25, 2019
9aa6877
fixing syntax
breznak Sep 25, 2019
97c8481
GH Actions: syntax,
breznak Sep 25, 2019
95c100a
GH Actions: set links for gcc to gcc-8
breznak Sep 25, 2019
440b713
fix Debug build, cannot use setup.py
breznak Sep 25, 2019
ccd9687
GH Actions: fix Debug build, see if gcc-8 is fixed
breznak Sep 25, 2019
43628c4
fix update-alternatives
breznak Sep 25, 2019
6b1ad36
requirements: prettytable required for tests if monitor
breznak Sep 25, 2019
b00e537
fix configure gcc cc
breznak Sep 25, 2019
a934eda
update alt
breznak Sep 25, 2019
12448e0
gcc-8 try 999
breznak Sep 25, 2019
08c8cb9
Revert "gcc-8 try 999"
breznak Sep 25, 2019
fde3bb8
try w/o requirements
breznak Sep 25, 2019
cb06516
tale of master and slave
breznak Sep 25, 2019
9170cd8
make more slaves
breznak Sep 25, 2019
5e7a574
setup.py: honor CC, CXX env variables
breznak Sep 25, 2019
6e6f29e
Revert "setup.py: honor CC, CXX env variables"
breznak Sep 25, 2019
4cb136b
whitespace
breznak Sep 25, 2019
f541f95
gcc-8 config
breznak Sep 25, 2019
b95e9be
use GITHUB_WORKSPACE
breznak Sep 25, 2019
d03e2ab
debug
breznak Sep 25, 2019
68092ac
fix calling cpp tests from python setup
breznak Sep 25, 2019
18f099f
add GH Release code
breznak Sep 25, 2019
149e964
GH Actions: add PYPI publishing
breznak Sep 25, 2019
d40c636
GH Actions: replace make with multiplatform cmake
breznak Sep 25, 2019
dd9a64a
Merge branch 'master_community' into ci_gh_actions
breznak Sep 26, 2019
2159d9d
GH Actions: fix version tag filter
breznak Sep 26, 2019
47b5af9
GH Actions: configure installs requirements.txt
breznak Sep 26, 2019
3f7a8bc
GH Actions: gate "make package" to run on tagged only
breznak Sep 26, 2019
9b33317
GH Actions: fix comment interpreted during pre-pypi
breznak Sep 26, 2019
d09a87d
GH Actions: separate ARM64 build into scheduled build
breznak Sep 26, 2019
4c28afe
requirements.txt included in dist/
breznak Sep 26, 2019
5a8f08e
debug Windows
breznak Sep 26, 2019
5803bd5
setup.py: add configure command
breznak Sep 26, 2019
17b9d5d
fix broken YAML test in Debug on Linux
breznak Sep 26, 2019
5513580
GH Actions: Debug build cleanup
breznak Sep 26, 2019
341b09a
Revert "fix broken YAML test in Debug on Linux"
breznak Sep 26, 2019
86250e7
GH Actions: run Debug build on OSX
breznak Sep 26, 2019
39058ea
fix debug build
breznak Sep 26, 2019
7166ffd
setup.py: parallel builds with -j4
breznak Sep 26, 2019
2b0abc5
MSVC: build with /MT instead of /MD for faster builds
breznak Sep 26, 2019
257267f
Revert "MSVC: build with /MT instead of /MD for faster builds"
breznak Sep 26, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,6 @@ jobs:
name: Installing python dependencies
command: |
python -m pip install --user --upgrade pip setuptools setuptools-scm wheel
python -m pip install --no-cache-dir --user -r bindings/py/packaging/requirements.txt --verbose || exit


# Build Release with python
- run:
Expand Down
32 changes: 32 additions & 0 deletions .github/workflows/arm64-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: ARM64 build

on:
# run every day of the week at 06:00
schedule:
- cron: 0 6 * * *

jobs:
build-arm64-docker:
name: Build for ARM64 on Docker
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARM build takes 4h, so separated to a standalone file and runs on schedure (nightly at 6am)

runs-on: ubuntu-18.04
steps:
- name: Install docker
run: |
# bug in ubuntu, conflicts with docker.io
sudo apt-get update
sudo apt-get remove --purge -y moby-engine moby-cli
sudo apt-get install -y qemu-system docker.io

- name: Setup ARM64 build env
run: |
uname -a
sudo docker run --rm --privileged multiarch/qemu-user-static:register --reset
uname -a

- uses: actions/checkout@v1

- name: ARM64 build
run: |
sudo docker build -t htm-arm64-docker --build-arg arch=arm64 .
uname -a

127 changes: 127 additions & 0 deletions .github/workflows/htmcore.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
name: htm.core build

on:
push:
branches:
- 'v*.*.*' # vX.Y.Z release tag
# run on pull_request events that target the master branch
pull_request:
branches:
- master
# run every day of the week at 02:00
schedule:
- cron: 0 2 * * *

jobs:
build:
name: Building on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
#max-parallel: 4
matrix:
python-version: [3.7]
os: [ubuntu-18.04, windows-2019, macOS-latest]

steps:
- uses: actions/checkout@v1

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}

- name: Versions
run: |
python --version
cmake --version
c++ --version

- name: Install dependencies (gcc-8)
if: matrix.os == 'ubuntu-18.04'
env:
CC: gcc-8
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setting custom gcc (gcc-8, so we use c++17 and not :boost) is quite challenging here

CXX: g++-8
run: |
sudo apt-get update
sudo apt-get -y install gcc-8 g++-8
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r requirements.txt
python setup.py configure

- name: Install dependencies
if: matrix.os != 'ubuntu-18.04'
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r requirements.txt
python setup.py configure

- name: build htmcore with setup.py
run: python setup.py install --user
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multi-platform Release build is darn easy now!


- name: C++ & Python Tests
run: python setup.py test

- name: Memory leaks check (valgrind)
if: matrix.os == 'ubuntu-18.04'
run: |
sudo apt-get -y install valgrind
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${PWD}/build/Release/lib valgrind --show-leak-kinds=definite,indirect,possible,reachable --track-origins=yes --num-callers=40 --error-exitcode=3 ./build/Release/bin/benchmark_hotgym 5 || exit 1

- name: Release (make package)
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
run: |
python setup.py bdist_wheel
cd build/scripts
cmake --build . --config Release --target install # aka make install ,but multiplatform
cmake --build . --config Release --target package # make package

- name: Release (deploy)
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
# from https://github.com/marketplace/actions/gh-release
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GH Release is easy now

uses: softprops/action-gh-release@v1
with:
files: |
build/scripts/htm_core-v*
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: pre-PyPI
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
#copy dist data to /dist, where PyPI Action expects it
run: |
cp -a build/Release/distr/dist .
ls dist

- name: Publish to PyPI
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PYPI is easy

uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.pypi_password }}
repository_url: https://test.pypi.org/legacy/ #TODO rm for real pypi



build-debug:
name: Build and test in Debug mode
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug build for c++ tests

#currently cannot run on Linux & Debug due to a bug in YAML parser: issue #218
runs-on: macOS-latest
steps:
- uses: actions/checkout@v1

- name: Install dependencies (Debug)
run: |
echo "built type: ${CMAKE_BUILD_TYPE}"
mkdir -p build/scripts
cd build/scripts
cmake ../.. -DCMAKE_BUILD_TYPE=Debug

- name: Debug build
run: |
cd build/scripts
make -j2 && make install

- name: C++ Tests
run: |
cd build/scripts
../Debug/bin/unit_tests
breznak marked this conversation as resolved.
Show resolved Hide resolved
9 changes: 2 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,8 @@ install:


script:
# Some tests (e.g., helloregion) expect this to be the current directory and
# this also matches current instructions in htm.core/README.md
# unit tests
- "cd $TRAVIS_BUILD_DIR/build/Release/bin"
- "${TRAVIS_BUILD_DIR}/build/Release/bin/unit_tests --gtest_output=xml:${TRAVIS_BUILD_DIR}/build/artifacts/unit_tests_report.xml"
# run python bindings tests
- echo "Python bindings tests"
# run c++ & python bindings tests
- echo "C++ & Python tests"
- cd ${TRAVIS_BUILD_DIR}
- "python setup.py test"
# run memory leak tests with valgrind
Expand Down
31 changes: 17 additions & 14 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,39 @@ ARG arch=amd64

# Multiarch Debian 10 Buster (amd64, arm64, etc).
# https://hub.docker.com/r/multiarch/debian-debootstrap
FROM multiarch/debian-debootstrap:$arch-buster
FROM multiarch/debian-debootstrap:${arch}-buster

RUN apt-get update
RUN apt-get install -y \
RUN apt-get install -y --no-install-suggests \
cmake \
g++ \
g++-8 \
git-core \
libyaml-dev \
python \
python-dev \
python-numpy \
python-pip
python3-minimal \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated our Dockerfile to python3

python3-dev \
python3-numpy \
python3-pip \
python3-venv

ADD . /usr/local/src/htm.core
WORKDIR /usr/local/src/htm.core

# Setup py env
#RUN python3 -m venv pyenv && . pyenv/bin/activate
RUN pip3 install --upgrade setuptools pip wheel
#RUN export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.7/dist-packages

# Install
RUN pip install --upgrade setuptools
RUN pip install wheel
RUN pip install \
RUN pip3 install \
# Explicitly specify --cache-dir, --build, and --no-clean so that build
# artifacts may be extracted from the container later. Final built python
# packages can be found in /usr/local/src/htm.core/bindings/py/dist
# --cache-dir /usr/local/src/htm.core/pip-cache \
# --build /usr/local/src/htm.core/pip-build \
# --no-clean \
-r bindings/py/packaging/requirements.txt
RUN python setup.py install
-r requirements.txt
RUN python3 setup.py install --force

# Test
RUN ./build/Release/bin/unit_tests
RUN python setup.py test
RUN python3 setup.py test

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Fork or download the HTM-Community htm.core repository from https://github.com/h
```
cd to-repository-root
python -m pip install --user --upgrade pip setuptools setuptools-scm wheel
python -m pip install --no-cache-dir --user -r bindings/py/packaging/requirements.txt
python -m pip install --no-cache-dir --user -r requirements.txt
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved requirements here, TODO might need move back (for pypi dist/ build)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as I mentioned before, in order to get the requirements.txt file into the wheel it needs to somehow get into the Release/distr/dist/

Copy link
Member Author

@breznak breznak Sep 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed by copying /requirements.txt to build/Release/distr/dist during make install step

```

Be sure you are running the right version of python. Check it with the following command:
Expand Down Expand Up @@ -275,7 +275,7 @@ distribution packages as listed and rename them as indicated. Copy these to
### There are two sets of Unit Tests:

* C++ Unit tests -- to run: `./build/Release/bin/unit_tests`
* Python Unit tests -- to run: `python setup.py test`
* Python Unit tests -- to run: `python setup.py test` (runs also the C++ tests above)

# Examples

Expand Down
11 changes: 2 additions & 9 deletions appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,6 @@ build_script:
- echo Binding version = %BINDINGS_VERSION%

- python -m pip install --user --upgrade pip setuptools setuptools-scm wheel || exit
- python -m pip install --no-cache-dir --user -r bindings/py/packaging/requirements.txt || exit

# we need to find the previous git tags so a shallow clone is not enough
- git fetch --depth=200
Expand All @@ -110,13 +109,7 @@ build_script:


after_build:
# Run unit_tests (C++)
- cd %HTM_CORE%
- mkdir %ARTIFACTS_DIR%
- cd %HTM_CORE%\build\Release\bin
- unit_tests.exe --gtest_output=xml:%ARTIFACTS_DIR%\unit_tests_report.xml

# Run python tests
# Run C++ & python tests
- cd %HTM_CORE%
- python setup.py test

Expand Down Expand Up @@ -155,4 +148,4 @@ on_success:
}




Loading