Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Add GHA workflow for minimal CI jobs #750

Merged
merged 43 commits into from
Jan 19, 2023
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
7fd4646
add GHA workflow for minimal CI jobs
atolopko-czi Jan 19, 2023
4058755
run "full" ci workflow only on main merges and releases
atolopko-czi Jan 19, 2023
65c6a5e
fix
atolopko-czi Jan 19, 2023
33065c3
temporarily disable r ci GHA workflow
atolopko-czi Jan 19, 2023
1668f79
add "full" to workflow name
atolopko-czi Jan 19, 2023
d330b5e
trigger python minimal ci workflow on python changes only
atolopko-czi Jan 19, 2023
07d82a5
tmp file to trigger GHA Python CI minimal workflow for testing
atolopko-czi Jan 19, 2023
84f9692
fix/testing
atolopko-czi Jan 19, 2023
ee3a2bd
fix/testing
atolopko-czi Jan 19, 2023
8d42d86
fix/testing
atolopko-czi Jan 19, 2023
c6dfb58
fix/testing
atolopko-czi Jan 19, 2023
8163d59
fix/testing
atolopko-czi Jan 19, 2023
cbb06e6
fix/testing
atolopko-czi Jan 19, 2023
122cad0
fix/testing
atolopko-czi Jan 19, 2023
6dbbf69
cleanup
atolopko-czi Jan 19, 2023
ee50906
test python ci full workflow
atolopko-czi Jan 19, 2023
e8146dc
test python ci full workflow
atolopko-czi Jan 19, 2023
face8d5
use reusable workflow for python-ci-{full,minimal}
atolopko-czi Jan 19, 2023
bdd6587
fix/testing
atolopko-czi Jan 19, 2023
5de96e7
fix/testing
atolopko-czi Jan 19, 2023
518c343
trigger
atolopko-czi Jan 19, 2023
c5fdc55
workflow name
atolopko-czi Jan 19, 2023
690a1a3
cleanup
atolopko-czi Jan 19, 2023
59ced62
test python ci full
atolopko-czi Jan 19, 2023
2406ef9
fix
atolopko-czi Jan 19, 2023
1f14178
cleanup, docs
atolopko-czi Jan 19, 2023
b5280bd
Merge branch 'main' into atol/693-ci-optimize-resource-usage
atolopko-czi Jan 19, 2023
b7357ff
always call reusable workflow from same branch
atolopko-czi Jan 19, 2023
c8b01b6
revert debug code
atolopko-czi Jan 19, 2023
0315f99
revert to original python GHA job triggers
atolopko-czi Jan 19, 2023
fa485c5
Merge branch 'main' into atol/693-ci-optimize-resource-usage
atolopko-czi Jan 19, 2023
3063097
rm redundant mypy check
atolopko-czi Jan 19, 2023
307a254
rm todo
atolopko-czi Jan 19, 2023
cc8ac5a
test pull_request trigger again
atolopko-czi Jan 19, 2023
4156a4b
test pull_request trigger again
atolopko-czi Jan 19, 2023
616b6a3
give up on pull_request trigger for now; use push
atolopko-czi Jan 19, 2023
20b4648
Merge branch 'main' into atol/693-ci-optimize-resource-usage
atolopko-czi Jan 19, 2023
94a4369
caching fix attempt
atolopko-czi Jan 19, 2023
bbadfcd
caching fix attempt
atolopko-czi Jan 19, 2023
34d41c9
trigger
atolopko-czi Jan 19, 2023
91e7023
caching fix attempt
atolopko-czi Jan 19, 2023
1b4286e
caching fix attempt
atolopko-czi Jan 19, 2023
5e2689d
rm testing file
atolopko-czi Jan 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/python-ci-full.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: TileDB-SOMA Python CI (Full)

on:
push:
branches: [main]
release:
types: [published]

jobs:
build:
strategy:
fail-fast: false
matrix:
# TODO: restore Windows build once we have C++/libtiledbsoma integration supported there
os: [ubuntu-22.04, macos-12]
# os: [ubuntu-22.04, macos-12, windows-2019]
python-version: ['3.7', '3.8', '3.9', '3.10']
include:
- runs-on: ubuntu-22.04
cc: gcc-11
cxx: g++-11
- runs-on: macos-12
cc: gcc-11
cxx: g++-11
uses: ./.github/workflows/python-ci-single.yml
with:
os: ${{ matrix.os }}
python_version: ${{ matrix.python-version }}
cc: ${{ matrix.cc }}
cxx: ${{ matrix.cxx }}
is_mac: ${{ contains(matrix.os, 'macos') }}
report_codecov: ${{ matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.9' }}
run_lint: ${{ matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.9' }}
secrets: inherit


30 changes: 30 additions & 0 deletions .github/workflows/python-ci-minimal.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: TileDB-SOMA Python CI (Minimal)

on:
push:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now differs from r-ci.yml, which is not ideal. And the GHA console shows push- and pull_request-triggered workflow runs somewhat differently. I'm going to test once more with pull_request...

paths-ignore:
- 'apis/r/**'
branches-ignore:
- main

jobs:
build:
strategy:
fail-fast: true
matrix:
os: [ubuntu-22.04]
python-version: ['3.10', '3.7']
include:
- runs-on: ubuntu-22.04
cc: gcc-11
cxx: g++-11
# TODO: remove version before merge to main
uses: ./.github/workflows/python-ci-single.yml
with:
os: ${{ matrix.os }}
python_version: ${{ matrix.python-version }}
cc: ${{ matrix.cc }}
cxx: ${{ matrix.cxx }}
report_codecov: ${{ matrix.python-version == '3.10' }}
run_lint: ${{ matrix.python-version == '3.10' }}
secrets: inherit
Original file line number Diff line number Diff line change
@@ -1,68 +1,97 @@
# This is a reusable workflow
# (https://docs.github.com/en/actions/using-workflows/reusing-workflows#calling-a-reusable-workflow),
# called by python-ci-full.yml and python-ci-minimal.yml.

name: TileDB-SOMA Python CI

on:
pull_request:
paths-ignore:
- 'apis/r/**'
push:
branches: [main]
release:
types: [published]

workflow_call:
inputs:
python_version:
required: true
type: string
os:
required: true
type: string
cc:
required: true
type: string
cxx:
required: true
type: string
report_codecov:
required: true
type: boolean
is_mac:
required: false
type: boolean
default: false
run_lint:
required: false
type: boolean
default: false

jobs:
build:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# TODO: restore Windows build once we have C++/libtiledbsoma integration supported there
os: [ubuntu-22.04, macos-12]
# os: [ubuntu-22.04, macos-12, windows-2019]
python-version: ['3.7', '3.8', '3.9', '3.10']
include:
- runs-on: ubuntu-22.04
cc: gcc-11
cxx: g++-11
- runs-on: macos-12
cc: gcc-11
cxx: g++-11
lint:
if: ${{ inputs.run_lint }}
runs-on: ${{ inputs.os }}
steps:
- name: Checkout TileDB-SOMA
uses: actions/checkout@v3
with:
fetch-depth: 0 # ensure we get all tags to inform package version determination

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ inputs.python_version }}

- name: Install testing/linting prereqs
run: python -m pip -v install -U pip typeguard pre-commit mypy types-setuptools

# needed for mypy type checking
- name: Install tiledbsoma
run: python -m pip -v install -e apis/python
env:
CC: ${{ inputs.cc }}
CXX: ${{ inputs.cxx }}

- name: Run pre-commit hooks on all files
run: pre-commit run -a -v

- name: Check C++ Format
run: ./scripts/run-clang-format.sh . clang-format 0 $(find libtiledbsoma -name "*.cc" -or -name "*.h")

build:
runs-on: ${{ inputs.os }}
steps:
- name: Select XCode version
if: inputs.is_mac
uses: maxim-lobanov/setup-xcode@v1
with:
# Pending https://github.com/actions/runner-images/issues/6350
xcode-version: '13.4'
if: ${{ runner.os == 'macOS' }}

- name: Checkout TileDB-SOMA
uses: actions/checkout@v3
with:
fetch-depth: 0 # ensure we get all tags to inform package version determination

- name: Set up Python ${{ matrix.python-version }}
- name: Set up Python ${{ inputs.python_version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
python-version: ${{ inputs.python_version }}
cache: pip
cache-dependency-path: ./apis/python/setup.py

- name: Install testing prereqs
run: python -m pip -v install -U pip pytest-cov typeguard

- name: Run pre-commit hooks on all files
if: matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.9'
run: python -m pip -v install pre-commit && pre-commit run -a -v

- name: Check C++ Format
if: matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.9'
run: ./scripts/run-clang-format.sh . clang-format 0 $(find libtiledbsoma -name "*.cc" -or -name "*.h")
run: python -m pip -v install -U pip pytest-cov typeguard types-setuptools

- name: Install tiledbsoma
run: python -m pip -v install -e apis/python
env:
CC: ${{ matrix.cc }}
CXX: ${{ matrix.cxx }}
CC: ${{ inputs.cc }}
CXX: ${{ inputs.cxx }}

- name: Show package versions
run: python scripts/show-versions.py
Expand All @@ -86,7 +115,7 @@ jobs:
run: ctest --test-dir build/libtiledbsoma -C Release --verbose

- name: Report coverage to Codecov
if: matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.9'
if: inputs.report_codecov
uses: codecov/codecov-action@v3
with:
flags: python
Expand Down