Skip to content

Commit

Permalink
🧑‍💻 Extensive and Reusable CI/CD Setup (#396)
Browse files Browse the repository at this point in the history
## Description

This PR tries to establish some new best practices for CI/CD within the
MQT.

First of all, this adds a Cirrus CI config for building Apple Silicon
and Linux ARM wheels as well as perform Python testing on Apple Silicon.

Most importantly, a series of smaller reusable workflows have been
created. These include workflows for different types of tests (Python,
C++, Windows, MacOS), Python packaging, and linting. The purpose of this
change is to improve the modularity and readability of the workflows,
and allow for better reuse across different workflows and projects. In
the best case, these workflows can directly be reused in our top-level
projects without having to write them from scratch again.

Furthermore, code coverage reports are now split between Python and C++
test workflows and will be automatically uploaded to Codecov using the
flags feature. Carry-forward support is enable for scenarios where not
all reports are uploaded.

Lastly, CI and CD workflows were split for better readability and
maintainability. Wheels are now only built on releases.
The respective CI runs are only started if some files relevant to that
check have been modified. A single pass-check eases branch protection.

## Checklist:

<!---
This checklist serves as a reminder of a couple of things that ensure
your pull request will be merged swiftly.
-->

- [x] The pull request only contains commits that are related to it.
- [x] I have added appropriate tests and documentation.
- [x] I have made sure that all CI jobs on GitHub pass.
- [x] The pull request introduces no new warnings and follows the
project's style guidelines.
  • Loading branch information
burgholzer authored Aug 14, 2023
2 parents bc2d7d4 + 6dbee15 commit 47ac7ec
Show file tree
Hide file tree
Showing 31 changed files with 999 additions and 600 deletions.
115 changes: 115 additions & 0 deletions .cirrus.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
.clone_script: &clone |
if [ -z "$CIRRUS_PR" ]; then
git clone --recursive --branch=$CIRRUS_BRANCH https://x-access-token:${CIRRUS_REPO_CLONE_TOKEN}@github.com/${CIRRUS_REPO_FULL_NAME}.git $CIRRUS_WORKING_DIR
git reset --hard $CIRRUS_CHANGE_IN_REPO
else
git clone https://x-access-token:${CIRRUS_REPO_CLONE_TOKEN}@github.com/${CIRRUS_REPO_FULL_NAME}.git $CIRRUS_WORKING_DIR
git fetch origin pull/$CIRRUS_PR/head:pull/$CIRRUS_PR
git checkout $CIRRUS_BASE_BRANCH
git -c user.email="[email protected]" merge --no-commit pull/$CIRRUS_PR
git submodule update --init --recursive
fi

.statistics_script: &statistics |
ccache -s -v
echo $(python -m pip cache dir)
echo $(python -m pip cache list)

macos_arm64_test_task:
name: 🐍 Test / 🍎 arm64
alias: macos_arm64_test
clone_script: *clone
macos_instance:
image: ghcr.io/cirruslabs/macos-monterey-xcode:latest
ccache_cache:
folder: .ccache
populate_script:
- mkdir -p .ccache
fingerprint_key: ccache-macosx_arm64
pip_cache:
folder: /Users/admin/Library/Caches/pip
prepare_env_script: |
brew install [email protected] ccache pipx ninja nox
ln -s $(which python3.10) python
export PATH=/opt/homebrew/opt/[email protected]/libexec/bin:$PATH
export PATH=/opt/homebrew/opt/ccache/libexec:$PATH
export PATH=/opt/homebrew/opt/pipx/libexec:$PATH
export PATH=/opt/homebrew/opt/ninja/libexec:$PATH
export PATH=/opt/homebrew/opt/nox/libexec:$PATH
echo "PATH=$PATH" >> $CIRRUS_ENV
echo "CCACHE_DIR=$PWD/.ccache" >> $CIRRUS_ENV
run_nox_script:
- nox -s tests-3.10 --verbose
statistics_script: *statistics

build_and_store_wheels: &BUILD_AND_STORE_WHEELS
install_cibuildwheel_script:
- python -m pip install cibuildwheel~=2.15.0
run_cibuildwheel_script:
- cibuildwheel
wheels_artifacts:
path: "wheelhouse/*"

linux_aarch64_wheels_task:
only_if: "$CIRRUS_RELEASE != ''"
name: 🐍 Packaging / 🎡 🐧 arm64
alias: linux_aarch64_wheels
clone_script: *clone
compute_engine_instance:
image_project: cirrus-images
image: family/docker-builder-arm64
architecture: arm64
platform: linux
cpu: 8
env:
CIBW_SKIP: "*-musllinux_*"
setup_pyhton_script:
- apt-get install -y python3-venv python-is-python3
<<: *BUILD_AND_STORE_WHEELS

macos_arm64_wheels_task:
only_if: "$CIRRUS_RELEASE != ''"
name: 🐍 Packaging / 🎡 🍎 arm64
alias: macos_arm64_wheels
clone_script: *clone
macos_instance:
image: ghcr.io/cirruslabs/macos-monterey-xcode:latest
ccache_cache:
folder: .ccache
populate_script:
- mkdir -p .ccache
fingerprint_key: ccache-macosx_arm64-wheels
pip_cache:
folder: /Users/admin/Library/Caches/pip
prepare_env_script: |
brew install [email protected] ccache pipx ninja nox
ln -s $(which python3.10) python
export PATH=/opt/homebrew/opt/[email protected]/libexec/bin:$PATH
export PATH=/opt/homebrew/opt/ccache/libexec:$PATH
export PATH=/opt/homebrew/opt/pipx/libexec:$PATH
export PATH=/opt/homebrew/opt/ninja/libexec:$PATH
export PATH=/opt/homebrew/opt/nox/libexec:$PATH
echo "PATH=$PATH" >> $CIRRUS_ENV
echo "CCACHE_DIR=$PWD/.ccache" >> $CIRRUS_ENV
<<: *BUILD_AND_STORE_WHEELS
statistics_script: *statistics

publish_task:
name: 🚀 Deploy to PyPI
container: { image: "python:3.10-bullseye" }
depends_on:
- linux_aarch64_wheels
- macos_arm64_wheels
only_if: "$CIRRUS_RELEASE != ''"
env:
TWINE_REPOSITORY: pypi
TWINE_USERNAME: __token__
TWINE_PASSWORD: "ENCRYPTED\
[776960f3f91eccd9d5036b8dd2ef02dd3d23caf52942443e\
d9aabf8fe4e7597406fd388e8b280a6c766703cc0e7b92fb]"
install_script: pip install twine
publish_script:
- curl -L https://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/wheels.zip -o wheels.zip
- unzip wheels.zip
- python -m twine check wheelhouse/*
- python -m twine upload wheelhouse/*
41 changes: 34 additions & 7 deletions .github/codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,46 @@ coverage:
range: 60..90
precision: 1
status:
project:
default:
project: off
patch: off

flag_management:
default_rules:
carryforward: true
statuses:
- type: project
target: auto
threshold: 0.5%
patch:
default:
removed_code_behavior: adjust_base
- type: patch
target: 90%
threshold: 1%
individual_flags:
- name: cpp
paths:
- "include"
- "src"
- "!src/mqt/**/*"
after_n_builds: 1
- name: python
paths:
- "src/mqt/**/*.py"
after_n_builds: 2
statuses:
- type: project
threshold: 0.5%
removed_code_behavior: adjust_base
- type: patch
target: 95%
threshold: 1%

parsers:
gcov:
branch_detection:
conditional: no
loop: no

codecov:
notify:
after_n_builds: 3
comment:
layout: "reach, diff, flags, files"
require_changes: true
show_carryforward_flags: true
72 changes: 72 additions & 0 deletions .github/contributing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Contributing

Thank you for your interest in contributing to this project.
We value contributions from people with all levels of experience.
In particular if this is your first pull request not everything has to be perfect.
We will guide you through the process.

We use GitHub to [host code](https://github.com/cda-tum/mqt-core), to [track issues and feature requests](https://github.com/cda-tum/mqt-core/issues), as well as accept [pull requests](https://github.com/cda-tum/mqt-core/pulls).
See <https://docs.github.com/en/get-started/quickstart> for a general introduction to working with GitHub and contributing to projects.

## Types of Contributions

You can contribute in several ways:

- 🐛 Report Bugs
: Report bugs at <https://github.com/cda-tum/mqt-core/issues> using the _🐛 Bug report_ issue template. Please make sure to fill out all relevant information in the respective issue form.

- 🐛 Fix Bugs
: Look through the [GitHub Issues](https://github.com/cda-tum/mqt-core/issues) for bugs. Anything tagged with "bug" is open to whoever wants to try and fix it.

- ✨ Propose New Features
: Propose new features at <https://github.com/cda-tum/mqt-core/issues> using the _✨ Feature request_ issue template. Please make sure to fill out all relevant information in the respective issue form.

- ✨ Implement New Features
: Look through the [GitHub Issues](https://github.com/cda-tum/mqt-core/issues) for features. Anything tagged with "feature" or "enhancement" is open to whoever wants to implement it. We highly appreciate external contributions to the project.

- 📝 Write Documentation
: MQT Core could always use some more documentation, and we appreciate any help with that.

## Get Started 🎉

Ready to contribute? Check out the {doc}`Development Guide <DevelopmentGuide>` to set up MQT Core for local development and learn about the style guidelines and conventions used throughout the project.

We value contributions from people with all levels of experience.
In particular if this is your first PR not everything has to be perfect.
We will guide you through the PR process.
Nevertheless, please try to follow the guidelines below as well as you can to help make the PR process quick and smooth.

## Core Guidelines

- ["Commit early and push often"](https://www.worklytics.co/blog/commit-early-push-often).
- Write meaningful commit messages (preferably using [gitmoji](https://gitmoji.dev) to give additional context to your commits).
- Focus on a single feature/bug at a time and only touch relevant files. Split multiple features into multiple contributions.
- If you added a new feature, you should add tests that ensure it works as intended. Furthermore, the new feature should be documented appropriately.
- If you fixed a bug, you should add tests that demonstrate that the bug has been fixed.
- Document your code thoroughly and write readable code.
- Keep your code clean. Remove any debug statements, left-over comments, or code unrelated to your contribution.
- Run `pre-commit run -a` to check your code for style and linting errors before committing.

## Pull Request Workflow

- Create PRs early. It is ok to create work-in-progress PRs. You may mark these as draft PRs on GitHub.
- Describe your PR. Start with a descriptive title, reference any related issues by including the issue number in the PR description, and add a comprehensive description of the changes. We provide a PR template that you can (and should) follow to create a PR. Do not delete any sections from the template.
- Whenever a PR is created or updated, several workflows on all supported platforms and versions of Python are executed. These workflows ensure that the project still builds, that all tests pass, and that the code is properly formatted and introduces no new linting errors. Your PR is expected to pass all these continuous integration (CI) checks before it can be merged. Here are some tips for finding the cause of certain failures:
- If any of the `CI / 🇨‌ Test` checks fail, this most likely indicates build errors or test failures in the C++ part of the code base. Look through the respective logs on GitHub for any error or failure messages.
- If any of the `CI / 🐍 Test` checks fail, this most likely indicates build errors or test failures in the Python part of the code base. Look through the respective logs on GitHub for any error or failure messages.
- If any of the `codecov/\*` checks fail, this means that your changes are not appropriately covered by tests or that the overall project coverage decreased too much. Ensure that you include tests for all your changes in the PR.
- If `cpp-linter` comments on your PR with a list of warnings, these have been raised by `clang-tidy` when checking the C++ part of your changes for warnings or style guideline violations. The individual messages frequently provide helpful suggestions on how to fix the warnings. If you don't see any messages, but the `🇨‌ Lint / 🚨 Lint` check is red, click on the `Details` link to see the full log of the check and a step summary.
- If the `pre-commit.ci` check fails, some of the `pre-commit` checks failed and could not be fixed automatically by the _pre-commit.ci_ bot. Such failures are most likely related to the Python part of the code base. The individual log messages frequently provide helpful suggestions on how to fix the warnings.
- If the `docs/readthedocs.org:mqt-core` check fails, the documentation could not be built properly. Inspect the corresponding log file for any errors.
- Once your PR is ready, change it from a draft PR to a regular PR and request a review from one of the project maintainers. Please make sure to only request a review once you are done with your changes and the PR is ready to be merged. If you are unsure whether your PR is ready for review, please ask in the PR comments.
- If your PR gets a "Changes requested" review, you will need to address the feedback and update your PR by pushing to the same branch. You don't need to close the PR and open a new one. Respond to review comments on the PR (e.g., with "done 👍" or "done in @\<commit\>") to let the reviewer know that you have addressed the feedback. Note that reviewers do not get a notification if you just react to the review comment with an emoji. You need to write a comment to notify the reviewer.
- Be sure to re-request review once you have made changes after a code review so that maintainers know that the requests have been addressed.

---

This document was inspired by and partially adapted from

- <https://matplotlib.org/stable/devel/coding_guide.html>
- <https://opensource.creativecommons.org/contributing-code/pr-guidelines/>
- <https://yeoman.io/contributing/pull-request.html>
- <https://github.com/scikit-build/scikit-build>
78 changes: 0 additions & 78 deletions .github/contributing.rst

This file was deleted.

11 changes: 11 additions & 0 deletions .github/support.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Support

If you are stuck with a problem using MQT Core or have questions, please get in touch at our [Issues](https://github.com/cda-tum/mqt-core/issues) or [Discussions](https://github.com/cda-tum/mqt-core/discussions). We'd love to help.

You can save time by following this procedure when reporting a problem:

- Do try to solve the problem on your own first.
- Search through past [Issues](https://github.com/cda-tum/mqt-core/issues) and [Discussions](https://github.com/cda-tum/mqt-core/discussions) to see if someone else already had the same problem.
- Before filing a bug report, try to create a minimal working example (MWE) that reproduces the problem. It is much easier to identify the cause for the problem if a handful of lines suffice to show that something is not working.

You can also always reach us at [[email protected]](mailto:[email protected]).
12 changes: 0 additions & 12 deletions .github/support.rst

This file was deleted.

12 changes: 12 additions & 0 deletions .github/workflows/cd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: CD
on:
release:
types: [published]
workflow_dispatch:

jobs:
python-packaging:
name: 🐍 Packaging
uses: ./.github/workflows/reusable-python-packaging.yml
with:
package-name: mqt.core
Loading

0 comments on commit 47ac7ec

Please sign in to comment.