Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Run coverage tests in parallel #399

Merged
merged 8 commits into from
Aug 27, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
chore: Replace poetry with uv
  • Loading branch information
aborgna-q committed Aug 23, 2024
commit 27ea976dbf46255f0469436ea96a3dd06fee21fd
68 changes: 48 additions & 20 deletions .github/workflows/pull-request.yaml
Original file line number Diff line number Diff line change
@@ -14,6 +14,9 @@ on:
env:
SCCACHE_GHA_ENABLED: "true"
RUSTC_WRAPPER: "sccache"
# A constant location for the uv cache
UV_CACHE_DIR: /tmp/.uv-cache


jobs:
check:
@@ -30,37 +33,50 @@ jobs:
uses: mozilla-actions/sccache-action@v0.0.3
- name: install-llvm
run: "sudo apt-get update && sudo apt-get install -y llvm-14"
- name: Install poetry
run: pipx install poetry
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3

- name: Set up uv
# Install a specific uv version using the installer
run: curl -LsSf https://astral.sh/uv/0.3.2/install.sh | sh
- name: "Set up Python"
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: "poetry"
- name: Restore uv cache
uses: actions/cache@v4
with:
path: /tmp/.uv-cache
key: uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
restore-keys: |
uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
uv-${{ runner.os }}

- name: Install Guppy
run: poetry install
run: uv sync

- name: Type check with mypy
run: poetry run mypy guppylang
run: uv run mypy guppylang

- name: Check formatting with ruff
run: poetry run ruff format --check guppylang
run: uv run ruff format --check guppylang

- name: Lint with ruff
run: poetry run ruff check guppylang
run: uv run ruff check guppylang

- name: Install Guppy with validation and llvm-based execution
run: poetry install --with execution,validation
run: uv sync --extra execution --extra validation

- name: Run tests
run: poetry run pytest
run: uv run pytest

- name: Install tket2 dependencies
run: poetry install --with pytket
# This removes the `execution` and `validation` extra groups
run: uv sync --extra pytket

- name: Rerun `py(...)` expression tests and pytket lowering with tket2 installed
run: poetry run pytest tests/integration/test_py.py tests/error/test_py_errors.py tests/integration/test_tket.py
run: uv run pytest tests/integration/test_py.py tests/error/test_py_errors.py tests/integration/test_tket.py

- name: Minimize uv cache
run: uv cache prune --ci

coverage:
if: github.event_name != 'merge_group'
@@ -70,23 +86,35 @@ jobs:
- uses: actions/checkout@v4
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.3
- name: Install poetry
run: pipx install poetry
- name: Set up Python 3.10
uses: actions/setup-python@v3

- name: Set up uv
# Install a specific uv version using the installer
run: curl -LsSf https://astral.sh/uv/0.3.2/install.sh | sh
- name: "Set up Python"
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: "poetry"
- name: Restore uv cache
uses: actions/cache@v4
with:
path: /tmp/.uv-cache
key: uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
restore-keys: |
uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
uv-${{ runner.os }}

- name: Install Guppy with validation, execution and pytket
run: poetry install --with execution,pytket,validation
run: uv sync --extra execution --extra pytket --extra validation

- name: Run python tests with coverage instrumentation
run: poetry run pytest --cov=./ --cov-report=xml
run: uv run pytest --cov=./ --cov-report=xml

- name: Upload python coverage to codecov.io
uses: codecov/codecov-action@v3
with:
files: coverage.xml
name: python
token: ${{ secrets.CODECOV_TOKEN }}

- name: Minimize uv cache
run: uv cache prune --ci
35 changes: 18 additions & 17 deletions .github/workflows/python-wheels.yml
Original file line number Diff line number Diff line change
@@ -17,6 +17,10 @@ on:
release:
types:
- published
# Temporary trigger to test the workflow
pull_request:
branches:
- main

jobs:
build-publish:
@@ -27,16 +31,17 @@ jobs:
- uses: actions/checkout@v4
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.4
- name: Install poetry
run: pipx install poetry
- name: Set up Python '3.10'

- name: Set up uv
# Install a specific uv version using the installer
run: curl -LsSf https://astral.sh/uv/0.3.2/install.sh | sh
- name: "Set up Python"
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: "poetry"

- name: Build sdist and wheels
run: poetry build -o dist
run: uvx --from build pyproject-build --installer uv --outdir dist

- name: Upload the built packages as artifacts
uses: actions/upload-artifact@v4
@@ -46,24 +51,20 @@ jobs:
dist/*.tar.gz
dist/*.whl

- name: Publish to test instance of PyPI (dry-run)
if: |
${{ github.event_name == 'push' && github.ref_type == 'branch' }} ||
${{ github.event_name == 'workflow_dispatch' && github.ref_type == 'branch'}}
- name: Test installing the built wheels
run: |
echo "Doing a dry-run publish to test-pypi..."
echo "Based on the following workflow variables, this is not a hugr-py version tag push:"
echo " - event_name: ${{ github.event_name }}"
echo " - ref_type: ${{ github.ref_type }}"
echo " - ref: ${{ github.ref }}"
poetry config repositories.test-pypi https://test.pypi.org/legacy/
poetry config pypi-token.test-pypi ${{ secrets.PYPI_TEST_PUBLISH }}
poetry publish -r test-pypi --dist-dir dist --skip-existing --dry-run
echo "Testing the newly built wheels..."
uv run --with guppylang --refresh-package guppylang --no-project -- python -c "import guppylang"

- name: Publish to PyPI
if: |
${{ (github.event_name == 'release' && github.ref_type == 'tag' && startsWith(github.ref, 'refs/tags/v'))}} ||
${{ (github.event_name == 'workflow_dispatch' && github.ref_type == 'tag' && startsWith(github.ref, 'refs/tags/v'))}} ||
run: |
echo "Publishing to PyPI..."
echo "Based on the following workflow variables, this is a new version tag push:"
echo " - event_name: ${{ github.event_name }}"
echo " - ref_type: ${{ github.ref_type }}"
echo " - ref: ${{ github.ref }}"
poetry config pypi-token.pypi ${{ secrets.PYPI_PUBLISH }}
poetry publish --dist-dir dist --skip-existing
14 changes: 7 additions & 7 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -26,31 +26,31 @@ repos:

- repo: local
hooks:
- id: poetry-install
name: sync poetry deps
- id: uv-install
name: sync uv deps
description: Ensure development tools are installed.
entry: poetry install --sync
entry: uv sync
language: system
files: poetry.lock
files: "uv.lock|pyproject.toml"
pass_filenames: false
- id: ruff-format
name: ruff format
description: Format python code with `ruff`.
entry: poetry run ruff format guppylang
entry: uv run ruff format guppylang
language: system
files: \.py$
pass_filenames: false
- id: ruff-check
name: ruff
description: Check python code with `ruff`.
entry: poetry run ruff check --fix --exit-non-zero-on-fix guppylang
entry: uv run ruff check --fix --exit-non-zero-on-fix guppylang
language: system
files: \.py$
pass_filenames: false
- id: mypy-check
name: mypy
description: Check python code with `mypy`.
entry: poetry run mypy guppylang
entry: uv run mypy guppylang
language: system
files: \.py$
pass_filenames: false
34 changes: 15 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -59,40 +59,33 @@ These instructions will get you a copy of the project up and running on your loc

### Prerequisites

- Python >= 3.10
- [Poetry](https://python-poetry.org/docs/#installation)
- [uv](https://docs.astral.sh/uv/getting-started/installation/)
- [Rust](https://www.rust-lang.org/tools/install) >= 1.75.0 (only needed for tests)

### Installing

Run the following to setup your virtual environment and install dependencies:

```sh
poetry install --with execution,validation
uv sync --extra execution --extra validation
```

Note that the `--with execution,validation` flag is optional and only needed to run integration tests.
Note that the `--extra` flags are optional and only needed to run integration tests.

The `validation` flag allows the tests to validate that the hugrs guppy outputs are well formed, and the `execution` flag allows tests to compile these hugrs to native code using [hugr-llvm](https://github.com/CQCL/hugr-llvm) to check the results are as expected.
The `validation` extra allows the tests to validate that the hugrs guppy outputs are well formed, and the `execution` extra allows tests to compile these hugrs to native code using [hugr-llvm](https://github.com/CQCL/hugr-llvm) to check the results are as expected.
This requires `llvm-14` as described in the `hugr-llvm` repo.

You can then activate the virtual environment and work within it with:

```sh
poetry shell
```

Consider using [direnv](https://github.com/direnv/direnv/wiki/Python#poetry) to
Consider using [direnv](https://direnv.net/docs/installation.html) to
automate this when entering and leaving a directory.

To run a single command in the shell, just prefix it with `poetry run`.
To run a single command in the shell, just prefix it with `uv run`.

### Pre-commit

Install the pre-commit hook by running:

```sh
poetry run pre-commit install
uv run pre-commit install
```


@@ -101,21 +94,24 @@ poetry run pre-commit install
Run tests using

```sh
poetry run pytest -v
uv run pytest -v
```

You have to install extra dependencies to test automatic circuit conversion from `pytket`:
You have to install extra dependencies to test automatic circuit conversion from `pytket`.

```sh
poetry install --with pytket
poetry run pytest -v # Now rerun tests
# Install extra dependencies
# Using `--inexact` to avoid removing the already installed extras.
uv sync --extra pytket --inexact
# Now rerun tests
uv run pytest -v
```


Integration test cases can be exported to a directory using

```sh
poetry run pytest --export-test-cases=guppy-exports
uv run pytest --export-test-cases=guppy-exports
```

which will create a directory `./guppy-exports` populated with hugr modules serialised in JSON.
14 changes: 7 additions & 7 deletions devenv.nix
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@
# for building optional tket2 dependency
# see https://github.com/CQCL/tket2/blob/main/devenv.nix
packages = [
pkgs.just
pkgs.llvmPackages_14.libllvm
pkgs.libffi
pkgs.libxml2
@@ -18,16 +19,15 @@
]
);

enterShell = ''
just setup-extras
source .venv/bin/activate
'';

languages.python = {
enable = true;
poetry = {
uv = {
enable = true;
install = {
enable = true;
groups = ["validation" "execution" "pytket"];
};

activate.enable = true;
};
};

2 changes: 1 addition & 1 deletion docs/build.sh
Original file line number Diff line number Diff line change
@@ -4,4 +4,4 @@ mkdir build

touch build/.nojekyll # Disable jekyll to keep files starting with underscores

poetry run sphinx-build -b html ./api-docs ./build/api-docs
uv run --extra docs sphinx-build -b html ./api-docs ./build/api-docs
13 changes: 8 additions & 5 deletions execute_llvm/pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
[build-system]
requires = ["maturin>=1.3,<2.0"]
build-backend = "maturin"

[project]
name = "execute-llvm"
requires-python = ">=3.8"
requires-python = ">=3.10,<3.13"
classifiers = [
"Programming Language :: Rust",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
]
dynamic = ["version"]

[build-system]
requires = ["maturin>=1.3,<2.0"]
build-backend = "maturin"

[tool.maturin]
features = ["pyo3/extension-module"]

[tool.uv]
no-build = true
Loading
Loading