Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update conda package publish for muliple python versions #1805

Merged
merged 13 commits into from
May 4, 2023
Merged

Update conda package publish for muliple python versions #1805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
127af59
Remove upper-bound on cupy version for conda package
oliverholworthy Apr 26, 2023
eada6b5
Update conda package publish to build for multiple python versions
oliverholworthy Apr 26, 2023
aa3e91c
Use find command to get CONDA_PACKAGE output
oliverholworthy Apr 26, 2023
20dd168
Remove Python 3.11 conda build (rapids does not yet support this)
oliverholworthy Apr 26, 2023
1927d43
Use strategy matrix for python version builds
oliverholworthy Apr 28, 2023
90eca06
quote python versions
oliverholworthy Apr 28, 2023
7ed3e31
Set concurrency to cancel builds
oliverholworthy Apr 28, 2023
16c8dc8
Use mambabuild to speed up conda build time
oliverholworthy Apr 28, 2023
355dc78
Add setup with conda-incubator
oliverholworthy Apr 28, 2023
b3036ee
remove pull_request workflow condition and add release branch push
oliverholworthy Apr 28, 2023
25d1b3d
Add Python version matrix for conda package publish
oliverholworthy Apr 28, 2023
e7052f7
Merge branch 'main' into conda-package-python-versions
oliverholworthy May 3, 2023
4ef4dfa
Merge branch 'main' into conda-package-python-versions
karlhigley May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 27 additions & 14 deletions .github/workflows/cpu-packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,23 @@ name: Packages
on:
workflow_dispatch:
push:
branches: [main]
branches:
- main
- release*
tags:
- "v[0-9]+.[0-9]+.[0-9]+"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build-conda:
name: Build Conda
runs-on: ${{ matrix.os }}
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we could add 3.9 and 3.10 here so the builds run in separate actions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read some possibly old docs that you could pass in multiple values for the python parameter, which seems like it doesn't work. Updated to the matrix version again and checking if this works now

os: [ubuntu-latest]

python-version: ["3.8", "3.9", "3.10"]
steps:
- uses: actions/checkout@v3
with:
Expand All @@ -30,22 +34,28 @@ jobs:
sudo apt-get install -y protobuf-compiler
- name: Install and upgrade python packages
run: |
python -m pip install --upgrade pip setuptools==59.4.0 wheel tox pybind11
python -m pip install --upgrade pip setuptools==59.4.0 wheel pybind11
python -m pip uninstall protobuf -y
python -m pip install --no-binary=protobuf protobuf
- uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
- name: Generate package for conda
id: conda_build
run: |
echo "conda pkgs dir $CONDA_PKGS_DIRS"
conda update conda
conda install conda-build pybind11
conda build --python ${{ matrix.python-version }} . -c defaults -c conda-forge -c numba -c rapidsai -c nvidia --output-folder ./conda_packages
export CONDA_PACKAGE=$(conda build --python ${{ matrix.python-version }} . -c defaults -c conda-forge -c numba -c rapidsai -c nvidia --output-folder ./conda_packages --output)
echo "conda_package : $CONDA_PACKAGE"
echo "conda_package=$CONDA_PACKAGE" >> $GITHUB_OUTPUT
conda install -c conda-forge mamba
mamba install -c conda-forge conda-build boa conda-verify pybind11
conda mambabuild --python ${{ matrix.python-version }} . -c defaults -c conda-forge -c numba -c rapidsai -c nvidia --output-folder ./conda_packages
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using mambabuild instead of regular conda build cut the build time down from 3-5 hours to about 10 minutes.

conda_package=$(find ./conda_packages/ -name "*.tar.bz2")
export CONDA_PACKAGE=$conda_package
echo "conda_package : $conda_package"
echo "conda_package=$conda_package" >> $GITHUB_OUTPUT
- name: Upload conda artifacts to github
uses: actions/upload-artifact@v3
with:
name: conda
name: conda-${{ matrix.python-version }}
path: ${{ steps.conda_build.outputs.conda_package }}

build-pypi:
Expand Down Expand Up @@ -115,15 +125,18 @@ jobs:
release-conda:
name: Release Conda
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10"]
if: "startsWith(github.ref, 'refs/tags/')"
needs: [build-conda]
steps:
- uses: actions/setup-python@v4
with:
python-version: 3.9
python-version: ${{ matrix.python-version }}
- uses: actions/download-artifact@v2
with:
name: conda
name: conda-${{ matrix.python-version }}
path: conda
- uses: conda-incubator/setup-miniconda@v2
with:
Expand Down
2 changes: 1 addition & 1 deletion conda/recipes/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ requirements:
- python
- cudf>=21.10.*
- dask-cudf>=21.10.*
- cupy>=7.2.0,<10.0.0a
- cupy>=7.2.0
- nvtx>=0.2.1
- pynvml

Expand Down