Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add wheel builds to rmm #1148

Merged
merged 100 commits into from
Nov 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
464aa91
Add initial cibuildwheel workflow
sevagh Jul 28, 2022
bc2d362
Add nvidia-smi to debug smoketest
sevagh Aug 11, 2022
f49eb33
Build and test ARM64 wheels for RMM
sevagh Aug 17, 2022
2c72c0c
Remove debug calls to nvidia-smi
sevagh Aug 18, 2022
cd31d84
Ignore wheelhouse dir
sevagh Aug 19, 2022
576505a
Run on new manylinux container
sevagh Aug 25, 2022
6d92685
Add unit tests
sevagh Aug 27, 2022
e9e4247
Use combined build
sevagh Aug 28, 2022
bab2f77
Switch from -cuda11 to -cu11 suffix
sevagh Sep 1, 2022
3fdc703
statically link CUDA runtime if building wheels
trxcllnt Sep 13, 2022
898f854
set include_package_data=True for scikit-build
trxcllnt Sep 14, 2022
3122914
Use new cibuildwheel branch
sevagh Sep 19, 2022
8cb51a5
Try building it in manylinux_2_27 container
sevagh Sep 20, 2022
327680e
Back to manylinux_2_31
sevagh Sep 20, 2022
273e7fe
Split rmm into arch-specific wheels for manylinux2014
sevagh Sep 20, 2022
41a3164
Test rmm in ubuntu 18.04
sevagh Sep 20, 2022
297383b
Switch to unified workflow for both arches
sevagh Sep 21, 2022
22f2828
Use single publish runner
sevagh Sep 21, 2022
55c327e
Switch to updated syntax with more bolts exposed
sevagh Sep 21, 2022
523222a
Override versioneer style to pep440-pre
sevagh Sep 27, 2022
545309b
Versioneer isn't that cool
sevagh Sep 27, 2022
89e35a3
Testing build-tag incrementer
sevagh Sep 28, 2022
67579b2
Improve versioneer override
sevagh Sep 28, 2022
b48d121
Add empty build tag
sevagh Sep 29, 2022
3857250
Verify test-checkout-src default behavior
sevagh Sep 30, 2022
e45f84b
Add LICENSE file for wheel
sevagh Oct 3, 2022
76a5fbf
Merge remote-tracking branch 'upstream/branch-22.10' into feat/cibuil…
sevagh Oct 3, 2022
6c6df2a
Bump build tag for LICENSE
sevagh Oct 3, 2022
905dbc4
Merge remote-tracking branch 'origin/branch-22.10' into feat/cibuildw…
vyasr Oct 10, 2022
add8649
Bump build tag.
vyasr Oct 10, 2022
5d628a2
Fix style.
vyasr Oct 10, 2022
4786160
Bump build tag again.
vyasr Oct 10, 2022
9564b2b
Add concurrency rules so that prior runs are canceled.
vyasr Oct 10, 2022
fbed104
Centralized logic in used workflow.
vyasr Oct 13, 2022
54fc2d2
Use project-wide settings, bump build tag
sevagh Oct 13, 2022
ba45124
Use stable version 22.10.00 override
sevagh Oct 17, 2022
6ed0534
python-versions also dropped
sevagh Oct 17, 2022
af55d45
Merge remote-tracking branch 'origin/branch-22.12' into feat/cibuildw…
vyasr Nov 2, 2022
fa6bdfe
Don't disable CUDA_STATIC_RUNTIME just because we're not building whe…
vyasr Nov 2, 2022
5d2c141
Properly set options in wheels.yml for static cudart.
vyasr Nov 2, 2022
0399b13
Fix comment.
vyasr Nov 2, 2022
5c01d45
Temporarily remove unnecessary workflows.
vyasr Nov 2, 2022
8fd902c
rmm doesn't actually need cuda to compile.
vyasr Nov 2, 2022
4be4834
Change command from py.test to pytest.
vyasr Nov 2, 2022
b82c4f6
Update cuda-python requirement and standardize location.
vyasr Nov 2, 2022
fbad4eb
Iterate on new wheel-ci-actions-3 branch
sevagh Nov 3, 2022
659d033
Remove conda workflows.
vyasr Nov 3, 2022
8c22f44
Try new per-arch cibuildwheel
sevagh Nov 3, 2022
4f70bf0
Bump
sevagh Nov 3, 2022
7d9ae17
Bump
sevagh Nov 3, 2022
e361d5d
Bump
sevagh Nov 3, 2022
7c5ff2d
Bump
sevagh Nov 3, 2022
7465a5f
Bump
sevagh Nov 4, 2022
70db765
Bump
sevagh Nov 4, 2022
8bb6912
Bump
sevagh Nov 4, 2022
f2b11af
Bump
sevagh Nov 4, 2022
087694b
Bump
sevagh Nov 4, 2022
b3b93cc
Bump
sevagh Nov 4, 2022
b9c8abd
Bump
sevagh Nov 4, 2022
1c47dd8
Bump
sevagh Nov 7, 2022
c4c6f18
Bump
sevagh Nov 7, 2022
b80f657
Bump
sevagh Nov 7, 2022
4f4bd9c
Test auto version guessing
sevagh Nov 8, 2022
f070f10
Bump
sevagh Nov 8, 2022
4f0fafa
Bump
sevagh Nov 8, 2022
b8667ea
Bump
sevagh Nov 8, 2022
41bc691
Bump
sevagh Nov 8, 2022
45bf708
Add workflow_call to invoke wheel build externally
sevagh Nov 8, 2022
d4bc26b
Bump
sevagh Nov 8, 2022
e910514
Bump
sevagh Nov 9, 2022
ebb9153
Specify repo and branch explicitly
sevagh Nov 9, 2022
681a7d0
Bump
sevagh Nov 9, 2022
51e1a39
Bump
sevagh Nov 9, 2022
99d83e5
Bump
sevagh Nov 9, 2022
37bce63
Align more with conda build.yaml
sevagh Nov 9, 2022
9ef5d35
Bump
sevagh Nov 9, 2022
a514586
Use build-type not build_type
sevagh Nov 9, 2022
e0b4842
Bump
sevagh Nov 9, 2022
2ddba76
Bump
sevagh Nov 9, 2022
b5b34c1
Bump
sevagh Nov 9, 2022
faebfb6
Bump
sevagh Nov 9, 2022
045ca6b
Bump
sevagh Nov 9, 2022
d9c9dae
Bump
sevagh Nov 9, 2022
cd5e76f
Bump
sevagh Nov 9, 2022
a10a8e5
Remove redundant license file.
vyasr Nov 9, 2022
26641c0
Add back license as symlink.
vyasr Nov 9, 2022
a0b2ffa
Revert "Remove conda workflows."
vyasr Nov 9, 2022
2eaa56a
Revert "Temporarily remove unnecessary workflows."
vyasr Nov 9, 2022
6de41e3
Switch to default branch for workflow.
vyasr Nov 9, 2022
0a7b19c
Specify main branch.
vyasr Nov 9, 2022
162380e
Merge remote-tracking branch 'origin/branch-22.12' into feat/cibuildw…
vyasr Nov 9, 2022
b9ca278
Address PR comments.
vyasr Nov 10, 2022
1c5dfb7
Remove unnecessary strings.
vyasr Nov 10, 2022
fd0b93c
Temporarily put back push.
vyasr Nov 10, 2022
693cf1f
Fix indentation.
vyasr Nov 10, 2022
04d343b
Confirmed yaml is valid, remove push again.
vyasr Nov 10, 2022
13a9d9e
Try specifying github.repository.
vyasr Nov 14, 2022
0e08199
Put back builds to see if it cancels other repo builds.
vyasr Nov 14, 2022
7d6310d
Revert "Put back builds to see if it cancels other repo builds."
vyasr Nov 14, 2022
3c963a5
Hardcode concurrency name.
vyasr Nov 15, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: RMM wheels

on:
workflow_call:
inputs:
versioneer-override:
type: string
default: ''
build-tag:
type: string
default: ''
branch:
required: true
type: string
date:
required: true
type: string
sha:
required: true
type: string
build-type:
type: string
default: nightly

concurrency:
# Hardcode rmm rather than using github.repository to support calling this
# from other workflows that build multiple wheels.
group: "rmm-${{ github.workflow }}-${{ github.ref }}"
cancel-in-progress: true

jobs:
rmm-wheels:
uses: rapidsai/shared-action-workflows/.github/workflows/wheels-manylinux.yml@main
with:
repo: rapidsai/rmm

build-type: ${{ inputs.build-type }}
branch: ${{ inputs.branch }}
sha: ${{ inputs.sha }}
date: ${{ inputs.date }}

package-dir: python
package-name: rmm

python-package-versioneer-override: ${{ inputs.versioneer-override }}
python-package-build-tag: ${{ inputs.build-tag }}
vyasr marked this conversation as resolved.
Show resolved Hide resolved

skbuild-configure-options: "-DRMM_BUILD_WHEELS=ON"

test-extras: test
test-unittest: "python -m pytest -v ./python/rmm/tests"
vyasr marked this conversation as resolved.
Show resolved Hide resolved
secrets: inherit
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,6 @@ ENV/

# RMM log files
rmm_log.txt

# cibuildwheel
/wheelhouse
15 changes: 11 additions & 4 deletions python/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ project(

option(FIND_RMM_CPP "Search for existing RMM C++ installations before defaulting to local files"
OFF)
option(RMM_BUILD_WHEELS "Whether this build is generating a Python wheel." OFF)

# If the user requested it we attempt to find RMM.
if(FIND_RMM_CPP)
Expand All @@ -40,12 +41,18 @@ else()
endif()

if(NOT rmm_FOUND)
# TODO: This will not be necessary once we upgrade to CMake 3.22, which will pull in the required
# languages for the C++ project even if this project does not require those languges.
enable_language(CUDA)
set(BUILD_TESTS OFF)
set(BUILD_BENCHMARKS OFF)
add_subdirectory(../ rmm-cpp)

set(_exclude_from_all "")
if(RMM_BUILD_WHEELS)
# Statically link dependencies if building wheels
set(CUDA_STATIC_RUNTIME ON)
# Don't install the rmm C++ targets into wheels
set(_exclude_from_all EXCLUDE_FROM_ALL)
endif()

add_subdirectory(../ rmm-cpp ${_exclude_from_all})
endif()

include(rapids-cython)
Expand Down
1 change: 1 addition & 0 deletions python/LICENSE
9 changes: 0 additions & 9 deletions python/setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,3 @@ skip=
build
dist
__init__.py


[options]
packages = find:
install_requires =
cuda-python>=11.5,<11.7.1
numpy>=1.19
numba>=0.49
python_requires = >=3.8
14 changes: 10 additions & 4 deletions python/setup.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# Copyright (c) 2019-2022, NVIDIA CORPORATION.

import os

from setuptools import find_packages
from skbuild import setup

import versioneer

setup(
name="rmm",
version=versioneer.get_version(),
name="rmm" + os.getenv("RAPIDS_PY_WHEEL_CUDA_SUFFIX", default=""),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am wondering if there is a way to keep the name unchanged. Could we have a common dependency? Are there other ways to encode this information?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on the solution that you are proposing with a common dependency? The purpose of this approach is to support installation into environments with different API incompatible versions of CUDA. Specifically, once we start supporting CUDA 12 we will need wheels built specifically for the appropriate CUDA major version. Without first class support in the wheel spec itself I am not sure how else we can handle this other than to have separate packages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jakirkham in order to expedite getting wheels done for this release I'm going to go ahead and merge this PR, but I'm happy to revise anything in follow-ups if we can come up with a better solution so please respond and continue the conversation. If we come up with something soon I would even be open to addressing it this release, although that might be tricky given how much of our wheels-related infrastructure is currently built around the -cu11 suffix (we've been cleaning lots of things up, and the suffix is configurable during builds, but I am sure we also have it hardcoded in many places like dependency installations that would also need updating).

version=os.getenv(
"RAPIDS_PY_WHEEL_VERSIONEER_OVERRIDE", default=versioneer.get_version()
vyasr marked this conversation as resolved.
Show resolved Hide resolved
),
description="rmm - RAPIDS Memory Manager",
url="https://github.com/rapidsai/rmm",
author="NVIDIA Corporation",
Expand All @@ -22,8 +26,10 @@
"Programming Language :: Python :: 3.9",
],
# Include the separately-compiled shared library
extras_require={"test": ["pytest", "pytest-xdist"]},
extras_require={"test": ["pytest"]},
packages=find_packages(include=["rmm", "rmm.*"]),
include_package_data=True,
python_requires=">=3.8",
package_data={
# Note: A dict comprehension with an explicit copy is necessary (rather
# than something simpler like a dict.fromkeys) because otherwise every
Expand All @@ -35,7 +41,7 @@
},
cmdclass=versioneer.get_cmdclass(),
install_requires=[
"cuda-python>=11.5,<11.7.1",
"cuda-python>=11.7.1,<12.0",
"numpy>=1.19",
"numba>=0.49",
],
Expand Down