Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH Change conda build directories to work with ccache #1452

Merged
merged 6 commits into from
Mar 26, 2021
+14 −9
Merged

ENH Change conda build directories to work with ccache #1452

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
76c5b7d
ENH Change conda build directories to work with ccache
dillon-cullinan Mar 15, 2021
6488783
Merge branch-0.19
dillon-cullinan Mar 25, 2021
e8d75a2
FIX Remove unnecessary var
dillon-cullinan Mar 25, 2021
dbb0b2f
FIX Copyright header in upload script
dillon-cullinan Mar 25, 2021
8366fe4
FIX Use git_url
dillon-cullinan Mar 25, 2021
3deb621
FIX Update artifact dir name
dillon-cullinan Mar 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions ci/cpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export GPUCI_CONDA_RETRY_SLEEP=30

# Use Ninja to build
export CMAKE_GENERATOR="Ninja"
export CONDA_BLD_DIR="${WORKSPACE}/.conda-bld"

################################################################################
# SETUP - Check environment
Expand Down Expand Up @@ -58,18 +59,20 @@ conda config --set ssl_verify False
gpuci_logger "Build conda pkg for libcugraph"
if [ "$BUILD_LIBCUGRAPH" == '1' ]; then
if [[ -z "$PROJECT_FLASH" || "$PROJECT_FLASH" == "0" ]]; then
conda build conda/recipes/libcugraph
gpuci_conda_retry build --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/libcugraph
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought you had mentioned that gpuci_conda_retry didn't work with the build command. is that not the case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gpuci_conda_retry doesn't work when we use conda build to get the file output names for uploading, as it somehow has more output than intended. See cudf: https://github.com/rapidsai/cudf/blob/branch-0.19/ci/cpu/build.sh#L72

else
conda build --dirty --no-remove-work-dir conda/recipes/libcugraph
gpuci_conda_retry build --no-build-id --croot ${CONDA_BLD_DIR} --dirty --no-remove-work-dir conda/recipes/libcugraph
mkdir -p ${CONDA_BLD_DIR}/libcugraph/work
cp -r ${CONDA_BLD_DIR}/work/* ${CONDA_BLD_DIR}/libcugraph/work
fi
fi

gpuci_logger "Build conda pkg for cugraph"
if [ "$BUILD_CUGRAPH" == "1" ]; then
if [[ -z "$PROJECT_FLASH" || "$PROJECT_FLASH" == "0" ]]; then
conda build conda/recipes/cugraph --python=$PYTHON
gpuci_conda_retry build --croot ${CONDA_BLD_DIR} conda/recipes/cugraph --python=$PYTHON
else
conda build conda/recipes/cugraph -c ci/artifacts/cugraph/cpu/conda-bld/ --dirty --no-remove-work-dir --python=$PYTHON
gpuci_conda_retry build --croot ${CONDA_BLD_DIR} conda/recipes/cugraph -c ci/artifacts/cugraph/cpu/.conda-bld/ --dirty --no-remove-work-dir --python=$PYTHON
fi
fi

Expand Down
5 changes: 3 additions & 2 deletions ci/cpu/upload.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/bin/bash
# Copyright (c) 2018-2021, NVIDIA CORPORATION.
#
# Adopted from https://github.com/tmcdonell/travis-scripts/blob/dfaac280ac2082cd6bcaba3217428347899f2975/update-accelerate-buildbot.sh

Expand Down Expand Up @@ -29,8 +30,8 @@ fi

gpuci_logger "Get conda file output locations"

export LIBCUGRAPH_FILE=`conda build conda/recipes/libcugraph --output`
export CUGRAPH_FILE=`conda build conda/recipes/cugraph --python=$PYTHON --output`
export LIBCUGRAPH_FILE=`conda build --no-build-id --croot ${CONDA_BLD_DIR} conda/recipes/libcugraph --output`
export CUGRAPH_FILE=`conda build --croot ${CONDA_BLD_DIR} conda/recipes/cugraph --python=$PYTHON --output`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this line contain --no-build-id also?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--no-build-id is only needed for libraries that are using ccache, as it prevents the random folder generation.


################################################################################
# UPLOAD - Conda packages
Expand Down
5 changes: 3 additions & 2 deletions ci/gpu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ function hasArg {
export PATH=/opt/conda/bin:/usr/local/cuda/bin:$PATH
export PARALLEL_LEVEL=${PARALLEL_LEVEL:-4}
export CUDA_REL=${CUDA_VERSION%.*}
export CONDA_ARTIFACT_PATH=${WORKSPACE}/ci/artifacts/cugraph/cpu/.conda-bld/

function cleanup {
gpuci_logger "Removing datasets and temp files"
Expand Down Expand Up @@ -101,11 +102,11 @@ else
chrpath -d libcugraph.so
patchelf --replace-needed `patchelf --print-needed libcugraph.so | grep faiss` libfaiss.so libcugraph.so

CONDA_FILE=`find $WORKSPACE/ci/artifacts/cugraph/cpu/conda-bld/ -name "libcugraph*.tar.bz2"`
CONDA_FILE=`find ${CONDA_ARTIFACT_PATH} -name "libcugraph*.tar.bz2"`
CONDA_FILE=`basename "$CONDA_FILE" .tar.bz2` #get filename without extension
CONDA_FILE=${CONDA_FILE//-/=} #convert to conda install
echo "Installing $CONDA_FILE"
conda install -c $WORKSPACE/ci/artifacts/cugraph/cpu/conda-bld/ "$CONDA_FILE"
conda install -c ${CONDA_ARTIFACT_PATH} "$CONDA_FILE"

echo "Build cugraph..."
$WORKSPACE/build.sh cugraph
Expand Down
2 changes: 1 addition & 1 deletion conda/recipes/cugraph/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ package:
version: {{ version }}

source:
path: ../../..
git_url: ../../..

build:
number: {{ GIT_DESCRIBE_NUMBER }}
Expand Down