Merge pull request #38 from rapidsai/branch-0.18

Merge latest branch-0.18

CHANGELOG.md

@@ -1,3 +1,11 @@
+# cuGraph 0.18.0 (Date TBD)
+
+## New Features
+
+## Improvements
+
+## Bug Fixes
+
 # cuGraph 0.17.0 (Date TBD)
 
 ## New Features
@@ -8,13 +16,18 @@
 - PR #1265 Implement Hungarian Algorithm
 - PR #1274 Add generic from_edgelist() and from_adjlist() APIs
 - PR #1279 Add self loop check variable in graph
+- PR #1277 SciPy sparse matrix input support for WCC, SCC, SSSP, and BFS
+- PR #1278 Add support for shortest_path_length and fix graph vertex checks
 
 ## Improvements
 - PR #1227 Pin cmake policies to cmake 3.17 version
 - PR #1267 Compile time improvements via Explicit Instantiation Declarations.
 - PR #1269 Removed old db code that was not being used
 - PR #1271 Add extra check to make SG Louvain deterministic
 - PR #1273 Update Force Atlas 2 notebook, wrapper and coding style
+- PR #1289 Update api.rst for MST
+- PR #1281 Update README
+- PR #1293: Updating RAFT to latest
 
 ## Bug Fixes
 - PR #1237 update tests for assymetric graphs, enable personalization pagerank
@@ -25,6 +38,9 @@
 - PR #1253 MG test fixes: updated additional comms.initialize() calls, fixed dask DataFrame comparisons
 - PR #1270 Raise exception for p2p, disable bottom up approach for bfs
 - PR #1275 Force local artifact conda install
+- PR #1285 Move codecov upload to gpu build script
+- PR #1290 Update weights check in bc and graph prims wrappers
+
 
 # cuGraph 0.16.0 (21 Oct 2020)
 

PRTAGS.md

@@ -5,8 +5,11 @@ PR = Pull Request
 
 |  TAG       |                                                       |
 |------------|-------------------------------------------------------|
-| WIP        | _Work In Progress_ - Within the RAPIDS cuGraph team, we try to open a PR when development starts.  This allows other to review code as it is being developed and provide feedback before too much code needs to be refactored.  It also allows process to be tracked |
+| WIP        | _Work In Progress_ - Within the RAPIDS cuGraph team, we try to open a PR when development starts.  This allows other to review code as it is being developed and provide feedback before too much code needs to be refactored.  It also allows process to be tracked.  __A WIP PR will not be merged into baseline__ |
 | skip-ci    | _Do Not Run CI_ - This flag prevents CI from being run.  It is good practice to include this with the **WIP** tag since code is typically not at a point where it will pass CI.  |
 | skip ci    | same as above                                          |
 | API-REVIEW | This tag request a code review just of the API portion of the code - This is  beneficial to ensure that all required arguments are captured.  Doing this early can save from having to refactor later. |
 | REVIEW     | The code is ready for a full code review.  Only code that has passed a code review is merged into the baseline  |
+
+
+Code must pass CI before it is merged

README.md

@@ -36,12 +36,12 @@ for i in range(len(df_page)):
 | Category     | Algorithm                              | Scale        |  Notes
 | ------------ | -------------------------------------- | ------------ | ------------------- |
 | Centrality   |                                        |              |                     |
-|              | Katz                                   | Single-GPU   |                     |
+|              | Katz                                   | Multi-GPU    |                     |
 |              | Betweenness Centrality                 | Single-GPU   |                     |
 |              | Edge Betweenness Centrality            | Single-GPU   |                     |
 | Community    |                                        |              |                     |
 |              | Leiden                                 | Single-GPU   |                     |
-|              | Louvain                                | Multiple-GPU |                     |
+|              | Louvain                                | Multi-GPU    |                     |
 |              | Ensemble Clustering for Graphs         | Single-GPU   |                     |
 |              | Spectral-Clustering - Balanced Cut     | Single-GPU   |                     |
 |              | Spectral-Clustering - Modularity       | Single-GPU   |                     |
@@ -57,25 +57,49 @@ for i in range(len(df_page)):
 | Layout       |                                        |              |                     |
 |              | Force Atlas 2                          | Single-GPU   |                     |
 | Link Analysis|                                        |              |                     |
-|              | Pagerank                               | Multiple-GPU |                     |
-|              | Personal Pagerank                      | Single-GPU  |                     |
+|              | Pagerank                               | Multi-GPU    |                     |
+|              | Personal Pagerank                      | Multi-GPU    |                     |
 |              | HITS                      				| Single-GPU   | leverages Gunrock   |
 | Link Prediction |                                     |              |                     |
 |              | Jaccard Similarity                     | Single-GPU   |                     |
 |              | Weighted Jaccard Similarity            | Single-GPU   |                     |
 |              | Overlap Similarity                     | Single-GPU   |                     |
 | Traversal    |                                        |              |                     |
-|              | Breadth First Search (BFS)             | Multiple-GPU |                     |
-|              | Single Source Shortest Path (SSSP)     | Multiple-GPU |                     |
+|              | Breadth First Search (BFS)             | Multi-GPU    |                     |
+|              | Single Source Shortest Path (SSSP)     | Multi-GPU    |                     |
 | Structure    |                                        |              |                     |
-|              | Renumbering                            | Single-GPU   | Also for multiple columns  |
-|              | Symmetrize                             | Single-GPU   |                     |
+|              | Renumbering                            | Single-GPU   | multiple columns, any data type  |
+|              | Symmetrize                             | Multi-GPU    |                     |
+| Other        |                                        |              |                     |
+|              | Hungarian Algorithm                    | Single-GPU   |                     |
+|              | Minimum Spanning Tree                  | Single-GPU   |                     |
+|              | Maximum Spanning Tree                  | Single-GPU   |                     |
 
+|  |  |
+
+</br></br>
 ## Supported Graph
 | Type            |  Description                                        |
 | --------------- | --------------------------------------------------- |
 | Graph           | An undirected Graph                                 |
 | DiGraph         | A Directed Graph                                    |
+| _Multigraph_      | _coming in 0.18_                                      |
+| _MultiDigraph_    | _coming in 0.18_                                      |
+|  |  |
+
+</br></br>
+## Supported Data Types
+cuGraph supports the creation of a graph several data types:
+* cuDF DataFrame
+* Pandas DataFrame
+
+cuGraph supports execution of graph algorithms from different graph objects
+* cuGraph Graph classes
+* NetworkX graph classes
+* CuPy sparse matrix
+* SciPy sparse matrix
+
+cuGraph tries to match the return type based on the input type.  So a NetworkX input will return the same data type that NetworkX would have.
 
 
 ## cuGraph Notice

build.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2019, NVIDIA CORPORATION.
+# Copyright (c) 2019-2020, NVIDIA CORPORATION.
 
 # cugraph build script
 
@@ -17,6 +17,7 @@ ARGS=$*
 # NOTE: ensure all dir changes are relative to the location of this
 # script, and that this script resides in the repo dir!
 REPODIR=$(cd $(dirname $0); pwd)
+LIBCUGRAPH_BUILD_DIR=${LIBCUGRAPH_BUILD_DIR:=${REPODIR}/cpp/build}
 
 VALIDARGS="clean libcugraph cugraph docs -v -g -n --allgpuarch --show_depr_warn -h --help"
 HELP="$0 [<target> ...] [<flag> ...]
@@ -33,19 +34,21 @@ HELP="$0 [<target> ...] [<flag> ...]
    --show_depr_warn - show cmake deprecation warnings
    -h               - print this text
 
- default action (no args) is to build and install 'libcugraph' then 'cugraph' targets and then docs
+ default action (no args) is to build and install 'libcugraph' then 'cugraph' then 'docs' targets
+
+ libcugraph build dir is: ${LIBCUGRAPH_BUILD_DIR}
+
+ Set env var LIBCUGRAPH_BUILD_DIR to override libcugraph build dir.
 "
-LIBCUGRAPH_BUILD_DIR=${LIBCUGRAPH_BUILD_DIR:=${REPODIR}/cpp/build}
 CUGRAPH_BUILD_DIR=${REPODIR}/python/build
 BUILD_DIRS="${LIBCUGRAPH_BUILD_DIR} ${CUGRAPH_BUILD_DIR}"
 
 # Set defaults for vars modified by flags to this script
-ARG_COUNT=${NUMARGS}
 VERBOSE=""
 BUILD_TYPE=Release
 INSTALL_TARGET=install
 BUILD_DISABLE_DEPRECATION_WARNING=ON
-BUILD_ALL_GPU_ARCH=0
+GPU_ARCH=""
 
 # Set defaults for vars that may not have been defined externally
 #  FIXME: if PREFIX is not set, check CONDA_PREFIX, but there is no fallback
@@ -58,6 +61,10 @@ function hasArg {
     (( ${NUMARGS} != 0 )) && (echo " ${ARGS} " | grep -q " $1 ")
 }
 
+function buildAll {
+    (( ${NUMARGS} == 0 )) || !(echo " ${ARGS} " | grep -q " [^-]\+ ")
+}
+
 if hasArg -h || hasArg --help; then
     echo "${HELP}"
     exit 0
@@ -76,30 +83,31 @@ fi
 # Process flags
 if hasArg -v; then
     VERBOSE=1
-    ARG_COUNT=$((ARG_COUNT -1))
 fi
 if hasArg -g; then
     BUILD_TYPE=Debug
-    ARG_COUNT=$((ARG_COUNT -1))
 fi
 if hasArg -n; then
     INSTALL_TARGET=""
-    ARG_COUNT=$((ARG_COUNT -1))
 fi
 if hasArg --allgpuarch; then
-    BUILD_ALL_GPU_ARCH=1
+    GPU_ARCH="-DGPU_ARCHS=ALL"
 fi
 if hasArg --show_depr_warn; then
     BUILD_DISABLE_DEPRECATION_WARNING=OFF
-    ARG_COUNT=$((ARG_COUNT -1))
 fi
 
 # If clean given, run it prior to any other steps
 if hasArg clean; then
-    # If the dirs to clean are mounted dirs in a container, the
-    # contents should be removed but the mounted dirs will remain.
-    # The find removes all contents but leaves the dirs, the rmdir
-    # attempts to remove the dirs but can fail safely.
+    # FIXME: ideally the "setup.py clean" command below would also be run to
+    # remove all the "inplace" python build artifacts, but currently, running
+    # any setup.py command has side effects (eg. cloning repos).
+    #(cd ${REPODIR}/python && python setup.py clean)
+
+    # If the dirs to clean are mounted dirs in a container, the contents should
+    # be removed but the mounted dirs will remain.  The find removes all
+    # contents but leaves the dirs, the rmdir attempts to remove the dirs but
+    # can fail safely.
     for bd in ${BUILD_DIRS}; do
 	if [ -d ${bd} ]; then
 	    find ${bd} -mindepth 1 -delete
@@ -110,53 +118,44 @@ fi
 
 ################################################################################
 # Configure, build, and install libcugraph
-if (( ${ARG_COUNT} == 0 )) || hasArg libcugraph; then
-    if (( ${BUILD_ALL_GPU_ARCH} == 0 )); then
-        GPU_ARCH=""
+if buildAll || hasArg libcugraph; then
+    if [[ ${GPU_ARCH} == "" ]]; then
         echo "Building for the architecture of the GPU in the system..."
     else
-        GPU_ARCH="-DGPU_ARCHS=ALL"
         echo "Building for *ALL* supported GPU architectures..."
     fi
-
     mkdir -p ${LIBCUGRAPH_BUILD_DIR}
     cd ${LIBCUGRAPH_BUILD_DIR}
     cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
-          ${GPU_ARCH} \
-	      -DDISABLE_DEPRECATION_WARNING=${BUILD_DISABLE_DEPRECATION_WARNING} \
-          -DCMAKE_BUILD_TYPE=${BUILD_TYPE} ..
+        ${GPU_ARCH} \
+        -DDISABLE_DEPRECATION_WARNING=${BUILD_DISABLE_DEPRECATION_WARNING} \
+        -DCMAKE_BUILD_TYPE=${BUILD_TYPE} ${REPODIR}/cpp
     make -j${PARALLEL_LEVEL} VERBOSE=${VERBOSE} ${INSTALL_TARGET}
 fi
 
 # Build and install the cugraph Python package
-if (( ${ARG_COUNT} == 0 )) || hasArg cugraph; then
-
+if buildAll || hasArg cugraph; then
     cd ${REPODIR}/python
+    # setup.py references an env var CUGRAPH_BUILD_PATH to find the libcugraph
+    # build. If not set by the user, set it to LIBCUGRAPH_BUILD_DIR
+    CUGRAPH_BUILD_PATH=${CUGRAPH_BUILD_PATH:=${LIBCUGRAPH_BUILD_DIR}}
+    env CUGRAPH_BUILD_PATH=${CUGRAPH_BUILD_PATH} python setup.py build_ext --inplace --library-dir=${LIBCUGRAPH_BUILD_DIR}
     if [[ ${INSTALL_TARGET} != "" ]]; then
-	python setup.py build_ext --inplace --library-dir=${LIBCUGRAPH_BUILD_DIR}
-	python setup.py install
-    else
-	python setup.py build_ext --inplace --library-dir=${LIBCUGRAPH_BUILD_DIR}
+	env CUGRAPH_BUILD_PATH=${CUGRAPH_BUILD_PATH} python setup.py install
     fi
 fi
 
-################################################################################
 # Build the docs
-if (( ${ARG_COUNT} == 0 )) || hasArg docs; then
-
+if buildAll || hasArg docs; then
     if [ ! -d ${LIBCUGRAPH_BUILD_DIR} ]; then
         mkdir -p ${LIBCUGRAPH_BUILD_DIR}
         cd ${LIBCUGRAPH_BUILD_DIR}
         cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
             -DDISABLE_DEPRECATION_WARNING=${BUILD_DISABLE_DEPRECATION_WARNING} \
-            -DCMAKE_BUILD_TYPE=${BUILD_TYPE} ..
+            -DCMAKE_BUILD_TYPE=${BUILD_TYPE} ${REPODIR}/cpp
     fi
-
     cd ${LIBCUGRAPH_BUILD_DIR}
-
     make -j${PARALLEL_LEVEL} VERBOSE=${VERBOSE} docs_cugraph
-
     cd ${REPODIR}/docs
     make html
-
 fi

ci/gpu/build.sh

@@ -127,4 +127,8 @@ else
     python ${WORKSPACE}/ci/utils/nbtestlog2junitxml.py nbtest.log
 fi
 
+if [ -n "\${CODECOV_TOKEN}" ]; then
+    codecov -t \$CODECOV_TOKEN
+fi
+
 return ${EXITCODE}

conda/environments/cugraph_dev_cuda10.1.yml

@@ -5,17 +5,17 @@ channels:
 - rapidsai-nightly
 - conda-forge
 dependencies:
-- cudf=0.17.*
-- libcudf=0.17.*
-- rmm=0.17.*
+- cudf=0.18.*
+- libcudf=0.18.*
+- rmm=0.18.*
 - cuxfilter=0.17.*
-- librmm=0.17.*
+- librmm=0.18.*
 - dask>=2.12.0
 - distributed>=2.12.0
-- dask-cuda=0.17*
-- dask-cudf=0.17*
+- dask-cuda=0.18*
+- dask-cudf=0.18*
 - nccl>=2.7
-- ucx-py=0.17*
+- ucx-py=0.18*
 - ucx-proc=*=gpu
 - scipy
 - networkx

conda/environments/cugraph_dev_cuda10.2.yml

@@ -5,17 +5,17 @@ channels:
 - rapidsai-nightly
 - conda-forge
 dependencies:
-- cudf=0.17.*
-- libcudf=0.17.*
-- rmm=0.17.*
+- cudf=0.18.*
+- libcudf=0.18.*
+- rmm=0.18.*
 - cuxfilter=0.17.*
-- librmm=0.17.*
+- librmm=0.18.*
 - dask>=2.12.0
 - distributed>=2.12.0
-- dask-cuda=0.17*
-- dask-cudf=0.17*
+- dask-cuda=0.18*
+- dask-cudf=0.18*
 - nccl>=2.7
-- ucx-py=0.17*
+- ucx-py=0.18*
 - ucx-proc=*=gpu
 - scipy
 - networkx

conda/environments/cugraph_dev_cuda11.0.yml

@@ -5,17 +5,17 @@ channels:
 - rapidsai-nightly
 - conda-forge
 dependencies:
-- cudf=0.17.*
-- libcudf=0.17.*
-- rmm=0.17.*
+- cudf=0.18.*
+- libcudf=0.18.*
+- rmm=0.18.*
 - cuxfilter=0.17.*
-- librmm=0.17.*
+- librmm=0.18.*
 - dask>=2.12.0
 - distributed>=2.12.0
-- dask-cuda=0.17*
-- dask-cudf=0.17*
+- dask-cuda=0.18*
+- dask-cudf=0.18*
 - nccl>=2.7
-- ucx-py=0.17*
+- ucx-py=0.18*
 - ucx-proc=*=gpu
 - scipy
 - networkx

cpp/CMakeLists.txt

@@ -16,7 +16,7 @@
 
 cmake_minimum_required(VERSION 3.12..3.17 FATAL_ERROR)
 
-project(CUGRAPH VERSION 0.17.0 LANGUAGES C CXX CUDA)
+project(CUGRAPH VERSION 0.18.0 LANGUAGES C CXX CUDA)
 
 ###################################################################################################
 # - build type ------------------------------------------------------------------------------------
@@ -316,7 +316,7 @@ else(DEFINED ENV{RAFT_PATH})
 
   ExternalProject_Add(raft
     GIT_REPOSITORY    https://github.com/rapidsai/raft.git
-    GIT_TAG           315de02f8e2304e078e5e0f6df23c6a6799e23f4
+    GIT_TAG           f75d7b437bf1da3df749108161b8a0505fb6b7b3
     PREFIX            ${RAFT_DIR}
     CONFIGURE_COMMAND ""
     BUILD_COMMAND     ""