Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Remove FAISS dependency, inherit other common dependencies from raft #1863

Merged
merged 74 commits into from
Jan 23, 2022
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
d9f5ec6
add option to build faiss shared libs
trxcllnt Sep 30, 2021
efe559d
update cuco and point to my raft fork
trxcllnt Sep 30, 2021
332fed7
create a faiss-exports export-set
trxcllnt Sep 30, 2021
8e509de
mark faiss to be found as part of resolving cugraph-exports dependencies
trxcllnt Sep 30, 2021
d0077e0
create faiss-exports export set
trxcllnt Sep 30, 2021
b32f1cc
create faiss-exports export set
trxcllnt Sep 30, 2021
9ea0f75
compile both compute and sm if building for a single GPU arch
trxcllnt Oct 1, 2021
e7e4160
Merge branch 'fix/build-shared-faiss' into fix/node-rapids-21.10
trxcllnt Oct 1, 2021
ce9d0df
remove faiss from cugraph dependencies
trxcllnt Oct 5, 2021
a9f3630
remove CUGRAPH_USE_FAISS_STATIC and BLAS_LIBRARIES
trxcllnt Oct 5, 2021
4155315
use fix/node-rapids-21.10 branch
trxcllnt Oct 5, 2021
896df79
Merge branch 'branch-21.10' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 5, 2021
de281b4
use rapidsai/raft again
trxcllnt Oct 5, 2021
4d579f8
use CUGRAPH_BRANCH_VERSION_raft
trxcllnt Oct 5, 2021
b3c7e61
fix FORK
trxcllnt Oct 5, 2021
5d0c4db
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 5, 2021
fa1e96c
update rapids-cmake version
trxcllnt Oct 6, 2021
92d794c
remove redundant dependencies that raft provides, remove dead code
trxcllnt Oct 7, 2021
7e635ff
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 7, 2021
91d217f
no faiss
trxcllnt Oct 7, 2021
583024c
switch to my fork to test in CI
trxcllnt Oct 14, 2021
6eeead5
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 14, 2021
fb155ab
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 14, 2021
1d71d15
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 18, 2021
efa2d26
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Oct 20, 2021
0ceb8d0
remove dead code
trxcllnt Oct 27, 2021
c3c7422
add workaround to fix path in CI
trxcllnt Oct 27, 2021
13022c1
print PATH and ld paths in build.sh
trxcllnt Oct 27, 2021
0e47956
don't add include dirs that we get from linked targets
trxcllnt Oct 27, 2021
dbc90e7
print all the custom gcc9 executables in CI
trxcllnt Oct 27, 2021
b012c62
print the contents of /usr/local/gcc9 in CI
trxcllnt Oct 27, 2021
23c146f
add custom binutils to path
trxcllnt Oct 29, 2021
68afa4d
print ld paths
trxcllnt Oct 29, 2021
242c7bb
force CXX11 ABI
trxcllnt Nov 2, 2021
639fd12
print env
trxcllnt Nov 2, 2021
64721b2
remove build.sh PATH workaround
trxcllnt Nov 4, 2021
822c747
link gtest and gmock to C tests
trxcllnt Nov 4, 2021
9d5c700
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 4, 2021
f62fb52
updating cuh to hpp for rng
divyegala Nov 9, 2021
15e89b9
updating raft tag
divyegala Nov 9, 2021
15010b6
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 10, 2021
aa68e0d
remove dead code
trxcllnt Nov 10, 2021
6e85d8f
add comment indicating which dependencies we're getting from RAFT
trxcllnt Nov 10, 2021
e0d20f4
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 11, 2021
028b893
Revert RAFT pin back to main
divyegala Nov 11, 2021
90281a6
Removing unnecessary include
divyegala Nov 11, 2021
02e755c
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 11, 2021
e204658
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 11, 2021
2f93d49
Merge remote-tracking branch 'upstream/branch-21.12' into imp-21.12-r…
divyegala Nov 12, 2021
04f6728
Merge branch 'imp-21.12-rng_detail' of github.com:divyegala/cugraph i…
trxcllnt Nov 12, 2021
bad10fa
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 12, 2021
f5710c3
remove dead code
trxcllnt Nov 12, 2021
0d7c110
remove unused variable
trxcllnt Nov 12, 2021
940c0e3
Merge branch 'branch-21.12' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 16, 2021
9729a3d
remove faiss patchelf
trxcllnt Nov 16, 2021
ac8fcfa
remove faiss from conda dev envs and recipe
trxcllnt Nov 16, 2021
8b0ef92
set BLAS_LIBRARIES to empty string to work around faiss' CMakeLists.txt
trxcllnt Nov 16, 2021
a167883
revert unneeded changes
trxcllnt Nov 16, 2021
1865b40
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 18, 2021
d5b5c5d
fix merge conflict
trxcllnt Nov 20, 2021
b732c18
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Nov 20, 2021
e69f402
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Dec 1, 2021
1c6b000
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Dec 15, 2021
877faad
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Jan 6, 2022
6fa9197
Merge branch 'fix/build-shared-faiss' of github.com:trxcllnt/cugraph …
trxcllnt Jan 6, 2022
fddef0e
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Jan 11, 2022
b264da1
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Jan 13, 2022
db9e359
update copyright headers
trxcllnt Jan 13, 2022
300b0e4
switch to use Robert's branch
trxcllnt Jan 13, 2022
879223c
add get_gtest.cmake
trxcllnt Jan 14, 2022
23de039
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Jan 19, 2022
c65d2ff
set RAFT_COMPILE_LIBRARIES OFF in get_raft.cmake
trxcllnt Jan 19, 2022
a8a13f5
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Jan 20, 2022
bb815aa
Merge branch 'branch-22.02' of github.com:rapidsai/cugraph into fix/b…
trxcllnt Jan 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 11 additions & 42 deletions cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
#=============================================================================

cmake_minimum_required(VERSION 3.20.1 FATAL_ERROR)
file(DOWNLOAD https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-21.10/RAPIDS.cmake
file(DOWNLOAD https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-21.12/RAPIDS.cmake
${CMAKE_BINARY_DIR}/RAPIDS.cmake)
include(${CMAKE_BINARY_DIR}/RAPIDS.cmake)

Expand Down Expand Up @@ -55,7 +55,11 @@ endforeach()
# arch > 80
list(POP_BACK usable_arch_values latest_arch)
list(TRANSFORM usable_arch_values APPEND "-real")
list(APPEND usable_arch_values ${latest_arch})
if (usable_arch_values)
list(APPEND usable_arch_values ${latest_arch})
else()
list(APPEND usable_arch_values ${latest_arch}-real)
endif()

set(CMAKE_CUDA_ARCHITECTURES ${usable_arch_values})

Expand All @@ -73,8 +77,6 @@ rapids_cmake_build_type(Release)
# - User Options --------------------------------------------------------------

option(BUILD_CUGRAPH_MG_TESTS "Build cuGraph multigpu algorithm tests" OFF)
set(BLAS_LIBRARIES "" CACHE STRING "Location of BLAS library for FAISS build.")
option(BUILD_STATIC_FAISS "Build the FAISS library for nearest neighbors search on GPU" OFF)
option(CMAKE_CUDA_LINEINFO "Enable the -lineinfo option for nvcc (useful for cuda-memcheck / profiler" OFF)
option(BUILD_TESTS "Configure CMake to build tests" ON)

Expand Down Expand Up @@ -122,37 +124,16 @@ if(OpenMP_FOUND)
list(APPEND CUGRAPH_CUDA_FLAGS -Xcompiler=${OpenMP_CXX_FLAGS})
endif()


################################################################################
# - find blas ------------------------------------------------------------------

if(NOT DEFINED BLAS_LIBRARIES)
find_package( BLAS REQUIRED )
else()
message(STATUS "Manually setting BLAS to ${BLAS_LIBRARIES}")
endif()

################################################################################
# - find CPM based dependencies -----------------------------------------------
###################################################################################################
# - find CPM based dependencies ------------------------------------------------------------------

rapids_cpm_init()

include(cmake/thirdparty/get_thrust.cmake)
include(cmake/thirdparty/get_faiss.cmake)
include(cmake/thirdparty/get_nccl.cmake)
include(cmake/thirdparty/get_rmm.cmake)

include(cmake/thirdparty/get_raft.cmake)
ChuckHastings marked this conversation as resolved.
Show resolved Hide resolved

include(cmake/thirdparty/get_cuco.cmake)
include(cmake/thirdparty/get_nccl.cmake)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

raft also uses nccl. Is there a reason for including the nccl dependency here as well? Seems like if we're going to remove dependencies we inherit from raft we should not include nccl here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I agree, just wasn't successful at making raft's get_nccl.cmake export its NCCL::NCCL target. Could use some help from @robertmaynard here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks RAFT only optional uses nccl. Specifically only the comms components of RAFT require NCCL.

What RAFT should do is have finer grained targets ( raft_comm ) which allow you to bring in the NCCL dependency.

I expect that CUML would want to keep NCCL as an optional dependency of RAFT since they don't require it when building for single gpu.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChuckHastings, isnt cugraph is still using the RAFT comms?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, cugraph uses the RAFT comms implementation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I misunderstood the original statement- you were saying cugraph should be getting nccl transitively through raft.

include(cmake/thirdparty/get_cuhornet.cmake)

include(cmake/thirdparty/get_gunrock.cmake)

if(BUILD_TESTS)
include(cmake/thirdparty/get_gtest.cmake)
endif()

################################################################################
# - libcugraph library target --------------------------------------------------

Expand Down Expand Up @@ -278,17 +259,9 @@ target_include_directories(cugraph
# - link libraries -------------------------------------------------------------
target_link_libraries(cugraph
PUBLIC
rmm::rmm
cugraph::Thrust
raft::raft
PRIVATE
cuco::cuco
CUDA::cublas
CUDA::curand
CUDA::cusolver
CUDA::cusparse
cugraph::cuHornet
FAISS::FAISS
gunrock
NCCL::NCCL
)
Expand Down Expand Up @@ -381,7 +354,6 @@ target_include_directories(cugraph_c
# - C-API link libraries -------------------------------------------------------
target_link_libraries(cugraph_c
PUBLIC
FAISS::FAISS
CUDA::cublas
CUDA::curand
CUDA::cusolver
Expand Down Expand Up @@ -434,12 +406,9 @@ cuGraph library is a collection of GPU accelerated graph algorithms that process

]=])

set(code_string
[=[
thrust_create_target(cugraph::Thrust FROM_OPTIONS)
]=])
set(code_string "")
trxcllnt marked this conversation as resolved.
Show resolved Hide resolved

rapids_export(INSTALL cugraph
rapids_export(INSTALL cugraph
EXPORT_SET cugraph-exports
GLOBAL_TARGETS cugraph cugraph_c
NAMESPACE cugraph::
Expand Down
35 changes: 0 additions & 35 deletions cpp/cmake/thirdparty/get_cuco.cmake

This file was deleted.

53 changes: 0 additions & 53 deletions cpp/cmake/thirdparty/get_faiss.cmake

This file was deleted.

24 changes: 0 additions & 24 deletions cpp/cmake/thirdparty/get_gtest.cmake

This file was deleted.

27 changes: 0 additions & 27 deletions cpp/cmake/thirdparty/get_rmm.cmake

This file was deleted.

28 changes: 0 additions & 28 deletions cpp/cmake/thirdparty/get_thrust.cmake

This file was deleted.

15 changes: 3 additions & 12 deletions cpp/tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,7 @@ target_include_directories(cugraphtestutil

target_link_libraries(cugraphtestutil
PUBLIC
cugraph
cuco::cuco
cugraph::cugraph
NCCL::NCCL
)

Expand Down Expand Up @@ -78,16 +77,12 @@ function(ConfigureTest CMAKE_TEST_NAME)
target_link_libraries(${CMAKE_TEST_NAME}
PRIVATE
cugraphtestutil
cugraph
cugraph::cugraph
GTest::gmock
GTest::gmock_main
GTest::gtest
GTest::gtest_main
NCCL::NCCL
CUDA::cublas
CUDA::cusparse
CUDA::cusolver
CUDA::curand
)

if(OpenMP_CXX_FOUND)
Expand Down Expand Up @@ -151,16 +146,12 @@ function(ConfigureTestMG CMAKE_TEST_NAME)
PRIVATE
cugraphmgtestutil
cugraphtestutil
cugraph
cugraph::cugraph
GTest::gmock
GTest::gmock_main
GTest::gtest
GTest::gtest_main
NCCL::NCCL
CUDA::cublas
CUDA::cusparse
CUDA::cusolver
CUDA::curand
MPI::MPI_CXX
)

Expand Down