Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor building #1332

Merged
merged 87 commits into from
Jan 1, 2020
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
3cb2757
Refactor building
dutor Nov 20, 2019
809df1c
Fix krb5 to depend on bison
dutor Nov 27, 2019
f1e87e9
Fixed required GCC version
dutor Nov 28, 2019
1611915
Fix building on ubuntu 16.04
dutor Nov 28, 2019
34ae1f2
Fixed cmake version requirement
dutor Nov 28, 2019
c3bfefc
Fixed mstch to depend on boost
dutor Nov 28, 2019
3d69e96
Added lz4
dutor Nov 28, 2019
4f89b5e
Added build-gcc.sh
dutor Nov 30, 2019
11ec02a
Added setup/resort-env.sh for GCC and CMake
dutor Nov 30, 2019
c8ac77e
Added hints for setup/restore-env.sh
dutor Nov 30, 2019
cd4ccc4
Fix to use readlink instead of realpath
dutor Nov 30, 2019
8f2b0a5
Renamed setup/restore-env to enable/disable
dutor Nov 30, 2019
a824ed4
Added binutils
dutor Nov 30, 2019
599242f
Fixed symlink creation in lz4
dutor Nov 30, 2019
e22b3e1
Stripped binutils
dutor Nov 30, 2019
980952f
Add make-package
dutor Dec 1, 2019
057d470
Add --prefix to build-gcc
dutor Dec 1, 2019
d1e8736
Fix prefix
dutor Dec 1, 2019
fb4a4f8
Fixed indentation
dutor Dec 1, 2019
66c7373
Fix make-package
dutor Dec 1, 2019
59989ed
Build binutils in source
dutor Dec 2, 2019
ed03b9d
Added gdb
dutor Dec 2, 2019
29e3928
Upgrade toolchain
dutor Dec 2, 2019
a552547
Remove gdb due to the need of a C++11 compiler
dutor Dec 3, 2019
bbc41d9
Downgrade
dutor Dec 3, 2019
326b4b3
fix typo
dutor Dec 3, 2019
8fa93ea
Fix checksum
dutor Dec 3, 2019
91c9702
Fixed building under GCC 4.4.7
dutor Dec 3, 2019
685d244
Added some comments
dutor Dec 3, 2019
ebea6f0
Add distro name
dutor Dec 4, 2019
0ce40b9
Remove --skip-old-files due to lack support by older tar command
dutor Dec 4, 2019
8471e19
Added guess-building-jobs-num.sh
dutor Dec 4, 2019
7cd974c
Added compiler testing script
dutor Dec 4, 2019
98af464
Added testing after build and install
dutor Dec 4, 2019
12fa3c7
Change enable-gcc.sh to enable
dutor Dec 4, 2019
3222c12
Fix guessing building jobs num
dutor Dec 4, 2019
b290cad
Make build log less verbose
dutor Dec 4, 2019
6c47611
Disable CMakel log
dutor Dec 4, 2019
669392f
Fixed exit on error
dutor Dec 5, 2019
50e4538
Added install-gcc.sh
dutor Dec 5, 2019
8864383
Address darionyaphet's comment
dutor Dec 5, 2019
a7843cb
Update cn docs on build
dutor Dec 5, 2019
8e5d1c7
Update cn docs on build
dutor Dec 5, 2019
33658f4
Added prebuilt GCC for glibc 2.17, 2.19
dutor Dec 5, 2019
9aa3572
Merge branch 'master' into refactor-building
dutor Dec 5, 2019
be22919
Changed prefix
dutor Dec 13, 2019
e8592e0
Ignore error when cleaning
dutor Dec 13, 2019
6934c2c
Fixed gcc/disable
dutor Dec 17, 2019
3ed629c
Reduce build size
dutor Dec 17, 2019
8ac52bd
Fixed possible depending on system libverto
dutor Dec 18, 2019
e46df1f
Merged upstream
dutor Dec 18, 2019
528b164
Removed rpath
dutor Dec 18, 2019
3c12057
Print on failure
dutor Dec 18, 2019
2aeca6e
Don't fail when grep not matched
dutor Dec 18, 2019
a0d86c9
Build third party automatically
dutor Dec 18, 2019
7a02fc8
Revert the last commit
dutor Dec 19, 2019
29f9651
Remove docs(will update in another PR)
dutor Dec 20, 2019
be2d112
Merged upstream
dutor Dec 20, 2019
b44f36c
Build third party package
dutor Dec 20, 2019
89ecc9d
Fix unknown automake building issue due to pipefail and multi-jobs
dutor Dec 20, 2019
74a4694
Changed default installation prefix to /opt/vesoft
dutor Dec 22, 2019
a8fe38c
Fixed static-libstdc++
dutor Dec 22, 2019
76406ee
Supported build with cxx11 abi
dutor Dec 22, 2019
8560842
Support for toggle cxx abi
dutor Dec 22, 2019
85bce00
Fix DISABLE_CXX11_ABI
dutor Dec 22, 2019
bc8751f
Fixed for RedHat devtoolset
dutor Dec 23, 2019
b0126da
Make Bison and krb5 relocatable
dutor Dec 23, 2019
e3e193e
Install third-party automatically
dutor Dec 23, 2019
1821ce0
Show selected version while downloading
dutor Dec 23, 2019
ed33656
Fixed relocatable for Bison in old CMake
dutor Dec 23, 2019
50acf8e
Changed presets of libc and libcxx
dutor Dec 24, 2019
9b4dc27
Add build-all-third-party.sh to build third-party with multiple versi…
dutor Dec 24, 2019
9a2351e
Unconditionally build flex and bison while building package
dutor Dec 24, 2019
1a84d71
Fixed ENV testing
dutor Dec 24, 2019
1a9d06a
Added prebuilt third party for libc-2.27
dutor Dec 24, 2019
6452f82
Deal with gold linker
dutor Dec 25, 2019
10d6a73
Renamed parser_benchmark to parser_bm
dutor Dec 25, 2019
4d38634
Merge remote-tracking branch 'upstream/master' into refactor-building
dutor Dec 25, 2019
8c5d21a
Moved setting on test binaries to nebula_add_executable
dutor Dec 25, 2019
7d699d5
Remove support for customizing level of debug info
dutor Dec 25, 2019
37dc887
Fixed for ubuntu 19.04 whose glibcxx version is messed
dutor Dec 26, 2019
78e7726
Address @laura-ding's comments
dutor Dec 27, 2019
c7d41da
Address @laura-ding's comments
dutor Dec 27, 2019
5411cf6
Merge branch 'master' into refactor-building
dutor Dec 30, 2019
64c9632
Merge branch 'master' into refactor-building
dutor Dec 31, 2019
f010158
Fixed for GCC 7
dutor Dec 31, 2019
3b3865e
Merge branch 'master' into refactor-building
dutor Jan 1, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
# ENABLE_NATIVE -- Build native client
# ENABLE_TESTING -- Build unit test
#
cmake_minimum_required(VERSION 3.0.0)
cmake_minimum_required(VERSION 3.5.0)

project("Nebula Graph" C CXX)

Expand Down Expand Up @@ -52,6 +52,7 @@ endif()

set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

set(CMAKE_EXE_LINKER_FLAGS "-static-libstdc++ -no-pie")

Expand Down Expand Up @@ -125,7 +126,11 @@ set(CMAKE_MODULE_PATH "${NEBULA_HOME}/cmake" ${CMAKE_MODULE_PATH})

# When NEBULA_THIRDPARTY_ROOT is null, set default value as /opt/nebula/third-party
if("${NEBULA_THIRDPARTY_ROOT}" STREQUAL "")
SET(NEBULA_THIRDPARTY_ROOT "/opt/nebula/third-party")
if (EXISTS ${CMAKE_BINARY_DIR}/third-party/install)
SET(NEBULA_THIRDPARTY_ROOT ${CMAKE_BINARY_DIR}/third-party/install)
else()
SET(NEBULA_THIRDPARTY_ROOT "/opt/nebula/third-party")
endif()
endif()

# third-party
Expand Down Expand Up @@ -340,7 +345,7 @@ set(THRIFT_LIBRARIES
set(ROCKSDB_LIBRARIES ${Rocksdb_LIBRARY})

# All compression libraries
set(COMPRESSION_LIBRARIES bz2 snappy zstd z)
set(COMPRESSION_LIBRARIES bz2 snappy zstd z lz4)
if (LIBLZMA_FOUND)
include_directories(SYSTEM ${LIBLZMA_INCLUDE_DIRS})
list(APPEND COMPRESSION_LIBRARIES ${LIBLZMA_LIBRARIES})
Expand Down Expand Up @@ -380,14 +385,15 @@ macro(nebula_link_libraries target)
boost_system
event
double-conversion
resolv
s2
${OPENSSL_SSL_LIBRARY}
${OPENSSL_CRYPTO_LIBRARY}
${KRB5_LIBRARIES}
${COMPRESSION_LIBRARIES}
${JEMALLOC_LIB}
${LIBUNWIND_LIBRARIES}
keyutils
dutor marked this conversation as resolved.
Show resolved Hide resolved
resolv
dutor marked this conversation as resolved.
Show resolved Hide resolved
dl
${GETTIME_LIB}
-pthread
Expand Down
2 changes: 1 addition & 1 deletion cmake/FindDoubleConversion.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# DoubleConversion_INCLUDE_DIR The double-conversion includes directories.
# DoubleConversion_LIBRARY The double-conversion library.

find_path(DoubleConversion_INCLUDE_DIR NAMES double-conversion.h)
find_path(DoubleConversion_INCLUDE_DIR NAMES double-conversion/double-conversion.h)
find_library(DoubleConversion_LIBRARY NAMES libdouble-conversion.a)

if(DoubleConversion_INCLUDE_DIR AND DoubleConversion_LIBRARY)
Expand Down
2 changes: 1 addition & 1 deletion src/graph/GroupByExecutor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ StatusOr<std::unique_ptr<InterimResult>> GroupByExecutor::setupInterimResult() {
if (rsWriter != nullptr) {
result->setInterim(std::move(rsWriter));
}
return result;
dutor marked this conversation as resolved.
Show resolved Hide resolved
return std::move(result);
}


Expand Down
226 changes: 226 additions & 0 deletions third-party/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
cmake_minimum_required(VERSION 3.5.0)

project("Nebula Third Party" C CXX)

set(CXX_STANDARD 14)
set(CXX_STANDARD_REQUIRED ON)

# Required CMake modules
include(ExternalProject)

# Get number of physical CPU cores and megabytes of available memory
cmake_host_system_information(RESULT num_cores QUERY NUMBER_OF_PHYSICAL_CORES)
cmake_host_system_information(RESULT available_memory_mb QUERY AVAILABLE_PHYSICAL_MEMORY)

execute_process(
COMMAND ldd --version
COMMAND head -1
COMMAND cut -d ")" -f 2
COMMAND cut -d " " -f 2
OUTPUT_VARIABLE GLIBC_VERSION
OUTPUT_STRIP_TRAILING_WHITESPACE
)

# Guess the number of building jobs based on the available memories
set(jobs_by_cpu ${num_cores})
math(EXPR jobs_by_mem "${available_memory_mb} / 1024 / 2")
if (jobs_by_mem EQUAL 0)
set(jobs_by_mem 1)
endif()

# Set the number of building jobs to min(jobs_by_mem, jobs_by_cpu),
# iff BUILDING_JOBS_NUM has not been set or set to 0.
if (NOT BUILDING_JOBS_NUM OR BUILDING_JOBS_NUM EQUAL 0)
set(BUILDING_JOBS_NUM ${jobs_by_cpu})
if (BUILDING_JOBS_NUM GREATER jobs_by_mem)
set(BUILDING_JOBS_NUM ${jobs_by_mem})
endif()
endif()

message(STATUS "Number of online physcial CPU cores: ${num_cores}")
message(STATUS "Available physical memory: ${available_memory_mb} MB")
message(STATUS "Building third party with ${BUILDING_JOBS_NUM} jobs")
message(STATUS "Glibc version: ${GLIBC_VERSION}")

if (NOT DOWNLOAD_DIR)
set(DOWNLOAD_DIR ${CMAKE_CURRENT_BINARY_DIR}/downloads)
endif()
set(BUILD_INFO_DIR ${CMAKE_CURRENT_BINARY_DIR}/build-info)
set(BUILDING_PATH "${CMAKE_INSTALL_PREFIX}/bin:${CMAKE_INSTALL_PREFIX}/sbin:$ENV{PATH}")
set(ACLOCAL_PATH "${CMAKE_INSTALL_PREFIX}/share/aclocal:${CMAKE_INSTALL_PREFIX}/share/aclocal-1.15")

set(extra_link_libs "-static-libstdc++ -static-libgcc")

if(GLIBC_VERSION VERSION_LESS 2.17)
set(extra_link_libs "${extra_link_libs} -lrt")
endif()

set(common_cmake_args
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
"-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -fPIC"
"-DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -fPIC"
"-DCMAKE_EXE_LINKER_FLAGS=${extra_link_libs}"
"-DCMAKE_SHARED_LINKER_FLAGS=${extra_link_libs}"
-DCMAKE_INCLUDE_PATH=${CMAKE_INSTALL_PREFIX}/include
-DCMAKE_LIBRARY_PATH=${CMAKE_INSTALL_PREFIX}/lib
)

set(common_configure_args
--prefix=${CMAKE_INSTALL_PREFIX}
--quiet
)

set(common_configure_envs
"env"
"CC=${CMAKE_C_COMPILER}"
"CXX=${CMAKE_CXX_COMPILER}"
"CFLAGS=${CMAKE_C_FLAGS} -fPIC -O2 -D_DEFAULT_SOURCE -D_GNU_SOURCE"
"CXXFLAGS=${CMAKE_CXX_FLAGS} -fPIC -O2 -D_DEFAULT_SOURCE -D_GNU_SOURCE"
"CPPFLAGS=-isystem ${CMAKE_INSTALL_PREFIX}/include"
"LDFLAGS=-L${CMAKE_INSTALL_PREFIX}/lib -L${CMAKE_INSTALL_PREFIX}/lib64"
"PATH=${BUILDING_PATH}"
"ACLOCAL_PATH=${ACLOCAL_PATH}"
)

set(ALL_TARGETS
libunwind
openssl
libevent
keyutils
krb5
jemalloc
bzip2
zstd
zlib
snappy
lzma
lz4
libaio
mstch
fatal
double-conversion
gflags
googletest
glog
boost
rocksdb
folly
wangle
proxygen
fbthrift
capstone
dutor marked this conversation as resolved.
Show resolved Hide resolved
s2geometry
dutor marked this conversation as resolved.
Show resolved Hide resolved
dutor marked this conversation as resolved.
Show resolved Hide resolved
)

list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/cmake)
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/../cmake)

find_package(Libtool 2.4.2 QUIET)
if (NOT Libtool_FOUND)
list(APPEND ALL_TARGETS libtool)
endif()

find_package(Autoconf 2.69 QUIET)
if (NOT Autoconf_FOUND)
list(APPEND ALL_TARGETS autoconf)
endif()

find_package(AutoconfArchive QUIET)
if (NOT AutoconfArchive_FOUND)
list(APPEND ALL_TARGETS autoconf-archive)
endif()

find_package(Automake 1.13.4 QUIET)
if (NOT Automake_FOUND)
list(APPEND ALL_TARGETS automake)
endif()

find_package(BISON 3.0.5 QUIET)
if (NOT BISON_FOUND)
list(APPEND ALL_TARGETS bison)
endif()

find_package(FLEX 2.6.0 QUIET)
if (NOT FLEX_FOUND)
list(APPEND ALL_TARGETS flex)
endif()

find_package(GPERF 2.8 QUIET)
if (NOT GPERF_FOUND)
list(APPEND ALL_TARGETS gperf)
endif()

find_package(Gettext QUIET)
if (NOT Gettext_FOUND)
list(APPEND ALL_TARGETS gettext)
endif()

foreach(target ${ALL_TARGETS})
include(externals/${target}.cmake)
list(APPEND CLEAN_TARGETS ${target}-clean)
endforeach()

macro(maybe_add_dependencies depender)
if (TARGET ${depender})
foreach (dependee ${ARGN})
if (TARGET ${dependee})
add_dependencies(${depender} ${dependee})
endif()
endforeach()
endif()
endmacro()

maybe_add_dependencies(autoconf-archive autoconf)
maybe_add_dependencies(automake autoconf autoconf-archive)
maybe_add_dependencies(libtool automake)

maybe_add_dependencies(glog libtool)
maybe_add_dependencies(bison libtool)
maybe_add_dependencies(flex libtool)
maybe_add_dependencies(zlib libtool)
maybe_add_dependencies(gettext libtool)
maybe_add_dependencies(lzma gettext)
maybe_add_dependencies(libevent libtool)
maybe_add_dependencies(gperf libtool)

maybe_add_dependencies(boost zlib bzip2 lzma)
maybe_add_dependencies(mstch boost)

maybe_add_dependencies(glog gflags)

maybe_add_dependencies(s2geometry googletest glog openssl)

maybe_add_dependencies(krb5 keyutils openssl gettext bison)
maybe_add_dependencies(folly glog boost double-conversion openssl libevent lzma zstd snappy lz4 libunwind libaio)
maybe_add_dependencies(wangle folly)
maybe_add_dependencies(fbthrift folly krb5 bison flex mstch zlib zstd wangle fatal)
maybe_add_dependencies(proxygen wangle libunwind gperf)
maybe_add_dependencies(rocksdb snappy zlib zstd bzip2 lzma libunwind)

add_custom_command(
TARGET zstd POST_BUILD
COMMAND
rm -rf ${CMAKE_INSTALL_PREFIX}/lib/libzstd.so*
)

add_custom_command(
TARGET rocksdb POST_BUILD
COMMAND
rm -rf ${CMAKE_INSTALL_PREFIX}/lib64/librocksdb.so*
)

add_custom_target(
clean-all
DEPENDS ${CLEAN_TARGETS}
)

add_custom_target(
pack-downloads
COMMAND
tar czvf nebula-third-party-src-1.0.tgz downloads
COMMAND
md5sum nebula-third-party-src-1.0.tgz
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/..
)
56 changes: 56 additions & 0 deletions third-party/build-cmake.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#! /usr/bin/env bash

# Copyright (c) 2019 vesoft inc. All rights reserved.
#
# This source code is licensed under Apache 2.0 License,
# attached with Common Clause Condition 1.0, found in the LICENSES directory.

# Usage: build-cmake.sh [prefix]

# Always use bash
shell=$(basename $(readlink /proc/$$/exe))
if [ ! x$shell = x"bash" ]
then
bash $0 $@
exit $?
fi

archive=cmake-3.15.5-Linux-x86_64.sh
url=https://cmake.org/files/v3.15/$archive
prefix=`pwd`/cmake-3.15.5

if [[ -n $1 ]]
then
prefix=$1
fi

if [[ -f $archive ]]
then
checksum=$(md5sum $archive | cut -d ' ' -f 1)
fi

if [[ ! $checksum = 35d56e9c27b4fd2819a11c29320c655a ]]
then
hash wget &> /dev/null && download_cmd="wget -c"
hash axel &> /dev/null && download_cmd="axel -a -n 16"
if [[ -z $download_cmd ]]
then
echo "Neither 'wget' nor 'axel' available for downloading" 1>&2;
exit 1;
fi

echo "Downloading with $download_cmd..."
if ! bash -c "$download_cmd $url"
then
echo "Download failed."
exit 1
fi
fi

mkdir -p $prefix
bash $archive --prefix=$prefix &> /dev/null <<EOF
yes
no
EOF

echo "CMake has been installed to prefix=$prefix"
Loading