Create successful and reproducible indexes of 20 OSS repos

[shrouxm]

Steps for everyone:

1. Find a popular open source C++ repository which is not already documented in this thread. if you can think of one yourself, do that! otherwise here's some lists: GitHub trending, CodeTriage, awesome-cpp.
2. Follow the documentation on this repo to create an lsif index of the chosen repository. For now, use the simplify-cmake branch. If you're unable to create an index, file a bug documenting what you tried.
3. Follow the sourcegraph documentation to upload the lsif index to sourcegraph.com, and confirm that the c++ codenav works (feel free to spend lots of time here, C++ is a strange beast so click on some strange code to make sure it's getting indexed properly!)
4. Document the steps from cloning to successful index creation (not including getting deps for and building lsif-clang)

Steps for Garo:

1. Reproduce all of the above

[efritz]

Some findings on MacOS 10.15.

nlohmann/json

Running commands

cmake -B build -S . -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cp ./build/compile_commands.json .
lsif-clang --extra-arg='-resource-dir=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3' --executor=all-TUs compile_commands.json > dump.lsif

exits with status 138.

...
[40/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-to_chars.cpp
[41/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-inspection.cpp
[42/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-serialization.cpp
[43/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-ordered_json.cpp
[44/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-merge_patch.cpp
[45/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-reference_access.cpp
[46/50] Processing file /Users/efritz/Downloads/cpptest/json/test/src/unit-assert_macro.cpp
[1]    93496 bus error  lsif-clang  --executor=all-TUs compile_commands.json > dump.lsif

protocolbuffers/protobuf

Running commands

cmake -B buildx -S cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cd buildx
make # NOTE: I ran make here to generate .pb.cc files (probably not necessary in general)
cp ./compile_commands.json ..
cd ../
lsif-clang --extra-arg='-resource-dir=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3' --executor=all-TUs compile_commands.json > dump.lsif

produces a valid index with 7837653 elements. Passes lsif-validate with 466 overlapping ranges. All of these ranges "kiss", which may be an error in lsif-validate or lsif-clang. I believe the intervals are half-open and this should be fine.

[shrouxm]

FFmpeg/FFmpeg

• Language: C
• Environment: Ubuntu 20.04
• Steps:

./configure
bear make
<unmodified lsif-clang invocation>

[shrouxm]

nlohmann/json

• Language: C++
• Environment: Ubuntu 20.04
• Steps:

cmake -B build -S . -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
ln -s $(pwd)/build/compile_commmands.json ./
<unmodified lsif-clang invocation>

[shrouxm]

tensorflow/tensorflow

• Language: C++
• Environment: Ubuntu 20.04
• Steps:
  • follow https://www.tensorflow.org/install/source to get deps

./configure
bear -o compile_commands_raw.json bazel build //tensorflow/tools/pip_package:build_pip_package
jq '[ .[] | select(.file | startswith("tensorflow")) ]' compile_commands_raw.json | grep -v -e '-fno-canonical-system-headers' > compile_commands.json
<unmodified lsif-clang invocation>

[shrouxm]

torvalds/linux

• Language: C
• Environment: Ubuntu 20.04
• Steps:
  • use the table in Documentation/process/changes.rst to install deps

make allyesconfig
make CC=clang-10 HOSTCC=clang-10
scripts/gen_compile_commands.py
<unmodified lsif-clang invocation>

[shrouxm]

envoyproxy/envoy

• Language: C++
• Environment: Ubuntu 20.04
• Steps:

./bazel/setup_clang.sh /usr/lib/llvm-10    `# or wherever your llvm installation lives`
echo 'build --config=clang' > user.bazelrc
TEST_TMPDIR=/tmp tools/gen_compilation_database.py --include_headers --run_bazel_build
<unmodified lsif-clang invocation>

[shrouxm]

grpc/grpc

• Language: C++
• Environment: Ubuntu 20.04
• Steps:

cmake -B build -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -Dgrpc_BUILD_TESTS=ON -G Ninja
ninja -C build $(grep -e 'build gens' build/build.ninja | awk '{print $2}')  `# generate pb`
lsif-clang build/compile_commands.json

[shrouxm]

falcosecurity/falco

• Language: C++/C
• Environment: Ubuntu 20.04
• Steps:

sudo apt install git cmake build-essential libncurses-dev pkg-config autoconf libtool libelf-dev -y
sudo apt install libssl-dev libc-ares-dev libprotobuf-dev protobuf-compiler libjq-dev libgrpc++-dev protobuf-compiler-grpc libcurl4-openssl-dev
cmake -B build -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -G Ninja
ninja -C build $(egrep -e '^build[^:]+.pb.(cc|h|c|cpp)[: ]' build/build.ninja | awk '{print $2}')
lsif-clang build/compile_commands.json

[macraig]

prabhuomkar/pytorch-cpp

• Language: C++
• Environment: macOS 10.15
• Steps:

cmake -B build -DPATH_TO_LLVM=/usr/local/opt/lib
make -j8
../lsif-clang/build/bin/lsif-clang --project-root=$(pwd) --executor=all-TUs compile_commands.json --extra-arg="-resource-dir=$(clang -print-resource-dir)" > dump.lsif