feat: Add Profiling Support with gperftools (#1274)

Closes: #1273

This PR adds two new build options: `ACTS_ENABLE_CPU_PROFILING` and `ACTS_ENABLE_MEMORY_PROFILING` that will link libraries and add compile options to enable the use of gperftools' CPU profiler and memory profiler/heap checker respectively. The libraries are linked in the `Core` package as the code here is always built and is the main point of interest for performance profiling.

This PR also adds documentation to `docs/howto` for installing, enabling and using gperftools with ACTS, including some information about basic commands that I found useful when profiling ACTS as well as relevant links.

CMakeLists.txt

@@ -69,6 +69,9 @@ option(ACTS_USE_SYSTEM_BOOST "Use a system-provided boost" ON)
 option(ACTS_SETUP_EIGEN3 "Explicitly set up Eigen3 for the project" ON)
 option(ACTS_USE_SYSTEM_EIGEN3 "Use a system-provided eigen3" ON)
 option(ACTS_BUILD_ODD "Build the OpenDataDetector" OFF)
+# profiling related optios
+option(ACTS_ENABLE_CPU_PROFILING "Enable CPU profiling using gperftools" OFF)
+option(ACTS_ENABLE_MEMORY_PROFILING "Enable memory profiling using gperftools" OFF)
 
 # handle option inter-dependencies and the everything flag
 # NOTE: ordering is important here. dependencies must come before dependees

Core/CMakeLists.txt

@@ -37,6 +37,42 @@ if(ACTS_LOG_FAILURE_THRESHOLD)
     PUBLIC -DACTS_LOG_FAILURE_THRESHOLD=${ACTS_LOG_FAILURE_THRESHOLD})
 endif()
 
+if(ACTS_ENABLE_CPU_PROFILING)
+  message(STATUS "added lprofiler")
+
+  if(DEFINED GPERF_INSTALL_DIR)
+    find_library(PROFILER_LIBRARY profiler HINTS ${GPERF_INSTALL_DIR})
+
+    target_link_libraries(
+      ActsCore
+      PUBLIC ${PROFILER_LIBRARY})
+  else()
+    target_link_libraries(
+      ActsCore
+      PUBLIC -lprofiler)
+  endif()
+
+  target_link_options(
+    ActsCore
+      PUBLIC "LINKER:-no-as-needed")
+endif()
+
+if(ACTS_ENABLE_MEMORY_PROFILING)
+  message(STATUS "added ltcmalloc")
+
+  if(DEFINED GPERF_INSTALL_DIR)
+    find_library(TCMALLOC_LIBRARY tcmalloc HINTS ${GPERF_INSTALL_DIR})
+
+    target_link_libraries(
+      ActsCore
+      PUBLIC ${TCMALLOC_LIBRARY})
+  else()
+    target_link_libraries(
+      ActsCore
+      PUBLIC -ltcmalloc)
+  endif()
+endif()
+
 install(
   TARGETS ActsCore
   EXPORT ActsCoreTargets

cmake/ActsCompilerOptions.cmake

@@ -10,6 +10,11 @@ set(ACTS_CXX_FLAGS_MINSIZEREL "")
 set(ACTS_CXX_FLAGS_RELEASE "")
 set(ACTS_CXX_FLAGS_RELWITHDEBINFO "")
 
+if(ACTS_ENABLE_CPU_PROFILING OR ACTS_ENABLE_MEMORY_PROFILING)
+  message(STATUS "Added -g compile flag")
+  set(ACTS_CXX_FLAGS "${ACTS_CXX_FLAGS} -g")
+endif()
+
 # Acts linker flags
 set(ACTS_EXE_LINKER_FLAGS_DEBUG "--coverage")
 set(ACTS_SHARED_LINKER_FLAGS_DEBUG "--coverage ")

docs/getting_started.md

@@ -57,6 +57,8 @@ For convenience, it is possible to build the required boost and eigen3 dependenc
 Other options are also
 available and are discussed in the [Building Acts](#building-acts) section.
 
+[Profiling](howto/profiling.md) details the prerequisites for profiling the ACTS project with gperftools.
+
 ## Building Acts
 
 Acts uses [CMake](https://cmake.org) to configure, build, and install the
@@ -243,7 +245,7 @@ components.
 | ACTS_BUILD_EVERYTHING               | Build with most options enabled (except HepMC3 and documentation)                                     |
 | ACTS_BUILD_PLUGIN_CUDA              | Build CUDA plugin                                                                                     |
 | ACTS_BUILD_PLUGIN_DD4HEP            | Build DD4hep geometry plugin                                                                          |
-| ACTS_BUILD_PLUGIN_EXATRKX           | Build Exa.TrkX plugin                                                                             |
+| ACTS_BUILD_PLUGIN_EXATRKX           | Build Exa.TrkX plugin                                                                                 |
 | ACTS_BUILD_PLUGIN_IDENTIFICATION    | Build Identification plugin                                                                           |
 | ACTS_BUILD_PLUGIN_JSON              | Build Json plugin                                                                                     |
 | ACTS_BUILD_PLUGIN_LEGACY            | Build legacy plugin                                                                                   |
@@ -274,7 +276,10 @@ components.
 | ACTS_USE_SYSTEM_EIGEN3              | Use the system eigen3 libraries (defaults to ON)                                                      |
 | ACTS_USE_SYSTEM_VECMEM              | Use system provided vecmem installation                                                               |
 | ACTS_USE_SYSTEM_PYBIND11            | Use pybind11 installed in the system                                                                  |
-| ACTS_USE_SYSTEM_ACTSDD4HEP          | Use ActsDD4hep glue library externally (and don't include it in the build)                            |
+| ACTS_USE_SYSTEM_ACTSDD4HEP          | Use ActsDD4hep glue library externally (and don't include it in the build)                            |                            
+| ACTS_ENABLE_CPU_PROFILING           | Link the profiler library to enable gperftool's CPU profiler                                          |
+| ACTS_ENABLE_MEMORY_PROFILING        | Link the tcmalloc library to enable gperftool's memory profiler and heap checker                      |
+| GPERF_INSTALL_DIR                   | Path to the directory that gperftools is installed in                                                 |
 
 All Acts-specific options are disabled or empty by default and must be
 specifically requested. Some of the options have interdependencies that are

docs/howto/profiling.md

@@ -0,0 +1,162 @@
+# Profiling
+
+Software profiling allows you to inspect the performance of a piece of software, seeing where the bottlenecks are and how they can be improved.
+gperftools is a software profiling package. It contains a CPU profiler, thread-caching malloc library, memory leak detection tool, memory allocation profiler and pprof (discussed later). More information about gperftools and its components can be found on the project's [GitHub](https://github.com/gperftools/gperftools) and [documentation page](https://gperftools.github.io/gperftools/).
+
+## Install gperftools
+
+It is strongly recommended to install [libunwind](http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz) before trying to configure or install gperftools.
+
+### Ubuntu
+
+If you're using Ubuntu you can use the following command to install gperftools:
+
+```
+apt install google-perftools libgoogle-perftools-dev
+```
+
+### Other Systems
+
+Alternatively, you can use the following commands to install it:
+
+```
+git clone https://github.com/gperftools/gperftools
+cd gperftools
+git tag -l # checkout the latest release version
+git checkout <gperftools-X.x>
+./autogen.sh
+./configure --prefix=<your/desired/install/dir>
+make
+make install
+```
+
+This will install gperftools in `your/desired/install/dir/lib` which is the path you should use when specifying where gperftools is, if necessary.
+
+If you wish to install gperftools to a directory that is not one of the standard directories for libraries and therefore not findable by the `-l` compiler flag, you will need to specify the path to it with the `GPERF_INSTALL_DIR` option at build time.
+Further information about installing gperftools is [here](https://github.com/gperftools/gperftools/blob/master/INSTALL).
+
+## pprof
+
+pprof is a tool for visualising and analysing profiling data.
+An older version of pprof comes bundled with gperftools but using the newer Go version comes with several benefits: nicer looking graphs and additional options that make looking through specific sections of a program easier being among them.
+
+### Install Go pprof (Optional)
+
+First, you must install Go. Instructions to do so are available [here](https://go.dev/doc/install).
+Optionally, you can install [Graphviz](http://www.graphviz.org/download/) to produce visualisations of profiles.
+
+Then, run the following command to install pprof itself:
+
+```
+go install github.com/google/pprof@latest
+```
+
+## Link gperftools Libraries When Compiling
+
+The library needed to run the CPU profilier should be linked into the ACTS project using the following build option:
+
+```
+-DACTS_ENABLE_CPU_PROFILING=ON
+```
+
+Similarly, to enable the memory profiler the following build option should be used:
+
+```
+-DACTS_ENABLE_MEMORY_PROFILING=ON
+```
+
+## Alternative to Recompiling
+
+Alternatively, you can avoid rebuiding the project by pointing the `LD_PRELOAD` environment variable to the profiler library for CPU profiling:
+
+```
+LD_PRELOAD="<path/to/libprofiler.so>" <other_options> <path/to/binary> <binary_flags>
+```
+
+You can do the same thing with the tcmalloc library for memory profiling:
+
+```
+LD_PRELOAD="<path/to/libtcmalloc.so>" <other_options> <path/to/binary> <binary_flags>
+```
+
+Using the `LD_PRELOAD` method is not recommended by the developers of gperftools so using the build options is preferable. Both CPU and memory profiling can be enabled at the same time but note that turning on memory profiling (or the heap checker) will affect performance.
+Specify multiple libraries to load with `LD_PRELOAD` using a space-separated list e.g.
+
+```
+LD_PRELOAD="<path/to/first/library> <path/to/second/library>"
+```
+
+Note that these steps don't turn on profiling, they only enable it to work. The following section details how to turn it on.
+
+## Produce a CPU Profile
+
+To turn on CPU profiling when running an executable define the `CPUPROFILE` environment variable when executing the program:
+
+```
+CPUPROFILE=<path/to/profile> <path/to/binary> [binary args]
+```
+
+This variable specifies where the profile will be written to.
+There are additional environemnt variables that modify the behaviour of the profiler.
+[Would you like to know more](https://github.com/gperftools/gperftools)?
+
+## Produce a Memory Profile
+
+To turn on memory profiling use the following command:
+
+```
+HEAPPROFILE=<path/to/profile> <path/to/binary> [binary args]
+```
+
+## Run the Heap Checker
+
+To run the heap checker for checking for memory leaks run the following command:
+
+```
+PPROF_PATH=<path/to/pprof> HEAPCHECK=normal <path/to/binary> [binary args]
+```
+
+The CPU profiler, memory profiler and heap checker can be used in tandem.
+
+## Using pprof
+
+### View Profile as a Graph
+
+A graphical representaion of a profile can be produced using:
+
+```
+pprof -pdf <path/to/binary> <path/to/profile> > <path/to/pdf>
+```
+
+Where `path/to/binary` is the binary is used to produce the profile in the first place.
+Other output formats are available.
+
+The following opens the graph in your web browser:
+
+```
+pprof -web <path/to/binary> <path/to/profile>
+```
+
+### Interactive Mode
+
+To launch pprof in interactive mode use the following command:
+
+```
+pprof <path/to/binary> <path/to/profile>
+```
+
+The following command will display the top x entries by the current sorting criteria:
+
+```
+top <number>
+```
+
+To view the statistics of a function line by line use:
+
+```
+list <nameOfFunction>
+```
+
+Various options can be specified to filter, sort and set the granularity of entries.
+There are also a number of other commands available.
+Read more about pprof [here](https://github.com/google/pprof).