Profile Guided Optimizations

[Ekleog-NEAR]

We should try to compile neard with PGO and see what gains there are to be had.

It might be very significant, just like it could be less than expected.

I’d still hope for a ~10% perf win, but it could be either more or less.

[Ekleog-NEAR]

Experiment report

TL;DR

PGO is not useful right now. The LTO we already have enabled seems to not bring much benefits either actually. This is probably due to IO being a severe bottleneck in our current system, and thus should be revisited once we land memtrie.

Summary

This experiment report is the last in a series of experiments I performed, trying to see how much speedup PGO could actually get us.

Unfortunately, trying to just enable PGO hits rust-lang/rust#115344. So we need to make a choice between LTO and PGO. Thin LTO does not work either.

It turns out that, at the time of my writing this report, PGO actually does not get us anything — it’s even slightly slower than non-PGO’d binaries, even though in my test the workload is exactly what the profile was built on, so it should showcase the maximal benefits.

My assumption is this is due to the fact that most of our time is actually spent in IO, and not computation. In particular, this means that optimizations from the compiler do not have a significant impact. As a consequence, while this experiment points towards not enabling PGO for now, we should revisit it once we have landed memtrie and IO is likely less of a bottleneck. At this time we can reuse the exact instructions described here to evaluate the impact.

Incidentally, this experiment also led to me figuring out that LTO seems to only get us roughly 1% of performance on my benchmark. So it doesn’t actually seem to get us much, which also confirms the hypothesis that most of the time is currently spent doing IO.

Setup

• On a machine provisioned by the tf-near-node action, setup as a validator with 200G root disk space and 3T data disk space
• Having caught up the chain state to block 116114510, then stopped the neard service
• With a nearcore checkout at commit 2c13078

Compiling the baseline

$ cargo clean
$ cargo build -p neard --release --target x86_64-unknown-linux-gnu
$ mv target/x86_64-unknown-linux-gnu/release/neard ../neard-base

Compiling the binary that generates profile data

As per the introductory paragraph, we need to change lto = "fat" to lto = false in Cargo.toml’s [profile.release] section here. It seems that just adding another RUSTFLAGS entry does not work: sysinfo then fails to build with error: lto can only be run for executables, cdylibs and static library outputs.

$ cargo clean
$ RUSTFLAGS="-C profile-generate=/tmp/pgo-data" cargo build -p neard --release --target x86_64-unknown-linux-gnu
$ mv target/x86_64-unknown-linux-gnu/release/neard ../neard-instrumented-for-pgo

Generating the profile data

$ ./neard-base flat-storage move-flat-head --shard-id 2 --version 3 back --blocks 100
$ ./neard-instrumented-for-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
$ ~/.rustup/toolchains/1.77.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata merge -o /tmp/pgo-data/merged.profdata /tmp/pgo-data

Compiling the PGO’d binary

$ cargo clean
$ RUSTFLAGS="-C profile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" cargo build -p neard --release --target x86_64-unknown-linux-gnu
$ # Observe there was no warning message, confirming that the PGO build succeeded
$ mv target/x86_64-unknown-linux-gnu/release/neard ../neard-built-with-pgo

Evaluating the results

$ hyperfine --warmup 2 --runs 3 -L version built-with-pgo,base,built-with-pgo,base --prepare './n
eard-{version} flat-storage move-flat-head --shard-id 2 --version 3 back --blocks 100' './neard-{version} view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking'
[...]
Summary
  ./neard-base view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking ran
    1.00 ± 0.01 times faster than ./neard-base view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
    1.01 ± 0.01 times faster than ./neard-built-with-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
    1.01 ± 0.01 times faster than ./neard-built-with-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking

So PGO seems to have basically no significant impact on performance.

Evaluating the impact of LTO removal

After compiling a neard with LTO disabled:

$ hyperfine --warmup 2 --runs 3 -L version built-with-pgo,base,without-lto,built-with-pgo,base,without-lto --prepare './neard-{version} flat-storage move-flat-head --shard-id 2 --version 3 back --blocks 100' './neard-{version} view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking'
Benchmark 1: ./neard-built-with-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
  Time (mean ± σ):      5.081 s ±  0.014 s    [User: 8.144 s, System: 2.416 s]
  Range (min … max):    5.065 s …  5.093 s    3 runs
 
Benchmark 2: ./neard-base view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
  Time (mean ± σ):      5.025 s ±  0.030 s    [User: 7.914 s, System: 2.375 s]
  Range (min … max):    4.992 s …  5.050 s    3 runs
 
Benchmark 3: ./neard-without-lto view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
  Time (mean ± σ):      5.052 s ±  0.022 s    [User: 8.089 s, System: 2.491 s]
  Range (min … max):    5.027 s …  5.069 s    3 runs
 
Benchmark 4: ./neard-built-with-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
  Time (mean ± σ):      5.111 s ±  0.086 s    [User: 7.938 s, System: 2.371 s]
  Range (min … max):    5.053 s …  5.210 s    3 runs
 
Benchmark 5: ./neard-base view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
  Time (mean ± σ):      5.068 s ±  0.035 s    [User: 8.113 s, System: 2.512 s]
  Range (min … max):    5.027 s …  5.091 s    3 runs
 
Benchmark 6: ./neard-without-lto view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
  Time (mean ± σ):      5.085 s ±  0.052 s    [User: 8.091 s, System: 2.378 s]
  Range (min … max):    5.029 s …  5.132 s    3 runs
 
Summary
  ./neard-base view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking ran
    1.01 ± 0.01 times faster than ./neard-without-lto view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
    1.01 ± 0.01 times faster than ./neard-base view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
    1.01 ± 0.01 times faster than ./neard-built-with-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
    1.01 ± 0.01 times faster than ./neard-without-lto view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking
    1.02 ± 0.02 times faster than ./neard-built-with-pgo view-state --readwrite apply-range --shard-id 2 --use-flat-storage benchmarking

This test was originally performed in order to figure out how big the benefits we could have once rust-lang/rust#115344 is fixed would be. However, it looks like neither LTO nor PGO actually bring much benefits currently.