Easier flamegraph / profiling support for datafusion benchmarks

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to make flamegraphs to understand performance of PRs like #2146 from @yjshen and other benchmarks in a cross platform, easy to use way. I don't want to switch between Instruments on Mac to perf on linux

It is not easy to get them (see this mailing list topic, and I hit the same "flamegraph takes forever to make" issue when I tried to run this on my development machine

Describe the solution you'd like
Use code in pprof crate -- @mkmik did this in influxdb_iox have used it to good effect in IOx

So this would look like adding an optional feature pprof to benchmark program that would generate flamegraphs and profile.proto format output. You would run it like this:

# --profile also writes flamegraph.svg and profile.proto files
target/release/tpch benchmark datafusion --profile --iterations 3 --path /tpch-parquet --format parquet --query 1

You could use the profile.proto in the (excellent) golang tooling like:

go tool pprof cpu.prof

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[tustvold]

Have you checked out cargo-flamegraph? I think it might fit the bill?

[alamb]

I did try to run cargo flamegraph ... and it was spending an absurd amount of time calling addr2line (as reported in the mailing list topic). Perhaps due to some old version of prof or something on the GCP debian machine I was using

[Dandandan]

FWIW, I never have experienced the in the mailinglist mentioned slowness of flamegraph (or the Hotspot UI I mentioned there), using a recent Pop!_OS version.

[tustvold]

Perhaps this might help? FWIW I also use the hotspot UI, which also has some nice functionality for digging into the profiles, viewing thread activity, etc... It chugs a bit on larger profiles (multiple GB) but otherwise works well enough for my purposes...

Edit: turns out cargo-flamegraph is already using inferno. Unfortunately pprof also uses it, so switching to using pprof may not help matters...

[bobtins]

@alamb someone drilled deep into why perf is slow.
TL;DR--GPLv2-license-compatible perf is really inefficient.
@Dandandan perf can be compiled with libbfd (fast) or without (slow). I am running Ubuntu but the author above only reports slow results on Debian. Maybe Pop!_OS is cheating and compiling with libbfd?
Summing up these results:

• me, Ubuntu 20.04: slow
• me, recompiled perf: fast
• @Dandandan , Pop!_OS: fast
• @alamb , MacOS: slow

@alamb applying the solution from IOx sounds like a great way to go. I haven't been on this for awhile (busy) but I probably would've taken a more laborious approach, perhaps reinventing a wheel.

[tustvold]

@bobtins I think you maybe meant to link to this https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement ?

The good news is a mitigation was merged at some point last year, and has apparently been released, so perhaps the GCP machines was running an old Debian version? - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=be8ecc57f180415e8a7c1cc5620c5236be2a7e56

[bobtins]

@tustvold I did try out hotspot; it's really nice! It can record data (which just runs perf) or open perf files that were recorded.

[bobtins]

@bobtins I think you maybe meant to link to this https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement ?

Hm, good news, thanks for the updated info.

[realno]

This is great, looking forward to seeing some results!

[alamb]

Thanks for the suggestions -- I'l try the various tools on this thread and will report back

[mkmik]

FWIW I found the pprof approach to be very useful for online performance analysis (e.g on demand on production services), and less useful when running ad-hoc performance analysis runs locally (e.g. benchmarks)

[houqp]

@alamb have you tried cargo-flamegraph's --no-inline argument?

[mingmwang]

I had use pprof-rs to benchmark the DataFusion/Arrow Parquet reader performance. It can generate the flamegraph pictures easily.

[alamb]

I don't think there is any work action for this, standard profiling tools work great. Closing