Failed to load deltas trying to profile ClickHouse binary

[danipozo]

When using the agent to profile on a system running ClickHouse, the agent is incapable of providing user-level stacks for the ClickHouse process.

Error logs

time="2024-11-08T10:33:25Z" level=error msg="Failed to handle mapping for PID 3105389, file /usr/bin/clickhouse: failed to load deltas: failed UpdateExeIDToStackDeltas for FileID 615f880a881cd2e0: no map available for 2791507 stack deltas"

How to reproduce

Run the server:

curl -sL https://github.com/ClickHouse/ClickHouse/releases/download/v24.10.1.2812-stable/clickhouse-common-static-24.10.1.2812-amd64.tgz | tar xzf - clickhouse-common-static-24.10.1.2812/usr/bin/clickhouse
mkdir ch-data && cd ch-data
../clickhouse-common-static-24.10.1.2812/usr/bin/clickhouse server

Then run the agent.

[rockdaboot]

The clickhouse binary is too large (max stack deltas hard-coded to 2^21 = 2097152).
The STACK_DELTA_BUCKET_LARGEST needs to be increased

support/ebpf/types.h
835:#define STACK_DELTA_BUCKET_LARGEST 21

This needs some code changes as well.

[florianl]

Hi @danipozo

Can you provide more information on how you visualize the stacks?
Do you use devfiler? - see https://github.com/open-telemetry/opentelemetry-ebpf-profiler?tab=readme-ov-file#visualizing-data-locally

If you are using devfiler then you can drop executables into the Executables tab to get them symbolized. Unlike for interpreted languages, there are no symbols by default for native languages, like C and C++. And as ClickHouse is written in C/C++.

But also, @rockdaboot points out another issue with the size of the executable in #230 (comment).

[danipozo]

Hi @florianl

Can you provide more information on how you visualize the stacks?

I'm using Parca for visualization. I have first used their parca-agent, which is now based on this one, and then reproduced the same failure with this agent on main.

If you are using devfiler then you can drop executables into the Executables tab to get them symbolized.

I have tried to do this but haven't been able to get the drag-and-drop behavior working locally.

Unlike for interpreted languages, there are no symbols by default for native languages, like C and C++.

Shouldn't the agent get them from the debug info of the executable?

But also, @rockdaboot points out another issue with the size of the executable in #230 (comment).

@rockdaboot is this a limitation that can be overcome? How hard/easy could it be?

[florianl]

Shouldn't the agent get them from the debug info of the executable?

For native languages, like C, C++, Rust and Go, debug info is not necessary to unwind the stack. For these native languages it is the task for the visualization to provide symbol information. I can not speak for Parca - so maybe someone else can help here further?

If you are using devfiler then you can drop executables into the Executables tab to get them symbolized.

I have tried to do this but haven't been able to get the drag-and-drop behavior working locally.

I dropped the clickhouse executable, that you have provided in #230 (comment), via drag & drop into devfilers Executable tab and get it symbolized.

But the point, @rockdaboot mentioned in #230 (comment), remains. So without further changes, it might not work out of the box.

[florianl]

With #231 applied and drag&drop the clickhouse executable into devfiler for symbolization, I do get stacks with symbol information for the provided clickhouse executable.

[danipozo]

Hey, thanks, that was fast!