feat(bpf): use time window for bpf sampling to replace per call based sampling

[rootfs]

From @vimalk78 finding, the per call based bpf sampling has very large cpu time variations.

Now changing to time window based sampling. The cpu time is much consistent and close to the probing results, while the overhead is reduced even more.

Disclaimer: some of the code is generated by ChatGPT.

Active Time (ms)	Idle Time (ms)	Average kepler_sched_switch_trace bpf runtime (ns)
5	95	400
20	80	875
50	50	1500
80	20	2100
1000 (default)	0	2500

[github-actions]

🤖 SeineSailor

Here is a concise summary of the pull request changes:

Summary: This pull request introduces significant changes to the BPF (Berkeley Packet Filter) implementation, replacing per-call sampling with time window-based sampling. This update aims to reduce CPU time variation and overhead. Additionally, a minor internal change is made to the dcgm.Init() function call.

Key Modifications:

1. Time window-based sampling for BPF: The new approach reduces CPU time variation and overhead by introducing global parameters for tracking and non-tracking periods, along with two BPF maps to manage the tracking state.
2. Updated do_kepler_sched_switch_trace function: This function now checks a tracking flag and updates the sampling state based on elapsed time.
3. Internal change to dcgm.Init() function call: The function now uses config.GetDCGMHostEngineEndpoint() instead of config.DCGMHostEngineEndpoint.

Impact on Codebase:

• The external interface remains unchanged, but the implementation of sampling and tracking has been significantly altered.
• The code generated by ChatGPT may require further review.

Observations and Suggestions:

• It would be beneficial to include additional testing to ensure the new time window-based sampling approach does not introduce any regressions or performance issues.
• Consider adding more detailed comments or documentation to explain the reasoning behind the changes and how they impact the codebase.
• Review the code generated by ChatGPT to ensure it meets the project's coding standards and best practices.

[rootfs]

converting to draft, pending test results.

[vimalk78]

Test results:

Below is a comparison of two keplers, one with sampling window enabled (100 ms active, 1000 ms idle), other without sampling.

We can see that on bare metal, the two keplers produce very close values for package power and core power, because the ratio of bpf cpu time, with sampling, is very close to without sampling.

process cpu time, exhaustive vs sampling

process core joules, exhaustive vs sampling

process package joules, exhaustive vs sampling

kepler cpu time, exhaustive vs sampling

As expected, the kepler with sampling consumes less cpu time, and less cpu instructions compared to kepler without sampling.

[rootfs]

@dave-tucker @sthaha @marceloamaral PTAL, thanks

[marceloamaral]

@rootfs @dave-tucker I’m concerned about the impact on VM power estimation. If we're undercounting CPU time, the power consumption will be underestimated as well.

To address this, we need to extrapolate the results, similar to how Linux handles counter multiplexing. For instance, if we collected data for only 1 second out of a 5-second window, we should multiply the results by 5 to estimate for the full 5 seconds.

All we need to do is track the collection interval and adjust the results accordingly.

[dave-tucker]

Some comments on the code.

I'm not going to repeat myself, so here is my canned response: #1685 (review)

@vimalk78 I'm not surprised the totals or the estimation are pretty much the same.
But can you run the same test again but show the distribution of CPU time for each process on the system?

My bet is we will find that per-process cpu time, cpu instructions etc.. are totally off.
Comparing to top or schapandre etc.. would yield very different results.

Given we only care about cpu time (at the moment), the most efficient option is to not use eBPF at all and just read utime from /proc/$pid/stat

[dave-tucker]

consider using a PERCPU_ARRAY since this would have the effect of making the time window per-cpu also, which may be desirable.

[rootfs]

changed to PERCPU_ARRAY

[dave-tucker]

given that map lookups are the most expensive part of the eBPF code it would be better to reduce them where possible. there's no reason to store tracking_flag in a map as far as I can tell since it's value doesn't need to persist between invocations.

[rootfs]

there is a thinking of using kepler userspace program to set the tracking flag. The actual mechanism is not quite clear yet. Will remove this map if that is a dead end.

[dave-tucker]

Why are the default values in the code 20 and 80, but here they are 1000 and 0?
If this is for coexistence with the other sampling feature it may be easier to set them all to 0 and update the eBPF code to only evaluate this code path if both ACTIVE and IDLE values are > 0.

[rootfs]

fixed

[rootfs]

@marceloamaral good point! at the moment, the sampled cpu time is not extrapolated. We can consider different scaling factors. One approach in my plan is to find the max and min cpu time from each sample, and use the mean cpu time to extrapolate the entire active + idle duration. This will account for the variable cpu utilization conditions. If that proves effective, we then will discuss removing the EXPERIMENTAL prefix from these params. wdyt?