Dynamic Integration of bpftrace Scripts into Kernel Collector

[cforce]

Is your feature request related to a problem? Please describe.

• Users can define custom telemetry logic based on their specific requirements.
• Dynamic loading allows for runtime changes without restarting the kernel collector.
• Encourages the community to share and contribute bpftrace scripts tailored for different use cases. See examples: https://github.com/iovisor/bpftrace#tools

Describe the solution you'd like

Enable users to dynamically integrate and execute bpftrace scripts as part of the OpenTelemetry-eBPF kernel collector. This feature aims to provide a flexible and extensible way for users to define custom tracing logic using bpftrace scripts, allowing them to gather specific telemetry data tailored to their application's needs.

Introduce a new plugin mechanism within the OpenTelemetry-eBPF collector that allows users to specify and load bpftrace scripts dynamically. The integration can be achieved through the following steps:

Script Loading:

Extend the kernel collector to recognize a designated directory or configuration file where users can place their bpftrace scripts.
Compilation and Execution:

Implement a compilation step within the kernel collector that compiles the bpftrace scripts into eBPF bytecode, leveraging the existing bpftrace project for this purpose.
Develop a mechanism to dynamically load and execute the compiled eBPF bytecode within the kernel collector.
Configuration:

Introduce configuration options for users to specify the directory or file path containing their bpftrace scripts.
Allow users to enable or disable specific scripts based on their requirements.
Lifecycle Management:

Implement mechanisms for script lifecycle management, including reloading scripts without restarting the kernel collector.
Example Usage

# Set environment variable for bpftrace script directory
export BPFTRACE_SCRIPT_DIR=/path/to/bpftrace/scripts

# Run kernel collector with bpftrace integration
sudo kernel-collector --log-console --enable-bpftrace

# Example bpftrace script: trace disk I/O latency
# /path/to/bpftrace/scripts/io_latency.bt
bpftrace -e 'tracepoint:block:block_rq_issue { printf("%d %s %d\n", pid, comm, args->bytes); }'

Describe alternatives you've considered

No response

Additional context

Security: Implement security measures to ensure that only authorized scripts are executed.
Performance: Evaluate the performance impact of dynamically loaded bpftrace scripts on the kernel collector.

[yonch]

👍
How would you envision processing the messages that are generated?

• I think that it might be possible to generate bpftrace code to serialize messages so scripts can encode in the right format.
• The collector's message handling code might be configured to forward some messages to the reducer without further processing

Not sure how the new messages might be processed in the reducer, do you have a plan for that? Would you run processing bpftrace scripts in the reducer?

[cforce]

The kernel collector is currently collecting a fixed set of metrics, which i would envision to allow by a new module to allow dynamic script based subscription for such collection but as you said, still use the existing reducer as consuming component for this collection afterwards so finally the otel collector can process and distribute remotely.