Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog post on Elastic's donation of the eBPF profiling agent #4613

Merged
merged 20 commits into from
Jun 7, 2024
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions content/en/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,16 @@ developer_note:
show_banner: true
limit_banner: 2 # Default to 2, adjust as needed
banners:
- message:
'<i class="fas fa-bullhorn"></i> We are happy to announce that Elastic''s
donation of the [**eBPF profiling agent has been
accepted**](/blog/2024/elastic-contributes-continuous-profiling-agent)!'
to: 2024-07-25
- message:
'<i class="fas fa-bullhorn"></i> Join us for [**OTel Community Day** on
June
25th](https://events.linuxfoundation.org/open-telemetry-community-day/)!'
to: 2024-06-25
- message:
'<i class="fas fa-bullhorn"></i> Help improve OTel and Prometheus
interoperability: [complete our
survey](https://forms.gle/bZAG9f7udoJsjZUG9) by May 31'
to: 2024-05-31
---

<div class="d-none"><a rel="me" href="https://fosstodon.org/@opentelemetry"></a></div>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
---
title: Elastic Contributes its Continuous Profiling Agent to OpenTelemetry
svrnm marked this conversation as resolved.
Show resolved Hide resolved
linkTitle: Elastic Contributes Profiling Agent # Mandatory, make sure that your short title.
date: 2024-06-06 # Put the current date, we will keep the date updated until your PR is merged
AlexanderWert marked this conversation as resolved.
Show resolved Hide resolved
# prettier-ignore
cSpell:ignore: Bahubali Christos Dmitry Filimonov Geisendörfer Halliday Kalkanis Shetti
author:
AlexanderWert marked this conversation as resolved.
Show resolved Hide resolved
>- # If you have only one author, then add the single name on this line in quotes.
[Bahubali Shetti](https://github.com/bshetti) (Elastic), [Alexander
Wert](https://github.com/AlexanderWert) (Elastic), [Morgan
McLean](https://github.com/mtwo) (Splunk), [Ryan
Perry](https://github.com/Rperry2174) (Grafana)
issue: https://github.com/open-telemetry/community/issues/1918
sig: Profiling SIG
---

Following significant collaboration between
[Elastic](https://www.elastic.co/observability-labs/blog/elastic-donation-proposal-to-contribute-profiling-agent-to-opentelemetry)
and [OpenTelemetry's profiling community](/blog/2024/profiling/), which included
a thorough review process, we’re excited to announce that the OpenTelemetry
project has accepted
[Elastic's donation of its continuous profiling agent](https://github.com/open-telemetry/community/issues/1918).

This marks a significant milestone in establishing profiling as a core telemetry
signal in OpenTelemetry. Elastic’s [eBPF based](https://ebpf.io/) profiling
agent observes code across different programming languages and runtimes,
third-party libraries, kernel operations, and system resources with low CPU and
memory overhead in production. Both, SREs and developers can now benefit from
these capabilities: quickly identifying performance bottlenecks, maximizing
resource utilization, reducing carbon footprint, and optimizing cloud spend.

Elastic’s decision to contribute the project to OpenTelemetry was made to
accelerate OpenTelemetry’s mission and enable effective observability through
high-quality, portable telemetry. This collaboration also shows the commitment
to vendor neutrality and community-driven development enhancing the overall
profiling and observability ecosystems.

The donation happened through a great and constructive cooperation between
Elastic and the OpenTelemetry community. We look forward to jointly establishing
continuous profiling as an integral part of OpenTelemetry.

With today’s acceptance, Elastic’s continuous profiling agent will be
contributed to OpenTelemetry. This agent will now be jointly supported by both
Elastic’s team as well as a diverse set of official maintainers from different
companies:

- Dmitry Filimonov (Grafana Labs)
- Felix Geisendörfer (Datadog)
- Jonathan Halliday (Red Hat)
- Christos Kalkanis (Elastic)

## What is continuous profiling?

[Continuous profiling](https://www.cncf.io/blog/2022/05/31/what-is-continuous-profiling/)
is a technique used to understand the behavior of a software application by
collecting information about its execution over time. This includes tracking the
duration of function calls, memory usage, CPU usage, and other system resources
along with associated metadata.

## Benefits of Continuous Profiling

Traditional profiling solutions, typically used for one-off, development time
optimizations, can have significant drawbacks limiting adoption in production
environments:

- Significant cost and performance overhead due to code instrumentation
- Disruptive service restarts
- Inability to get visibility into third-party libraries

Continuous profiling, however, runs in the background with minimal overhead,
providing real-time, actionable insights without the need to replicate issues in
separate environments.

This allows SREs, DevOps, and developers to see how code affects performance and
cost, making code and infrastructure improvements easier.

## Contribution of comprehensive profiling abilities

The continuous profiling agent, that Elastic is donating, is
[based on eBPF](https://ebpf.io/) and by that a whole system, always-on solution
that observes code and third-party libraries, kernel operations, and other code
you don't own. It eliminates the need for code instrumentation
(run-time/bytecode), recompilation, or service restarts with low overhead, low
CPU (~1%), and memory usage in production environments.

The donated profiling agent facilitates identifying non-optimal code paths,
uncovering "unknown unknowns", and provides comprehensive visibility into the
runtime behavior of all applications. The continuous profiling agent provides
support for a wide range of runtimes and languages, such as:

- C/C++
- Rust
- Zig
- Go
- Java
- Python
- Ruby
- PHP
- Node.js / V8
- Perl
- .NET

## Benefits to OpenTelemetry

This contribution not only boosts the standardization of continuous profiling
for observability but also accelerates its adoption as a key signal in
OpenTelemetry. Customers benefit from a vendor-agnostic method of collecting
profiling data correlating it with existing signals, like tracing, metrics, and
logs, opening new potential for observability insights and a more efficient
troubleshooting experience.

### User Benefits of OpenTelemetry Profiling
AlexanderWert marked this conversation as resolved.
Show resolved Hide resolved

OpenTelemetry-based continuous profiling unlocks the following possibilities for
users:

- Continuous profiling data compliments the existing signals (traces, metrics
and logs) by providing detailed, code-level insights on the services'
behavior.

- Seamless correlation with other OpenTelemetry signals such as traces,
increasing fidelity and investigatory depth.

- Estimate environmental impact: Combining profiling data with OpenTelemetry's
resource information (i.e. resource attributes) allows to derive insights into
the services' carbon footprint.

- Through a detailed breakdown of services' resource utilization, profiling data
provides actionable information on performance optimization opportunities.

- Improved vendor neutrality: a vendor-agnostic eBPF-based profiling agent
removes the need to rely on proprietary agents to collect profiling telemetry.

With these benefits, SREs, developers, and DevOps, can now manage the overall
application’s efficiency on the cloud while ensuring their engineering teams
optimize it.

As the next step, the OpenTelemetry profiling SIG, that Elastic is a part of,
will jointly work on integrating the donated agent into OpenTelemetry's
components ecosystem. We look forward to providing a fully integrated and usable
version of the new OpenTelemetry eBPF profiling agent to the users, soon.
32 changes: 32 additions & 0 deletions static/refcache.json
Original file line number Diff line number Diff line change
Expand Up @@ -1959,6 +1959,10 @@
"StatusCode": 200,
"LastSeen": "2024-06-04T17:29:51.894512776+02:00"
},
"https://energy.mit.edu/news/energy-efficient-computing/": {
"StatusCode": 200,
"LastSeen": "2024-06-06T19:18:50.285562+02:00"
},
"https://equalitymi.org/": {
"StatusCode": 200,
"LastSeen": "2024-01-18T19:02:20.222589-05:00"
Expand Down Expand Up @@ -2171,6 +2175,10 @@
"StatusCode": 200,
"LastSeen": "2024-01-30T16:14:58.646823-05:00"
},
"https://github.com/AlexanderWert": {
"StatusCode": 200,
"LastSeen": "2024-06-06T19:18:46.235285+02:00"
},
"https://github.com/Aneurysm9": {
"StatusCode": 200,
"LastSeen": "2024-01-18T20:06:35.565079-05:00"
Expand Down Expand Up @@ -2287,6 +2295,10 @@
"StatusCode": 200,
"LastSeen": "2024-01-18T19:12:45.986279-05:00"
},
"https://github.com/Rperry2174": {
"StatusCode": 200,
"LastSeen": "2024-06-06T19:18:46.659794+02:00"
},
"https://github.com/Samudraneel24": {
"StatusCode": 200,
"LastSeen": "2024-01-30T05:18:13.282737-05:00"
Expand Down Expand Up @@ -2451,6 +2463,10 @@
"StatusCode": 200,
"LastSeen": "2024-01-30T15:25:06.510641-05:00"
},
"https://github.com/bshetti": {
"StatusCode": 200,
"LastSeen": "2024-06-06T19:18:45.889432+02:00"
},
"https://github.com/bufbuild/connect-opentelemetry-go": {
"StatusCode": 200,
"LastSeen": "2024-01-18T19:11:18.472599-05:00"
Expand Down Expand Up @@ -6175,10 +6191,18 @@
"StatusCode": 206,
"LastSeen": "2024-05-15T19:23:44.426379755+03:00"
},
"https://opentelemetry.io/blog/2024/profiling/": {
"StatusCode": 206,
"LastSeen": "2024-06-06T19:18:48.714842+02:00"
},
"https://opentelemetry.io/blog/2024/scaling-collectors/": {
"StatusCode": 206,
"LastSeen": "2024-05-06T07:53:28.903161-07:00"
},
"https://opentelemetry.io/community/mission/": {
"StatusCode": 206,
"LastSeen": "2024-06-06T19:18:50.53437+02:00"
},
"https://opentelemetry.io/docs/collector": {
"StatusCode": 206,
"LastSeen": "2024-02-23T22:55:03.656226-05:00"
Expand Down Expand Up @@ -8603,6 +8627,10 @@
"StatusCode": 206,
"LastSeen": "2024-01-18T20:05:14.383107-05:00"
},
"https://www.cncf.io/blog/2022/05/31/what-is-continuous-profiling/": {
"StatusCode": 206,
"LastSeen": "2024-06-06T19:18:49.123716+02:00"
},
"https://www.cncf.io/blog/2023/01/11/a-look-at-the-2022-velocity-of-cncf-linux-foundation-and-top-30-open-source-projects/": {
"StatusCode": 206,
"LastSeen": "2024-01-30T05:18:24.717687-05:00"
Expand Down Expand Up @@ -8683,6 +8711,10 @@
"StatusCode": 206,
"LastSeen": "2024-01-30T06:06:13.672817-05:00"
},
"https://www.elastic.co/observability-labs/blog/elastic-donation-proposal-to-contribute-profiling-agent-to-opentelemetry": {
"StatusCode": 206,
"LastSeen": "2024-06-06T19:18:47.94157+02:00"
},
"https://www.envoyproxy.io": {
"StatusCode": 206,
"LastSeen": "2024-01-18T19:56:01.373375-05:00"
Expand Down