Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate HTTP trace context in Go #491

Merged
merged 10 commits into from
Dec 11, 2023

Conversation

grcevski
Copy link
Contributor

@grcevski grcevski commented Dec 6, 2023

This PR effectively finishes the work to propagate the trace context for HTTP Go applications (no gRPC), by writing the discovered trace id and context id in the HTTP headers. Few details on how this works:

  • If there's a 'Traceparent' header already found in the headers of a client request we don't do anything.
  • We only activate the BPF side of the code if the kernel is not in lockdown mode or it's older than 5.14 (where the bpf_probe_write_user was banned).
  • This PR also started processing the flags field from incoming headers, so that we respect the sampling wishes of other SDKs.

TODO:

  • Add integration tests. The integration tests will use the same logic for checking if the kernel allows bpf_probe_write_user and only execute if that's allowed. Otherwise these new tests will be skipped, e.g. setups with Secure Boot ON.
  • In another PR: once we determine a new span id for the Beyla generated Traceparent, we should overwrite the incoming server request headers with this span id, so that any traces made using the standard Go SDK embed themselves as children of the new eBPF span.

@grcevski grcevski requested a review from mariomac as a code owner December 6, 2023 22:47
@codecov-commenter
Copy link

codecov-commenter commented Dec 6, 2023

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (fd1de8d) 77.25% compared to head (d57d21c) 66.23%.
Report is 4 commits behind head on main.

Files Patch % Lines
pkg/internal/ebpf/nethttp/nethttp.go 87.09% 3 Missing and 1 partial ⚠️
pkg/internal/ebpf/common/spanner.go 50.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #491       +/-   ##
===========================================
- Coverage   77.25%   66.23%   -11.02%     
===========================================
  Files          67       64        -3     
  Lines        5302     5228       -74     
===========================================
- Hits         4096     3463      -633     
- Misses        981     1489      +508     
- Partials      225      276       +51     
Flag Coverage Δ
integration-test 66.23% <86.11%> (+0.23%) ⬆️
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@mariomac mariomac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing!!

@@ -61,6 +61,8 @@ int uprobe_ServeHTTP(struct pt_regs *ctx) {

if (req) {
server_trace_parent(goroutine_addr, &invocation.tp, (void*)(req + req_header_ptr_pos));
// TODO: if context propagation is supported, overwrite the header value in the map with the
// new span context and the same thread id.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if we don't do it now? Is the feature incomplete?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see it as incomplete in terms of Beyla instrumentation, but it would be nice to make Beyla work well for instrumented applications with the Go SDK. As it stands now, Beyla will correctly propagate the context, but if there's SDK instrumentation as well, the two spans will look parallel, just as it was before. If we overwrite the header value, we'll manage to nest them. So it's an extension to the feature to make it work with auto and manual instrumentation.

@@ -86,6 +86,7 @@ services:
- --config=/configs/beyla-config.yml
volumes:
- ./configs/:/configs
- ./system/sys/kernel/security:/sys/kernel/security
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will we have to document this e.g. to allow running Beyla in Kubernetes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh this is a very good point. I need to document this. As it stands now, if security isn't there we'll assume we can propagate the context, since the security file is missing. But it could be because the users didn't mount it and the host machine doesn't allow it. I'll follow-up with a PR on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened an issue so I don't forget #502

Comment on lines +34 to +37
//go:generate $BPF2GO -cc $BPF_CLANG -cflags $BPF_CFLAGS -target amd64,arm64 bpf ../../../../bpf/go_nethttp.c -- -I../../../../bpf/headers -DNO_HEADER_PROPAGATION
//go:generate $BPF2GO -cc $BPF_CLANG -cflags $BPF_CFLAGS -target amd64,arm64 bpf_debug ../../../../bpf/go_nethttp.c -- -I../../../../bpf/headers -DBPF_DEBUG -DNO_HEADER_PROPAGATION
//go:generate $BPF2GO -cc $BPF_CLANG -cflags $BPF_CFLAGS -target amd64,arm64 bpf_tp ../../../../bpf/go_nethttp.c -- -I../../../../bpf/headers
//go:generate $BPF2GO -cc $BPF_CLANG -cflags $BPF_CFLAGS -target amd64,arm64 bpf_tp_debug ../../../../bpf/go_nethttp.c -- -I../../../../bpf/headers -DBPF_DEBUG
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to grow exponentially 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, I've been thinking about this, but I can't find a good solution that allows us to support various configurations :(.

@grcevski grcevski merged commit b96e341 into grafana:main Dec 11, 2023
4 checks passed
@grcevski grcevski deleted the propagate_context_go branch December 11, 2023 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants