[system-probe] Add packet counting to existing UDP byte counters #28546

pimlu · 2024-08-19T19:48:23Z

What does this PR do?

This PR begins tracing UDP packet counts in the agent and adds some tests to ensure it works as expected for both IPv4 and IPv6.

Motivation

Context - NPM-2973. The agent already has tracked UDP byte counts so it was determined we can leverage the existing kprobes to track packet counts in the same places.

Additional Notes

Possible Drawbacks / Trade-offs

This does not track UDP sends that have corking or MSG_MORE which are not commonly used. Doing this requires tracing udp_send_skb which probably will have to wait until after we deprecate prebuilt as mentioned in this thread

Describe how to test/QA your changes

This PR leverages the existing test suite and adds some new ones:

DD_SYSTEM_PROBE_CONFIG_BPF_DEBUG=true sudo -E go test -v -tags "linux linux_bpf npm test" ./pkg/network/tracer --run TestTracerSuite

pr-commenter · 2024-08-19T20:23:05Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=42853416 --os-family=ubuntu

Note: This applies to commit b9a0ad6

pr-commenter · 2024-08-19T20:52:32Z

eBPF complexity changes

Summary result: ✅ - stable

Highest complexity change (%): +1.55%
Highest complexity change (abs.): +16 instructions
Programs that were above the 85.0% limit of instructions and are now below: 0
Programs that were below the 85.0% limit of instructions and are now above: 0

tracer details

tracer

Program	Avg. complexity	Distro with highest complexity	Distro with lowest complexity
kretprobe__ip_make_skb	🔴 1148.7 (+16.0, +1.41%)	fedora_38/arm64: 🔴 1303.0 (+16.0, +1.24%)	centos_8/arm64: 🔴 1048.0 (+16.0, +1.55%)
kretprobe__tcp_close_clean_protocols	⚪ 212.4 (+0.0, +0.00%)	centos_8/arm64: ⚪ 225.0 (+0.0, +0.00%)	debian_10/x86_64: ⚪ 199.0 (+0.0, +0.00%)
kretprobe__tcp_close_flush	⚪ 216.4 (+0.0, +0.00%)	ubuntu_18.04/x86_64: ⚪ 217.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 216.0 (+0.0, +0.00%)
kretprobe__tcp_done_flush	⚪ 216.4 (+0.0, +0.00%)	ubuntu_18.04/x86_64: ⚪ 217.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 216.0 (+0.0, +0.00%)
kretprobe__tcp_read_sock	⚪ 1036.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1128.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 990.0 (+0.0, +0.00%)
kretprobe__tcp_recvmsg	⚪ 1036.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1128.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 990.0 (+0.0, +0.00%)
kretprobe__tcp_retransmit_skb	⚪ 475.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 559.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 434.0 (+0.0, +0.00%)
kretprobe__tcp_sendmsg	⚪ 1036.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1128.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 990.0 (+0.0, +0.00%)
kretprobe__tcp_sendpage	⚪ 1036.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1128.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 990.0 (+0.0, +0.00%)
kretprobe__udp_destroy_sock	⚪ 217.4 (+0.0, +0.00%)	ubuntu_18.04/x86_64: ⚪ 218.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 217.0 (+0.0, +0.00%)
kretprobe__udp_recvmsg	⚪ 9.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 9.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 9.0 (+0.0, +0.00%)
kretprobe__udp_recvmsg_pre_4_7_0	⚪ 1386.0 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1612.0 (+0.0, +0.00%)	amazon_5.4/x86_64: ⚪ 1129.0 (+0.0, +0.00%)
kretprobe__udp_sendpage	🔴 894.0 (+8.0, +0.90%)	fedora_38/arm64: 🔴 965.0 (+8.0, +0.84%)	centos_8/arm64: 🔴 847.0 (+8.0, +0.95%)
kretprobe__udpv6_destroy_sock	⚪ 217.4 (+0.0, +0.00%)	ubuntu_18.04/x86_64: ⚪ 218.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 217.0 (+0.0, +0.00%)
kretprobe__udpv6_recvmsg	⚪ 9.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 9.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 9.0 (+0.0, +0.00%)
kretprobe__udpv6_recvmsg_pre_4_7_0	⚪ 1386.0 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1612.0 (+0.0, +0.00%)	amazon_5.4/x86_64: ⚪ 1129.0 (+0.0, +0.00%)
socket__classifier_dbs	⚪ 2427.0 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 2434.0 (+0.0, +0.00%)	amazon_5.4/x86_64: ⚪ 2420.0 (+0.0, +0.00%)
socket__classifier_entry	⚪ 2547.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 2562.0 (+0.0, +0.00%)	amazon_5.4/x86_64: ⚪ 2534.0 (+0.0, +0.00%)
socket__classifier_grpc	⚪ 6754.0 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 6775.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 6719.0 (+0.0, +0.00%)
socket__classifier_queues	⚪ 7023.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 8247.0 (+0.0, +0.00%)	amazon_5.4/x86_64: ⚪ 4617.0 (+0.0, +0.00%)
tracepoint__net__net_dev_queue	⚪ 971.7 (+0.0, +0.00%)	fedora_38/arm64: ⚪ 1183.0 (+0.0, +0.00%)	centos_8/arm64: ⚪ 800.0 (+0.0, +0.00%)

This report was generated based on the complexity data for the current branch HEAD (pipeline [https://gitlab.ddbuild.io/DataDog/datadog-agent/-/pipelines/42483018]42483018) and the main branch (commit fe13dea). Objects without changes are not reported. Contact #ebpf-platform if you have any questions/feedback.

Table complexity legend: 🔵 - new; ⚪ - unchanged; 🟢 - reduced; 🔴 - increased

akarpz · 2024-08-19T21:38:52Z

can you add a description? also it's helpful to put the ticket # in the title and/or under the motivation section

pkg/network/tracer/tracer_linux_test.go

pr-commenter · 2024-08-21T19:42:43Z

Regression Detector

Regression Detector Results

Run ID: f71704c0-7619-46c5-8e50-c37c9f99ecf3 Metrics dashboard Target profiles

Baseline: 207b216
Comparison: b9a0ad6

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	basic_py_check	% cpu utilization	+0.96	[-1.59, +3.51]	Logs
➖	otel_to_otel_logs	ingress throughput	+0.28	[-0.53, +1.09]	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.00, +0.00]	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	Logs
➖	file_tree	memory utilization	-0.21	[-0.27, -0.15]	Logs
➖	idle	memory utilization	-0.53	[-0.57, -0.50]	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.73	[-1.51, +0.05]	Logs
➖	pycheck_lots_of_tags	% cpu utilization	-0.82	[-3.18, +1.54]	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-4.32	[-16.70, +8.06]	Logs

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
✅	idle	memory_usage	10/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

pkg/network/tracer/tracer_linux_test.go

going to test ipv6 instead if it's not already

pkg/network/tracer/tracer_linux_test.go

hestonhoffman

Docs 👍

pimlu · 2024-09-05T17:42:03Z

/merge

dd-devflow · 2024-09-05T17:42:10Z

🚂 MergeQueue: pull request added to the queue

The median merge time in main is 23m.

Use /merge -c to cancel this operation!

pimlu requested a review from a team as a code owner August 19, 2024 19:48

github-actions bot added component/system-probe team/networks labels Aug 19, 2024

akarpz reviewed Aug 19, 2024

View reviewed changes

pkg/network/tracer/tracer_linux_test.go Outdated Show resolved Hide resolved

pimlu force-pushed the stuart.geipel/udp-counting branch from 9ba64e0 to ebe8f63 Compare August 22, 2024 14:54

hmahmood reviewed Aug 23, 2024

View reviewed changes

pkg/network/tracer/tracer_linux_test.go Outdated Show resolved Hide resolved

hmahmood reviewed Aug 23, 2024

View reviewed changes

pkg/network/tracer/tracer_linux_test.go Outdated Show resolved Hide resolved

pimlu added 9 commits August 26, 2024 11:25

Add packet counting to existing UDP byte counters, and more tests

7de69b7

add actual cork test (not MSG_MORE)

7325746

remove the cork tests

7b11557

going to test ipv6 instead if it's not already

add ipv6 test to peek count

87d5fba

also increment in fentry

43d1a4e

fix segment count on prebuilt pre-4.7

2abede5

add more tests

2adcb84

hasan feedback

66a752a

add sanity check

036bfaa

pimlu force-pushed the stuart.geipel/udp-counting branch from 9b93718 to 036bfaa Compare August 26, 2024 18:25

hmahmood reviewed Aug 26, 2024

View reviewed changes

pkg/network/tracer/tracer_linux_test.go Show resolved Hide resolved

add release note

b9a0ad6

pimlu requested a review from a team as a code owner August 26, 2024 18:35

hestonhoffman approved these changes Aug 26, 2024

View reviewed changes

hmahmood approved these changes Sep 5, 2024

View reviewed changes

dd-mergequeue bot merged commit d29f884 into main Sep 5, 2024
295 checks passed

dd-mergequeue bot deleted the stuart.geipel/udp-counting branch September 5, 2024 18:02

github-actions bot added this to the 7.58.0 milestone Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[system-probe] Add packet counting to existing UDP byte counters #28546

[system-probe] Add packet counting to existing UDP byte counters #28546

pimlu commented Aug 19, 2024 •

edited by jira bot

Loading

pr-commenter bot commented Aug 19, 2024 •

edited

Loading

pr-commenter bot commented Aug 19, 2024 •

edited

Loading

tracer

akarpz commented Aug 19, 2024 •

edited

Loading

pr-commenter bot commented Aug 21, 2024 •

edited

Loading

Fine details of change detection per experiment

Explanation

hestonhoffman left a comment

pimlu commented Sep 5, 2024

dd-devflow bot commented Sep 5, 2024

[system-probe] Add packet counting to existing UDP byte counters #28546

[system-probe] Add packet counting to existing UDP byte counters #28546

Conversation

pimlu commented Aug 19, 2024 • edited by jira bot Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

pr-commenter bot commented Aug 19, 2024 • edited Loading

Test changes on VM

pr-commenter bot commented Aug 19, 2024 • edited Loading

eBPF complexity changes

Summary result: ✅ - stable

tracer

akarpz commented Aug 19, 2024 • edited Loading

pr-commenter bot commented Aug 21, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Bounds Checks

Explanation

hestonhoffman left a comment

Choose a reason for hiding this comment

pimlu commented Sep 5, 2024

dd-devflow bot commented Sep 5, 2024

pimlu commented Aug 19, 2024 •

edited by jira bot

Loading

pr-commenter bot commented Aug 19, 2024 •

edited

Loading

pr-commenter bot commented Aug 19, 2024 •

edited

Loading

akarpz commented Aug 19, 2024 •

edited

Loading

pr-commenter bot commented Aug 21, 2024 •

edited

Loading