WIP NETOBSERV-1550: Using batchAPIs to help with CPU and memory resources #256

msherif1234 · 2024-01-24T16:30:43Z

Description

cilium recently added batchAPI support for PerCPU maps this PR to migrate ebpf agent to use batchapis

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

openshift-ci-robot · 2024-01-24T16:30:48Z

openshift-ci · 2024-01-24T16:30:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from msherif1234. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2024-01-24T16:35:26Z

Codecov Report

Attention: Patch coverage is 0% with 90 lines in your changes are missing coverage. Please review.

Project coverage is 33.44%. Comparing base (b63f483) to head (edd8134).
Report is 1 commits behind head on main.

❗ Current head edd8134 differs from pull request most recent head 5fdf081. Consider uploading reports for the commit 5fdf081 to get more accurate results

Files	Patch %	Lines
pkg/ebpf/tracer_batchapis.go	0.00%	57 Missing ⚠️
pkg/ebpf/tracer.go	0.00%	33 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #256      +/-   ##
==========================================
- Coverage   34.04%   33.44%   -0.61%     
==========================================
  Files          47       48       +1     
  Lines        3836     3905      +69     
==========================================
  Hits         1306     1306              
- Misses       2444     2513      +69     
  Partials       86       86

Flag	Coverage Δ
unittests	`33.44% <0.00%> (-0.61%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

openshift-ci-robot · 2024-01-24T19:09:48Z

msherif1234 · 2024-01-26T22:41:43Z

/ok-to-test

github-actions · 2024-01-26T22:43:29Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:6d184cc

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=6d184cc make set-agent-image

msherif1234 · 2024-01-26T22:54:09Z

/ok-to-test

github-actions · 2024-01-26T22:56:05Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:bfa5ac7

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=bfa5ac7 make set-agent-image

msherif1234 · 2024-01-27T13:52:40Z

ran scale based of 4.14
https://docs.google.com/spreadsheets/d/14taH8UGgjiLNqjgCRq66mcNBeNCDCnYEFGsur8Qig9I/edit#gid=2136829211

summary showing increase in ebpf resources

cpuEBPFTotals | cpuEBPFTotals | avg(value) | Fail | 78.57% | 3.405002158 | 6.080411792 |

rssEBPFTotals | rssEBPFTotals | avg(value) | Fail | 53.61% | 3404791063 | 5230041771 |

msherif1234 · 2024-01-29T17:07:33Z

(pprof) top10 -cum
Showing nodes accounting for 70ms, 3.14% of 2230ms total
Dropped 56 nodes (cum <= 11.15ms)
Showing top 10 nodes out of 92
      flat  flat%   sum%        cum   cum%
         0     0%     0%     1770ms 79.37%  github.com/netobserv/netobserv-ebpf-agent/pkg/flow.(*MapTracer).evictFlows
         0     0%     0%     1770ms 79.37%  github.com/netobserv/netobserv-ebpf-agent/pkg/flow.(*MapTracer).evictionSynchronization
         0     0%     0%     1760ms 78.92%  github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf.(*FlowFetcher).LookupAndDeleteMap
         0     0%     0%     1620ms 72.65%  github.com/cilium/ebpf.(*Map).BatchLookupAndDelete (inline)
         0     0%     0%     1620ms 72.65%  github.com/cilium/ebpf.(*Map).batchLookup
         0     0%     0%     1620ms 72.65%  github.com/cilium/ebpf.(*Map).batchLookupPerCPU
      40ms  1.79%  1.79%     1510ms 67.71%  github.com/cilium/ebpf/internal/sysenc.Unmarshal
      30ms  1.35%  3.14%     1350ms 60.54%  encoding/binary.Read
         0     0%  3.14%     1190ms 53.36%  github.com/cilium/ebpf.unmarshalBatchPerCPUValue
         0     0%  3.14%     1180ms 52.91%  github.com/cilium/ebpf.unmarshalPerCPUValue
(pprof)

msherif1234 · 2024-02-06T15:17:17Z

added bench mark testing for iterate vs batchdelete api

$ go test ./pkg/ebpf/ -exec sudo -bench=BenchmarkFlowFetcher_LookupAndDeleteMap -benchmem -count 5 -run=^#
goos: linux
goarch: amd64
pkg: github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf
cpu: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDelete-12         	     403	   2507858 ns/op	  757583 B/op	    2943 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDelete-12         	     446	   2531754 ns/op	  746563 B/op	    2838 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDelete-12         	     488	   2234317 ns/op	  737511 B/op	    2753 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDelete-12         	     526	   2209894 ns/op	  730663 B/op	    2688 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/BatchLookupAndDelete-12         	     477	   2251203 ns/op	  739670 B/op	    2774 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12       	     386	   2796254 ns/op	  598852 B/op	    4355 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12       	     345	   3105146 ns/op	  613746 B/op	    4492 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12       	     370	   2940347 ns/op	  604619 B/op	    4406 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12       	     304	   3723941 ns/op	  631809 B/op	    4664 allocs/op
BenchmarkFlowFetcher_LookupAndDeleteMap/IterateLookupAndDelete-12       	     326	   3699242 ns/op	  621145 B/op	    4566 allocs/op
PASS
ok  	github.com/netobserv/netobserv-ebpf-agent/pkg/ebpf	70.103s

msherif1234 · 2024-02-13T13:41:43Z

started a repro upstream cilium/ebpf#1343

msherif1234 · 2024-02-14T21:03:35Z

/ok-to-test

msherif1234 · 2024-02-22T11:57:02Z

/ok-to-test

github-actions · 2024-02-22T11:58:50Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:baad512

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=baad512 make set-agent-image

openshift-ci-robot · 2024-03-01T08:47:35Z

jotak · 2024-03-01T08:48:36Z

@msherif1234 I've created a new jira for this PR, NETOBSERV-1550, and the former is used for not-batched LookupAndDelete with my PR #283

jotak · 2024-03-01T08:58:03Z

pkg/ebpf/tracer.go

+			for i, id := range ids[:count] {
+				for j := 0; j < ebpf.MustPossibleCPU(); j++ {
+					flows[id] = append(flows[id], metrics[i*ebpf.MustPossibleCPU()+j])
+				}
+			}


Can you explain this, I'm not sure to understand?

we are now getting percpu metrics so we need to combine all metrics for each CPU together and assign them to the right flow id

Signed-off-by: Mohamed Mahmoud <[email protected]>

msherif1234 · 2024-03-25T14:29:40Z

/ok-to-test

msherif1234 · 2024-03-25T14:56:28Z

/ok-to-test

codecov-commenter · 2024-03-25T14:57:09Z

Codecov Report

Attention: Patch coverage is 0% with 90 lines in your changes are missing coverage. Please review.

Project coverage is 33.44%. Comparing base (b63f483) to head (edd8134).
Report is 1 commits behind head on main.

❗ Current head edd8134 differs from pull request most recent head 5fdf081. Consider uploading reports for the commit 5fdf081 to get more accurate results

Files	Patch %	Lines
pkg/ebpf/tracer_batchapis.go	0.00%	57 Missing ⚠️
pkg/ebpf/tracer.go	0.00%	33 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #256      +/-   ##
==========================================
- Coverage   34.04%   33.44%   -0.61%     
==========================================
  Files          47       48       +1     
  Lines        3836     3905      +69     
==========================================
  Hits         1306     1306              
- Misses       2444     2513      +69     
  Partials       86       86

Flag	Coverage Δ
unittests	`33.44% <0.00%> (-0.61%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Mohamed Mahmoud <[email protected]>

msherif1234 · 2024-03-25T15:09:59Z

/ok-to-test

github-actions · 2024-03-25T15:12:01Z

New image:
quay.io/netobserv/netobserv-ebpf-agent:f8e7e13

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=f8e7e13 make set-agent-image

msherif1234 · 2024-04-16T13:07:45Z

I will close this PR as it never shows any real value switching to batchAPIs vs what we have today should we ever reconsider we can reopen it

openshift-ci-robot added the jira/valid-reference label Jan 24, 2024

openshift-ci bot added the do-not-merge/work-in-progress label Jan 24, 2024

jpinsonneau marked this pull request as draft January 25, 2024 11:24

msherif1234 force-pushed the batch-apis branch 3 times, most recently from 93db299 to 53cbc3a Compare January 26, 2024 22:41

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 26, 2024

msherif1234 force-pushed the batch-apis branch from 53cbc3a to 781299c Compare January 26, 2024 22:53

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 26, 2024

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 26, 2024

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 6, 2024

msherif1234 force-pushed the batch-apis branch 2 times, most recently from ab946c5 to 4597d54 Compare February 6, 2024 18:20

openshift-merge-robot added the needs-rebase label Feb 14, 2024

msherif1234 force-pushed the batch-apis branch from 652cac7 to 59faed7 Compare February 14, 2024 20:46

openshift-merge-robot removed the needs-rebase label Feb 14, 2024

msherif1234 force-pushed the batch-apis branch from 59faed7 to bf3361e Compare February 14, 2024 20:50

openshift-merge-robot added the needs-rebase label Feb 21, 2024

msherif1234 force-pushed the batch-apis branch from 8e763f4 to 237607c Compare February 22, 2024 11:55

openshift-merge-robot removed the needs-rebase label Feb 22, 2024

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Feb 22, 2024

jotak changed the title ~~WIP NETOBSERV-559: Using batchAPIs to help with CPU and memory resources~~ WIP NETOBSERV-1550: Using batchAPIs to help with CPU and memory resources Mar 1, 2024

jotak reviewed Mar 1, 2024

View reviewed changes

msherif1234 force-pushed the batch-apis branch from 237607c to 3e298df Compare March 25, 2024 11:59

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 25, 2024

NETOBSERV-559: Using batchAPIs to help with CPU and memory resources

7fa4fa0

Signed-off-by: Mohamed Mahmoud <[email protected]>

msherif1234 force-pushed the batch-apis branch from 3e298df to 7fa4fa0 Compare March 25, 2024 12:10

msherif1234 requested a review from jotak March 25, 2024 12:13

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 25, 2024

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 25, 2024

msherif1234 force-pushed the batch-apis branch from edd8134 to 5fdf081 Compare March 25, 2024 14:47

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 25, 2024

update workflow to go1.22 as its required

5781630

Signed-off-by: Mohamed Mahmoud <[email protected]>

msherif1234 force-pushed the batch-apis branch from 5fdf081 to 5781630 Compare March 25, 2024 15:06

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 25, 2024

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 25, 2024

msherif1234 closed this Apr 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP NETOBSERV-1550: Using batchAPIs to help with CPU and memory resources #256

WIP NETOBSERV-1550: Using batchAPIs to help with CPU and memory resources #256

msherif1234 commented Jan 24, 2024 •

edited

Loading

openshift-ci-robot commented Jan 24, 2024 •

edited by openshift-ci bot

Loading

Description

Dependencies

Checklist

openshift-ci bot commented Jan 24, 2024

codecov bot commented Jan 24, 2024 •

edited

Loading

openshift-ci-robot commented Jan 24, 2024 •

edited by openshift-ci bot

Loading

Description

Dependencies

Checklist

msherif1234 commented Jan 26, 2024

github-actions bot commented Jan 26, 2024

msherif1234 commented Jan 26, 2024

github-actions bot commented Jan 26, 2024

msherif1234 commented Jan 27, 2024

msherif1234 commented Jan 29, 2024 •

edited

Loading

msherif1234 commented Feb 6, 2024 •

edited

Loading

msherif1234 commented Feb 13, 2024

msherif1234 commented Feb 14, 2024

msherif1234 commented Feb 22, 2024

github-actions bot commented Feb 22, 2024

openshift-ci-robot commented Mar 1, 2024 •

edited by openshift-ci bot

Loading

Description

Dependencies

Checklist

jotak commented Mar 1, 2024

jotak Mar 1, 2024

msherif1234 Mar 4, 2024

msherif1234 commented Mar 25, 2024

msherif1234 commented Mar 25, 2024

codecov-commenter commented Mar 25, 2024

msherif1234 commented Mar 25, 2024

github-actions bot commented Mar 25, 2024

msherif1234 commented Apr 16, 2024

WIP NETOBSERV-1550: Using batchAPIs to help with CPU and memory resources #256

WIP NETOBSERV-1550: Using batchAPIs to help with CPU and memory resources #256

Conversation

msherif1234 commented Jan 24, 2024 • edited Loading

Description

Dependencies

Checklist

openshift-ci-robot commented Jan 24, 2024 • edited by openshift-ci bot Loading

Description

Dependencies

Checklist

openshift-ci bot commented Jan 24, 2024

codecov bot commented Jan 24, 2024 • edited Loading

Codecov Report

openshift-ci-robot commented Jan 24, 2024 • edited by openshift-ci bot Loading

Description

Dependencies

Checklist

msherif1234 commented Jan 26, 2024

github-actions bot commented Jan 26, 2024

msherif1234 commented Jan 26, 2024

github-actions bot commented Jan 26, 2024

msherif1234 commented Jan 27, 2024

msherif1234 commented Jan 29, 2024 • edited Loading

msherif1234 commented Feb 6, 2024 • edited Loading

msherif1234 commented Feb 13, 2024

msherif1234 commented Feb 14, 2024

msherif1234 commented Feb 22, 2024

github-actions bot commented Feb 22, 2024

openshift-ci-robot commented Mar 1, 2024 • edited by openshift-ci bot Loading

Description

Dependencies

Checklist

jotak commented Mar 1, 2024

jotak Mar 1, 2024

Choose a reason for hiding this comment

msherif1234 Mar 4, 2024

Choose a reason for hiding this comment

msherif1234 commented Mar 25, 2024

msherif1234 commented Mar 25, 2024

codecov-commenter commented Mar 25, 2024

Codecov Report

msherif1234 commented Mar 25, 2024

github-actions bot commented Mar 25, 2024

msherif1234 commented Apr 16, 2024

msherif1234 commented Jan 24, 2024 •

edited

Loading

openshift-ci-robot commented Jan 24, 2024 •

edited by openshift-ci bot

Loading

codecov bot commented Jan 24, 2024 •

edited

Loading

openshift-ci-robot commented Jan 24, 2024 •

edited by openshift-ci bot

Loading

msherif1234 commented Jan 29, 2024 •

edited

Loading

msherif1234 commented Feb 6, 2024 •

edited

Loading

openshift-ci-robot commented Mar 1, 2024 •

edited by openshift-ci bot

Loading