Report packet-capture statistics by host #179

thatplguy · 2022-11-11T06:41:41Z

This PR adds per-host packet capture statistics for TLS handshakes and HTTP requests, which readily have the host available. Per-host stats are uploaded with telemetry data and summarized as CLI output alongside per-port statistics.

This also adds hard limits on the number of ports, interfaces, and hosts that the CLI tracks for telemetry data. The CLI is a long-running process, and our telemetry data monotonically increases. We should consider capturing telemetry data in rolling windows in the future. For now, I chose a limit of 10,000 each for ports, interfaces, and hosts; if we assume hosts and interfaces are ~10 bytes, then that maxes out at ~2Mb. These are most useful in the first 60 seconds of running the CLI, so 10K should be more than enough.

Depends on postmanlabs/observability-shared-libs#174.

Here's what the CLI packet capture summary looks like with hosts:

[INFO] Created new trace on Akita Cloud: akita://akibox:trace:mud-fisher-74a4bce9
[INFO] Running learn mode on interfaces awdl0, llw0, utun4, en0, utun3, lo0, anpi2, utun5, utun2, utun6, anpi1, anpi0, utun0, utun1
[INFO] --filter flag is not set; capturing all network traffic to and from your services.
[INFO] Send SIGINT (Ctrl-C) to stop...
[INFO] Printing packet capture statistics after 60 seconds of capture.
[INFO] TCP port  8000:  1386 packets (50% of total), 99 HTTP requests, 99 HTTP responses, 0 TLS handshakes, 18 unparsed packets.
[INFO] TCP port   443:  1226 packets (44% of total), no HTTP requests or responses, 18 TLS handshakes indicating encrypted traffic.
[INFO] TCP port 64791:   136 packets (4% of total), no HTTP requests or responses; the data to this service could not be parsed.
[INFO] Host localhost:8000                                                   99 HTTP requests, 0 TLS handshakes.
[INFO] Host 560E52A873D46C9D31B9710C15766566.gr7.us-east-1.eks.amazonaws.com no HTTP requests, 4 TLS handshakes indicating encrypted traffic.
[INFO] Host sqs.us-west-1.amazonaws.com                                      no HTTP requests, 4 TLS handshakes indicating encrypted traffic.

apidump/summary.go

liujed · 2022-11-11T10:11:12Z

apidump/summary.go

+	s.printPortHighlights(top)
+	s.printHostHighlights(top)


Can we clarify to the user that these highlights do not describe disjoint sets of traffic?

I added section labels to that effect in dd88251.

liujed · 2022-11-11T10:14:42Z

apidump/summary.go

+		if left.HTTPRequests != right.HTTPRequests {
+			return left.HTTPRequests > right.HTTPRequests
+		} else if left.HTTPResponses != right.HTTPResponses {
+			return left.HTTPResponses > right.HTTPResponses


Should we sort by the sum instead?

Makes sense. Changed in bf4d2b3.

liujed · 2022-11-11T10:31:01Z

trace/stats.go

@@ -131,11 +172,39 @@ func (s *PacketCounter) Summary(n int) *PacketCountSummary {
 	s.mutex.RLock()
 	defer s.mutex.RUnlock()

+	topByPort, byPortOverflow := s.byPort.TopN(n, func(c *PacketCounts) int { return c.TCPPackets })
+	topByInterface, byInterfaceOverflow := s.byInterface.TopN(n, func(c *PacketCounts) int { return c.TCPPackets })
+	topByHost, byHostOverflow := s.byHost.TopN(n, func(c *PacketCounts) int { return c.TCPPackets })


Aren't the TCP counts in byHost all zero, according to the XXX comment on line 37?

Yes, good point. Fixed in bf4d2b3.

liujed · 2022-11-11T10:35:08Z

trace/stats.go

+		limit: limit,
+		m:     make(map[T]*PacketCounts),
+
+		// Accumulate across all interfaces and ports.


And hosts too?

Fixed in bf4d2b3.

liujed · 2022-11-11T10:42:11Z

trace/stats.go

@@ -27,40 +28,71 @@ func (d *PacketCountDiscard) Update(_ PacketCounts) {
 // In the future, this could put counters on a pipe and do the increments
 // in a separate goroutine, but we would *still* need a mutex to read the
 // totals out.
-// TODO: limit maximum size
 type PacketCounter struct {


Documentation should be updated.

Updated in bf4d2b3.

trace/stats.go

thatplguy added 6 commits November 10, 2022 16:13

Quick test of counting by port

cfc7697

Add limits to the packet capture telemetry

805f4bd

Summarize packet capture by host, where possible

1d36b30

Get akita-libs from branch cns/telemetry

b878f9a

Refactor summarization

83c1164

Fix TLSClientHello collector, add comments

b623537

thatplguy requested a review from a team November 11, 2022 06:41

liujed approved these changes Nov 11, 2022

View reviewed changes

thatplguy added 6 commits November 11, 2022 10:36

Sort per-host telemetry by total count

d7d0245

Update comments and address PR feedback

bf4d2b3

Collect telemetry data for hosts with unknown hostnames

88eda07

Address remaining PR feedback

dd88251

Tweak host summary cutoff

35dba59

Bump akita-libs

4dd39bb

thatplguy merged commit cf4ce98 into main Nov 11, 2022

thatplguy deleted the cns/tls-hosts branch November 11, 2022 22:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report packet-capture statistics by host #179

Report packet-capture statistics by host #179

thatplguy commented Nov 11, 2022

liujed Nov 11, 2022

thatplguy Nov 11, 2022

liujed Nov 11, 2022

thatplguy Nov 11, 2022

liujed Nov 11, 2022

thatplguy Nov 11, 2022

liujed Nov 11, 2022

thatplguy Nov 11, 2022

liujed Nov 11, 2022

thatplguy Nov 11, 2022

Report packet-capture statistics by host #179

Report packet-capture statistics by host #179

Conversation

thatplguy commented Nov 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment