Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Trafficdump tool #1004

Merged
merged 17 commits into from
Feb 8, 2022
Merged

Add Trafficdump tool #1004

merged 17 commits into from
Feb 8, 2022

Conversation

pstibrany
Copy link
Member

@pstibrany pstibrany commented Feb 2, 2022

What this PR does: This PR adds trafficdump tool, which can read packets from device or from captured tcpdump output, reassemble them into TCP streams and parse HTTP requests and responses. It then prints requests and responses as json (one request/response par per line) for further processing.

It has some Mimir-specific and generic HTTP features:

  • filter requests based on Tenant (in Basic or X-Scope-OrgId header)
  • filter requests based on URL path
  • filter requests based on status code of the response
  • decode Mimir push requests
  • filter requests based on matching series in push requests

Trafficdump can be used to inspect both remote-write requests and queries.

Checklist

  • [na] Tests updated
  • Documentation added
  • [na] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@pstibrany pstibrany changed the title Added trafficdump tool. Trafficdump tool Feb 2, 2022
.golangci.yml Outdated Show resolved Hide resolved
@@ -0,0 +1,13 @@
Trafficdump tool can read packets from device or from captured tcpdump output, reassemble them into TCP streams
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdbaldry, did we want to keep this doc in the README.md file? I think so, but cannot remember.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to have this in the technical documentation as "About mimirtool"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split that off into a subsequent issue or PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to have this in the technical documentation as "About mimirtool"

Relates to #1014.

Copy link
Contributor

@osg-grafana osg-grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unblocking, and can update docs later too.

@pstibrany
Copy link
Member Author

I plan to write a documentation shortly describing all our tools.

@osg-grafana osg-grafana changed the title Trafficdump tool Add Trafficdump tool Feb 4, 2022
tools/trafficdump/README.md Outdated Show resolved Hide resolved
tools/trafficdump/main.go Outdated Show resolved Hide resolved
tools/trafficdump/main.go Outdated Show resolved Hide resolved
tools/trafficdump/parser.go Outdated Show resolved Hide resolved
@replay
Copy link
Contributor

replay commented Feb 4, 2022

I think it would be useful to have a flag which makes it not show all the warnings and errors, but only the information which it parsed successfully.

For example I tested it by doing this:

./trafficdump.linux  -r ./tcpdump -request.decode-remote-write -select '{pod="distributor-5684946f49-gfclc"}' > parsed 2>&1

And it did successfully parse out the metrics of the specified selector, but there was also quite a lot of warning/error noise:

replay@mauro-workstation:~/go/src/github.com/grafana/mimir/tools/trafficdump$ wc -l parsed
84856 parsed
replay@mauro-workstation:~/go/src/github.com/grafana/mimir/tools/trafficdump$ grep 'dropping response' parsed | wc -l
34827
replay@mauro-workstation:~/go/src/github.com/grafana/mimir/tools/trafficdump$ grep 'malformed HTTP' parsed | wc -l
45125

Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impressing tooling! I'm 👍 : it's just a tooling that doesn't interfere with Mimir. You already successfully used it, in my opinion for such tooling there's no need for a deep review, especially if you already successfully used it, so let's get in, let's keep using it and improving over time based on hands-on experience.

@osg-grafana
Copy link
Contributor

I plan to write a documentation shortly describing all our tools.

See #1098.

pstibrany and others added 12 commits February 8, 2022 11:23
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Only process packets with src or dest port equal to given value.
Reuse buffers for parsing push requests.
Synchronize output.
Log message when internal queues are full and we need to drop parsed request or response.

Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
- only parse file, no live capturing
- only print requests/responses that were parsed successfully by default
- expose -ignore-path flag
- typos/cleanup

Signed-off-by: Peter Štibraný <[email protected]>
… without libpcap. Removed linux build.

Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
@pstibrany pstibrany merged commit 58ff956 into main Feb 8, 2022
@pstibrany pstibrany deleted the trafficdump branch February 8, 2022 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants