tap: introduce HTTP tap filter

[mattklein123]

This is a MVP for the HTTP tap filter. It includes minimal
infrastructure for the following:

1. Generic tap configuration which in the future will be used for
   static config, XDS config, etc. In this MVP the tap can be
   configured via a /tap admin endpoint.
2. Generic output configuration which in the future will be used for
   different output sinks such as files, gRPC API, etc. In this MVP
   the tap results are streamed back out the /tap admin endpoint.
3. Matching infrastructure. In this MVP only matching on request and
   response headers are implemented. Both logical AND and logical OR
   matches are possible.
4. In this MVP request/response body is not considered at all.
5. All docs are included and with all the caveats the filter is ready
   to use for the limited cases it supports (which are likely still to
   be useful).

There is a lot of follow on work which I will do in subsequent PRs.
This includes:

1. Merging the existing capture transport socket into this framework.
2. Implementing body support, both for matching on body contents as
   well as outputting body data.
3. Tap rate limiting so too many streams do not get tapped.
4. gRPC matching. Using reflection and loaded proto definitions, it will
   be possible to match on gRPC fields.
5. JSON matching. If the body parses as JSON, we can allow matching on
   JSON fields.

Part of #1413.

Risk Level: Low. New feature with no dependencies. Minor changes to a few other files.
Testing: Unit/integration tests.
Docs Changes: Added.
Release Notes: Added.

[mattklein123]

@dio @lizan since this feature is of general interest to the Envoy/Istio community can one/both of you potentially own review of it? @htuch also in case you are interested. My plan is to refactor the transport socket capture feature to work within this framework next.

[lizan]

First pass, didn't take detailed look at tests yet.

[mattklein123]

Replicating a comment from Slack:

@alyssawilk @zuercher @lizan With Lizan's help getting a log, I'm realizing that I don't think I can get the tap integration test working on OSX. The issue comes down to #4294. Basically, the admin /tap response is a streamed response that never ends, and the test relies on the admin client being able to hang up when it's done getting data, but on OSX Envoy never sees the hangup and cleans up the admin client. Any thoughts here on what I can do? I can't think of any way to fix this so I'm wondering if I shouldn't run the test on OSX? Ideas? https://github.com/envoyproxy/envoy/pull/5515/files#diff-975313274dec5c7ffca88fb1be76d717R95

[mattklein123]

To follow up here, @lizan had the very good idea to have the test use HTTP/2 vs. HTTP/1 which will work. Unfortunately, admin is hard coded to use HTTP/1 currently, so I will make admin do auto codec detection. I will end up splitting all the admin changes out into a separate PR once I have this passing tests.

[mattklein123]

@htuch @lizan I bit the bullet and rewrote the matching system to be a lot more generic and include and/or/any/not, etc. In doing so, I realized that the matching we need to do to support streaming matching (compute a match as data goes by vs. have everything available) is a lot more complicated than what we do for other matching in Envoy, and I'm skeptical we can share everything at a high level. I do think we can share the low-level matchers (headers, subnet, etc.) and I will work on that.

Anyway, PTAL. I added a bunch of comments on how the matchers work with regard to streaming. Even though this MVP does not support stream tapping today, we are setup to do so, and it's something that I know we will need to support shortly.

[htuch]

I like the match design, hoping we can make this more generic and shared over time. I'll take another pass in the next day with a finer granularity, but it basically looks good to go.

[mattklein123]

@htuch @lizan updated per comments

[htuch]

Cool. I'm good with the entire PR modulo what we're doing with matchers/streams, where I have some open questions.

[htuch]

How come we need the vector inlining? Can't we just build a regular tree and maintain a pointer to the current node of interest, tracking the leaf-parent path booleans as a vector if necessary?

[mattklein123]

Sorry I don't follow what you are suggesting. This implementation allows a per-request status vector to be created and then trivially updated without worrying about pointers. IMO it's both the simplest and most efficient implementation for what we need to do.

[htuch]

Fair enough, I don't disagree that it's an efficient implementation. I'm visualizing this as at any point in time, for a given request, you have the match tree (immutable and shared) and your current status, which is those parts that have matched. I can see the attractiveness of the flattening due to the fact that you can easily map either way between match predicate and status.

[mattklein123]

@htuch updated PTAL

[htuch]

LGTM, fine to address comment in later PR.

[htuch]

Should we have more complete tests here? Happy to punt this to a later PR.

[mattklein123]

Yeah, we should have targeted unit tests here. All of the other matchers have coverage via the integration tests, but will definitely add more UTs here in future PRs.

[repokitteh-read-only]

🙀 Error while processing event:

evaluation error
error: cannot load github.com/repokitteh/modules/lib/circleci.star: load("github.com/repokitteh/modules/lib/circleci.star") error: module load error

🐱

Caused by: a #5515 (review) was submitted by @htuch.

see: more, trace.