feat(app): Backend frame count metrics #3308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cratelyn
force-pushed
the
kate/record-body-data
branch
5 times, most recently
from
044cc73
to
41d2c9c
Compare
cratelyn
changed the title
cratelyn
force-pushed
the
kate/record-body-data
branch
2 times, most recently
from
f9d86cb
to
c08a6b5
Compare
cratelyn
commented
Nov 1, 2024
linkerd/app/outbound/src/http/logical/policy/route/backend/metrics.rs
Outdated
Show resolved
Hide resolved
cratelyn
changed the title
olix0r
reviewed
Nov 1, 2024
cratelyn
added a commit
that referenced
this pull request
Nov 10, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added Signed-off-by: katelyn martin <[email protected]>
cratelyn
force-pushed
the
kate/record-body-data
branch
from
cd2d1b9
to
3b1db24
Compare
cratelyn
added a commit
that referenced
this pull request
Nov 10, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added Signed-off-by: katelyn martin <[email protected]>
cratelyn
added a commit
that referenced
this pull request
Nov 10, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. Signed-off-by: katelyn martin <[email protected]>
cratelyn
added a commit
that referenced
this pull request
Nov 10, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. Signed-off-by: katelyn martin <[email protected]>
|
this pr has been rebased onto |
olix0r
reviewed
Nov 15, 2024
olix0r
reviewed
Nov 19, 2024
There was a problem hiding this comment.
I through together a quick example of what a histogram implementation here looks like: kate/record-body-data...ver/kate/record-body-data (It compiles, though I didn't check the test status).
This replaces the counter/total metrics pair: the frame_size counters give us the total number of frames processed, and the sum gives us the total number of bytes. Getting in the proper metrics in a first pass saves us a deprecation cycle.
I chose a fairly constrained set of buckets to give us orders of magnitude: 128, 1024, 10240 (and we'll get an +Inf).
I also tried to handle the wrapping case a little more gracefully.
cratelyn
force-pushed
the
kate/record-body-data
branch
from
52f55c6
to
cc04760
Compare
|
📝 NB: c874231 was lacking a DCO signoff, so i've added one on your behalf to satiate CI, @olix0r. that's now 225674b. see; https://github.com/linkerd/linkerd2-proxy/runs/33282345859 |
this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC response bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this middleware allows operators to reason about how large or small the packets being served in a backend's response bodies are. a route-level middleware that instruments request bodies will be added in a follow-on PR. ### 📝 changes an overview of changes made here: * the `linkerd-http-prom` has a new `body_data` submodule. it exposes `request` and `response` halves, to be explicit about which body is being instrumented on a `tower::Service`. * the `linkerd-http-prom` crate now has a collection of new dependencies: `bytes` is added as a dependency in order to inspect the data chunk when the inner body yields a new frame. `futures-util` and `http-body` are added as dev-dependencies for the accompanying test coverage. * body metrics are affixed to the `RouteBackendMetrics<L>` structure, and registered at startup. Signed-off-by: katelyn martin <[email protected]>
Signed-off-by: katelyn martin <[email protected]>
continuing this theme of inlining, we inline the passthrough methods on `Body` as well. Signed-off-by: katelyn martin <[email protected]>
Signed-off-by: katelyn martin <[email protected]>
Signed-off-by: katelyn martin <[email protected]>
Signed-off-by: katelyn martin <[email protected]>
Signed-off-by: Oliver Gould <[email protected]>
see: * prometheus/client_rust#242 * prometheus/client_rust#241 for now, refactor this test so that it gates all use of the (proposed) `sum()` and `count()` accessors behind a temporary feature gate. Signed-off-by: katelyn martin <[email protected]>
cratelyn
force-pushed
the
kate/record-body-data
branch
from
95e796f
to
c728e18
Compare
|
this pr has been rebased onto |
cratelyn
force-pushed
the
kate/record-body-data
branch
from
c728e18
to
33f5f60
Compare
changes from this branch have been incorporated here. i've added a DCO signoff, here. |
cratelyn
added a commit
that referenced
this pull request
Nov 21, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. Signed-off-by: katelyn martin <[email protected]>
cratelyn
added a commit
that referenced
this pull request
Nov 21, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, (b) the number of bytes included in body frames, and (c) a distribution of the size of frames yielded. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. see prometheus/client_rust#241 and prometheus/client_rust#242, which track upstream proposals to add accessors to `Histogram` that will allow us to make test assertions that metrics are working properly. for now, these are feature gated as also done in #3308. Signed-off-by: katelyn martin <[email protected]>
cratelyn
added a commit
that referenced
this pull request
Nov 21, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, (b) the number of bytes included in body frames, and (c) a distribution of the size of frames yielded. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. see prometheus/client_rust#241 and prometheus/client_rust#242, which track upstream proposals to add accessors to `Histogram` that will allow us to make test assertions that metrics are working properly. for now, these are feature gated as also done in #3308. Signed-off-by: katelyn martin <[email protected]>
cratelyn
added a commit
that referenced
this pull request
Nov 21, 2024
### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, (b) the number of bytes included in body frames, and (c) a distribution of the size of frames yielded. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. see prometheus/client_rust#241 and prometheus/client_rust#242, which track upstream proposals to add accessors to `Histogram` that will allow us to make test assertions that metrics are working properly. for now, these are feature gated as also done in #3308. Signed-off-by: katelyn martin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
☁️ overview
this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC
response bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes
included in body frames.
this middleware allows operators to reason about how large or small the packets being served in a backend's
response bodies are.
a route-level middleware that instruments request bodies will be added in a follow-on PR.
📝 changes
an overview of changes made here:
the
linkerd-http-promhas a newbody_datasubmodule. it exposesrequestandresponsehalves, tobe explicit about which body is being instrumented on a
tower::Service.the
linkerd-http-promcrate now has a collection of new dependencies:bytesis added as a dependencyin order to inspect the data chunk when the inner body yields a new frame.
futures-utilandhttp-bodyare added as dev-dependencies for the accompanying test coverage.
body metrics are affixed to the
RouteBackendMetrics<L>structure, and registered at startup.🔗 related