-
Notifications
You must be signed in to change notification settings - Fork 272
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Try to stop OTLP controllers when Telemetry is dropped #3143
Conversation
This comment has been minimized.
This comment has been minimized.
CI performance tests
|
We already have code to specifically drop tracers and we are adding some additional logic to do the same thing with metrics exporters. fixes: #3140
I'm not sure if this is a useful metric to maintain long-term, but it's useful right now.
1bb1825
to
11bf9b9
Compare
I've figured out a way of testing this on my local machine and this change does improve the story for OTLP metrics in the router. If I do this with the current dev router, then I never see that 400 http requests are counted. I usually get about 300, but clearly the metrics associated with the last few requests are being dropped at router shutdown. Aside: In both implementations no metrics are lost during router reload. Summary: In order to avoid losing metrics during router shutdown we should make this change. It's probably an edge case, since losing metrics during shutdown shouldn't substantially affect many users, but it's a small improvement and worth making. |
We already have code to specifically drop tracers and we are adding some additional logic to do the same thing with metrics exporters.
fixes: #3140
Checklist
Complete the checklist (and note appropriate exceptions) before a final PR is raised.
Exceptions
Note any exceptions here
Notes
[^1]. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this.
[^2]. Configuration is an important part of many changes. Where applicable please try to document configuration examples.
[^3]. Tick whichever testing boxes are applicable. If you are adding Manual Tests:
- please document the manual testing (extensively) in the Exceptions.
- please raise a separate issue to automate the test and label it (or ask for it to be labeled) as
manual test