Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stackdriver exporter sometimes loses metric on shutdown #2011

Closed
quentin-cha opened this issue Feb 4, 2021 · 9 comments
Closed

stackdriver exporter sometimes loses metric on shutdown #2011

quentin-cha opened this issue Feb 4, 2021 · 9 comments
Assignees
Labels
area/monitoring kind/feature lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@quentin-cha
Copy link

/area monitoring
/kind feature

Expected Behavior

stackdriver exporter do not lose metrics when shutting down

Actual Behavior

stackdriver exporter sometimes lose metrics when shutting down

Additional Info

Currently during process shutdown, stackdriver exporters may lose metrics.
This is happening because:

  • Flushing isn't happening correctly, this will soon be fixed.
  • Shutdown sequence needs to be: Stop exporter -> Wait for minimum stackdriver upload interval (~10s) -> Flush
@quentin-cha
Copy link
Author

/assign

@skonto
Copy link
Contributor

skonto commented Mar 10, 2021

@quentin-cha what is left to be done here? I see that the fix flushes the interval reader so we need to bring the dep in, other than that what is pending the flushing part? @evankanderson Any reason we dont flush the exporter after we replace it here?
Currently we only flush exporters when the process is shutting down.

@quentin-cha
Copy link
Author

Apologies for not being able to work on this. Besides pulling in the dep, we need to actually invoke StopMetricsExporter() before we invoke FlushExporter(). The complication is that export to stackdriver is rate limited to once very ~10s so in order to guarantee not losing any metrics, there should be a 10s sleep between StopMetricsExporter() and FlushExporter()

@evankanderson
Copy link
Member

I've been chatting with @MontyCarter, and he suggested that we may be able to retire the Stackdriver exporter before 0.22.

I'm assuming this affects OpenCensus as well, though?

@skonto
Copy link
Contributor

skonto commented Mar 12, 2021

It affects opencensus. Should we open an issue for the retirement thing, @MontyCarter gentle ping.

@github-actions
Copy link
Contributor

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 11, 2021
@MontyCarter
Copy link
Contributor

Sorry. Not sure how I missed this. Anyway, the stackdriver exporter can be removed.

@github-actions github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 12, 2021
@github-actions
Copy link
Contributor

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 10, 2021
@dprotaso
Copy link
Member

stack driver exporter was removed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/monitoring kind/feature lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Development

No branches or pull requests

6 participants