Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Querier is reporting failure at synchronizing new blocks for not existent deletion-marks #2387

Closed
kubicgruenfeld opened this issue Apr 2, 2020 · 3 comments
Closed

Querier is reporting failure at synchronizing new blocks for not existent deletion-marks #2387

kubicgruenfeld opened this issue Apr 2, 2020 · 3 comments
Assignees
@pstibrany

Comments

@kubicgruenfeld
Copy link
Contributor

I may have found a small bug. We have failures reported every 5 minutes when querier is looking for new blocks in object storage in the metric cortex_querier_thanos_objstore_bucket_operation_failures_total . I just tcpdumped it and the queriers are looking for the deletion-mark.json which are not there for all blocks and so are reporting 404 not found.
Queriers are also not reporting any error in Logs.
@pstibrany pstibrany self-assigned this Apr 2, 2020
@pstibrany
Copy link
Contributor

We need to fix this in Thanos first. Created thanos-io/thanos#2365.

@pstibrany
Copy link
Contributor

@kubicgruenfeld Thank you for reporting and investigation! Much appreciated!

@pstibrany pstibrany mentioned this issue Apr 15, 2020
Merged
3 tasks
@pstibrany
Copy link
Contributor

Closed by #2467

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pstibrany pstibrany

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kubicgruenfeld @pstibrany