Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compact tenant TSDBs after 3h inactivity #5909

Merged
merged 1 commit into from
Nov 22, 2022

Conversation

fpetkovski
Copy link
Contributor

@fpetkovski fpetkovski commented Nov 20, 2022

The Receiver tenant retention mechanism has an edge case when a tenant TSDB gets evicted and uploaded to S3. Since there is a delay between uploads to S3 and downloads in Store Gateways, if a user executes a query between the upload and download time, they may not see the latest head block from the evicted tenant.

As a result, this commit decouples head compaction from tenant eviction. Head compaction, as in Prometheus, will happen if there are no new appends after 1.5x max-block-size. This will also cause the compacted block to be uploaded to S3 by the shipper. Eviction will then kick in after tsdb retention time, which will cause the tenant TSDB to be deleted.

By this time, the latest head block would have been picked up by store-gateway and would be available during query execution.

Signed-off-by: Filip Petkovski [email protected]

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • Force tenant head compaction after 1.5 times max-block-size if there are no new appends to the tenant.

Verification

Unit tests and manual verification.

The Receiver tenant retention mechanism has an edge case when a
tenant TSDB gets evicted and uploaded to S3. Since there is a delay
between uploads to S3 and downloads in Store Gateways, if a user
executes a query between the upload and download time, they may
not see the latest head block from the evicted tenant.

As a result, this commit decouples head compaction from tenant eviction.
Head compaction, as in Prometheus, will happen if there are no new appends
after 1.5x max-block-size. This will also cause the compacted block to be
uploaded to S3 by the shipper. Eviction will then kick in which will cause
the tenant TSDB to be deleted.

By this time, the latest head block would have been picked up by store-gateway
and would be available during query execution.

Signed-off-by: Filip Petkovski <[email protected]>
Copy link
Member

@saswatamcode saswatamcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing! 🌟

Copy link
Member

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bwplotka bwplotka merged commit 3c2657d into thanos-io:main Nov 22, 2022
ngraham20 pushed a commit to ngraham20/thanos that referenced this pull request May 18, 2023
The Receiver tenant retention mechanism has an edge case when a
tenant TSDB gets evicted and uploaded to S3. Since there is a delay
between uploads to S3 and downloads in Store Gateways, if a user
executes a query between the upload and download time, they may
not see the latest head block from the evicted tenant.

As a result, this commit decouples head compaction from tenant eviction.
Head compaction, as in Prometheus, will happen if there are no new appends
after 1.5x max-block-size. This will also cause the compacted block to be
uploaded to S3 by the shipper. Eviction will then kick in which will cause
the tenant TSDB to be deleted.

By this time, the latest head block would have been picked up by store-gateway
and would be available during query execution.

Signed-off-by: Filip Petkovski <[email protected]>

Signed-off-by: Filip Petkovski <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants