Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterize ingester upload compact blocks #5959

Conversation

CharlieTLe
Copy link
Member

What this PR does:
In v1.15.2, ingesters configured with OOO samples ingestion enabled
could hit this bug (#5402)
where ingesters would not upload compacted blocks
(thanos-io/thanos#6462).

In v1.16.1, ingesters are configured to always upload compacted blocks
(#5625).

In v1.17, ingesters stopped uploading compacted blocks
(#5735).

This can cause problems for users upgrading from v1.15.2 with OOO
ingestion enabled to v1.17 because both versions are hard coded to
disable uploading compacted blocks from the ingesters.

The workaround was to downgrade from v1.17 to v1.16 to allow those
compacted blocks to be uploaded (and eventually deleted).

The new flag is set to true by default which reverts the behavior of the
ingester uploading compacted blocks back to v1.16.

Which issue(s) this PR fixes:
Fixes #5402

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

… parameterize uploading compacted blocks

In v1.15.2, ingesters configured with OOO samples ingestion enabled
could hit this bug (cortexproject#5402)
where ingesters would not upload compacted blocks
(thanos-io/thanos#6462).

In v1.16.1, ingesters are configured to always upload compacted blocks
(cortexproject#5625).

In v1.17, ingesters stopped uploading compacted blocks
(cortexproject#5735).

This can cause problems for users upgrading from v1.15.2 with OOO
ingestion enabled to v1.17 because both versions are hard coded to
disable uploading compacted blocks from the ingesters.

The workaround was to downgrade from v1.17 to v1.16 to allow those
compacted blocks to be uploaded (and eventually deleted).

The new flag is set to true by default which reverts the behavior of the
ingester uploading compacted blocks back to v1.16.

Signed-off-by: Charlie Le <[email protected]>
@friedrichg friedrichg requested a review from yeya24 May 18, 2024 08:09
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Good catch!

@yeya24 yeya24 merged commit cb4633e into cortexproject:master May 19, 2024
16 checks passed
yeya24 pushed a commit that referenced this pull request May 20, 2024
… parameterize uploading compacted blocks (#5959)

In v1.15.2, ingesters configured with OOO samples ingestion enabled
could hit this bug (#5402)
where ingesters would not upload compacted blocks
(thanos-io/thanos#6462).

In v1.16.1, ingesters are configured to always upload compacted blocks
(#5625).

In v1.17, ingesters stopped uploading compacted blocks
(#5735).

This can cause problems for users upgrading from v1.15.2 with OOO
ingestion enabled to v1.17 because both versions are hard coded to
disable uploading compacted blocks from the ingesters.

The workaround was to downgrade from v1.17 to v1.16 to allow those
compacted blocks to be uploaded (and eventually deleted).

The new flag is set to true by default which reverts the behavior of the
ingester uploading compacted blocks back to v1.16.

Signed-off-by: Charlie Le <[email protected]>
@yeya24 yeya24 mentioned this pull request May 20, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unshipped blocks when out of order writes are enabled
2 participants