release-21.2: changefeedccl: resurrect manual protected timestamps #77589
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
samiskin
force-pushed
the
backport-changefeed-active-protected-ts
branch
from
11d326c to
03d512f
Compare
samiskin
changed the title
Previously changefeeds only laid down protected timestamp records to protect against either an ongoing backfill or the changefeed lagging behind. This is insufficient in cases such as if the gcttl is very short, recurring errors retry the changefeed for too long, or in upcoming work to enable serverless to shut down idle changefeeds. This PR removes the manual PTS protection on backfills and begins an async routine on the changeFrontier that updates the protected timestamp record to the current highwater mark. Fixes cockroachdb#76247 Release note (enterprise change): changefeeds running on tables with a low gcttl will function more reliably due to protected timestamps being maintained for the changefeed targets at the resolved timestamp of the changefeed. The frequency at which the protected timestamp is updated to the resolved timestamp can be configured through the `changefeed.protect_timestamp_interval` cluster setting. If the changefeed lags too far behind such that storage of old data becomes an issue, cancelling the changefeed will release the protected timestamps and allow garbage collection to resume. If `protect_data_from_gc_on_pause` is unset, pausing the changefeed will release the existing protected timestamp record.
samiskin
force-pushed
the
backport-changefeed-active-protected-ts
branch
from
03d512f to
8272030
Compare
samiskin
requested review from
gh-casper and
miretskiy
and removed request for
a team
|
@miretskiy There are 4 commits, with the 2 that aren't cherry-picks being the actual changes from the master versions of the PRs |
miretskiy
approved these changes
Mar 10, 2022
| // ActiveProtectedTimestamps enables always having protected timestamps laid | ||
| // down that are periodically advanced to the highwater mark. | ||
| var ActiveProtectedTimestamps = settings.RegisterBoolSetting( | ||
| "changefeed.active_protected_timestamps", |
There was a problem hiding this comment.
let's rename to changefeed.active_protected_timestamps.enabled
| // water) will have a protected timestamp. | ||
| return nil | ||
| if cp.ProtectedTimestampRecord == uuid.Nil { | ||
| // This case shouldn't occur when ActiveProtectedTimestamps is enabled |
There was a problem hiding this comment.
I think the comment should be updated; this event can easily happen if Active PTS was off before, and now it's on (btw, this will definitely happen if you're upgrading cluster).
I think flipping this flag, even multiple times, is probably okay.
| var ActiveProtectedTimestamps = settings.RegisterBoolSetting( | ||
| "changefeed.active_protected_timestamps", | ||
| "if set, rather than only protecting changefeed targets from garbage collection during backfills, data will always be protected up to the changefeed's frontier", | ||
| false, |
There was a problem hiding this comment.
I think we need to add this flag into retiredSettings in registry.go (on master).
samiskin
force-pushed
the
backport-changefeed-active-protected-ts
branch
2 times, most recently
from
0d5d5bb to
10fa208
Compare
samiskin
force-pushed
the
backport-changefeed-active-protected-ts
branch
from
10fa208 to
b5d95d8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 1/1 commits from #76605.
Backport 1/1 commits from #77606.
The changes from the master version are in their own commits, mostly around fixing tests. I also had to backport a small change from #75897 where I split out
isBehindinto its own function.Release justification: new feature that is desired by doordash that is gated behind a cluster setting
/cc @cockroachdb/release
Previously changefeeds only laid down protected timestamp records to
protect against either an ongoing backfill or the changefeed lagging
behind. This is insufficient in cases such as if the gcttl is very
short, recurring errors retry the changefeed for too long, or in
upcoming work to enable serverless to shut down idle changefeeds.
This PR removes the manual PTS protection on backfills and begins an
async routine on the changeFrontier that updates the protected timestamp
record to the current highwater mark.
Fixes #76247
Release note (enterprise change): changefeeds running on tables with a
low gcttl will function more reliably due to protected timestamps being
maintained for the changefeed targets at the resolved timestamp of the
changefeed. The frequency at which the protected timestamp is updated
to the resolved timestamp can be configured through the
changefeed.protect_timestamp_interval cluster setting. If the
changefeed lags too far behind such that storage of old data becomes an
issue, cancelling the changefeed will release the protected timestamps
and allow garbage collection to resume. If
protect_data_from_gc_on_pause is unset, pausing the changefeed will
release the existing protected timestamp record.