streamingccl: tighten replication timestamp semantics #92788
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
adityamaru
force-pushed
the
start-time-fixup
branch
from
d604467 to
6cf6ab4
Compare
adityamaru
requested review from
benbardin,
stevendanna,
lidorcarmel and
a team
and removed request for
a team and
benbardin
lidorcarmel
approved these changes
Dec 7, 2022
pkg/ccl/streamingccl/streamingest/stream_replication_e2e_test.go
Outdated
Show resolved
Hide resolved
pkg/ccl/streamingccl/streamingest/stream_replication_e2e_test.go
Outdated
Show resolved
Hide resolved
adityamaru
force-pushed
the
start-time-fixup
branch
3 times, most recently
from
4a364f5 to
8fafe9f
Compare
Previously, each partition would reach out to the source cluster and pick its own timestamp from which it would start ingesting MVCC versions. This timestamp was used by the rangefeed setup by the partition, to run its initial scan. Eventually, all the partitions would replicate up until a certain timestamp and cause the frontier to be bumped but it was possible for different partitions to begin ingesting at different timestamps. This change makes it such that during replication planning when we create the producer job on the source cluster, we return a timestamp alongwith the StreamID. This becomes the timestamp at which each ingestion partition sets up the inital scan of the rangefeed, and consequently become the inital timestamp at which all data is ingested. We stash this timestamp in the replication job details and never update its value. On future resumptions of the replication job, if there is a progress high water, we will not run an initial rangefeed scan but instead start the rangefeed from the previous progress highwater. The motivation for this change was to know the lower bound on both the source and destination cluster for MVCC versions that have been streamed. This is necessary to bound the fingerprinting on both clusters to ensure a match. Release note: None Fixes: cockroachdb#92742
adityamaru
force-pushed
the
start-time-fixup
branch
from
f4225c6 to
523b79d
Compare
|
Both failures are flakes: TFTR! bors r=lidorcarmel |
|
Build failed: |
|
Failed on |
stevendanna
approved these changes
Dec 12, 2022
| // start ingesting data in the replication job. This timestamp is empty unless | ||
| // the replication job resumes after a progress checkpoint has been recorded. | ||
| // While it is empty we use the InitialScanTimestamp described below. | ||
| optional util.hlc.Timestamp previous_high_water_timestamp = 2 [(gogoproto.nullable) = false]; |
|
bors retry |
|
Build failed: |
|
Now its bors retry |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, each partition would reach out to the
source cluster and pick its own timestamp from which it would start ingesting MVCC versions. This timestamp was used by the rangefeed setup by the partition, to run its initial scan. Eventually, all the partitions would replicate up until a certain timestamp and cause the frontier to be bumped but it was possible for different partitions to begin ingesting at different timestamps.
This change makes it such that during replication planning when we create the producer job on the source cluster, we return a timestamp alongwith the StreamID. This becomes the timestamp at which each ingestion partition sets up the inital scan of the rangefeed, and consequently become the inital timestamp at which all data is ingested. We stash this timestamp in the replication job details and never update its value. On future resumptions of the replication job, if there is a progress high water, we will not run an initial rangefeed scan but instead start the rangefeed from the previous progress highwater.
The motivation for this change was to know the lower bound on both the source and destination cluster for MVCC versions that have been streamed. This is necessary to bound the fingerprinting on both clusters to ensure a match.
Release note: None
Fixes: #92742