-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kv/kvserver: allow the merge transaction to be pushed #60567
Conversation
a35460d
to
bc1d7b8
Compare
This probably deserves some more advanced testing involving pushing the merge transaction after it has sent the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This probably deserves some more advanced testing involving pushing the merge transaction after it has sent the SubsumeRequest.
I'm going to be adding a test in to make sure that two ranges with global_reads: true
can be merged, which will force the merge txn to get pushed when writing to local descriptors, so feel free to hold off on this test if you aren't feeling inspired to add it now.
Reviewed 6 of 6 files at r1, 1 of 2 files at r2.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner and @andreimatei)
pkg/kv/sender.go, line 294 at r1 (raw file):
GetSteppingMode(ctx context.Context) (curMode SteppingMode) // ForceRefreshTransaction attempts to refresh a transactions read timestamp
nit: same question about naming. Consider renaming to ManualRefresh
and moving this next to ManualRestart
.
pkg/kv/txn.go, line 1299 at r1 (raw file):
} // ForceRefresh forces a refresh of the read timestamp of a transaction to
nit: did you consider the name ManualRefresh
to parallel the existing ManualRestart
?
pkg/kv/kvclient/kvcoord/txn_coord_sender_test.go, line 2454 at r1 (raw file):
} require.NoError(t, <-errCh) require.NoError(t, txn.ForceRefresh(ctx))
Comment that the fact that this doesn't block on the testing machinery indicates that it's a no-op.
pkg/kv/kvserver/client_merge_test.go, line 1325 at r2 (raw file):
} if mergePostSplitTS.LessEq(splitTS) { t.Fatalf("expected merge to start before concurrent split, %v <= %v", mergePostSplitTS, splitTS)
s/start before/finish after/
?
This commit adds support for client-initiated refreshes of transactions. The implementation is somewhat simplistic in that it hijacks existing logic that occurs during the sending of a request. This makese the implementation more uniform with the rest of the client library at the cost of being somewhat awkward and implicit from a code-reading perspective. The motivation for this change is to provide the necessary tools to allow the merge transaction to get pushed. The adoption follows in the next commit. Release note: None
Historically we have not allowed the merge transaction to be pushed. The reason we disabled this was because of the hazard due to attempting to refresh reads on the RHS of the merge after the SubsumeRequest has been sent. The `SubsumeRequest` effectively freezes the RHS until the merge commits or aborts. In order to side-step this hazard, this change ensures that nothing should prevent the merge transaction from either committing or aborting. Release note: None
bc1d7b8
to
3ff8b35
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so feel free to hold off on this test if you aren't feeling inspired to add it now.
Excellent
Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @andreimatei and @nvanbenschoten)
pkg/kv/sender.go, line 294 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
nit: same question about naming. Consider renaming to
ManualRefresh
and moving this next toManualRestart
.
Done.
pkg/kv/txn.go, line 1299 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
nit: did you consider the name
ManualRefresh
to parallel the existingManualRestart
?
Done.
pkg/kv/kvclient/kvcoord/txn_coord_sender_test.go, line 2454 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Comment that the fact that this doesn't block on the testing machinery indicates that it's a no-op.
Done.
pkg/kv/kvserver/client_merge_test.go, line 1325 at r2 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
s/start before/finish after/
?
Done.
TFTR! bors r=nvanbenschoten |
Build succeeded: |
Made possible by cockroachdb#60567. This commit adds a new test called TestStoreRangeSplitAndMergeWithGlobalReads that tests that a range configured to serve global reads can be split and merged. In essence, this tests whether the split and merge transactions can handle having their timestamp bumped by the closed timestamp on the ranges they're operating on. The test revealed that range merges did have issues in these cases, because SubsumeRequests assumed that the intent on the RHS's local descriptor was below the node's HLC clock. This is no longer always true, so we now perform the inconsistent scan at hlc.MaxTimestamp, just like QueryIntent requests do.
Fixes cockroachdb#60760. Fallout from cockroachdb#60567. The refreshed BatchRequest was nil on the error path, which was resulting in a nil-pointer exception. This commit fixes this by passing the original BatchRequest to updateStateLocked, like the TxnCoordSender normally does.
Made possible by cockroachdb#60567. This commit adds a new test called TestStoreRangeSplitAndMergeWithGlobalReads that tests that a range configured to serve global reads can be split and merged. In essence, this tests whether the split and merge transactions can handle having their timestamp bumped by the closed timestamp on the ranges they're operating on. The test revealed that range merges did have issues in these cases, because SubsumeRequests assumed that the intent on the RHS's local descriptor was below the node's HLC clock. This is no longer always true, so we now perform the inconsistent scan at hlc.MaxTimestamp, just like QueryIntent requests do.
59566: kvserver: introduce a Raft-based transport for closedts r=andreimatei a=andreimatei This patch introduces a replacement for the existing closed timestamp mechanism / transport. The new mechanism is gated by a cluster version. Raft commands now carry increasing closed timestamps generated by the propBuf by using the recent request Tracker for synchronizing with in-flight requests (i.e. not closing timestamps below them). Raft commands get a closed ts field, and the range state gets the field as well. The propBuf pays attention to the range's closed timestamp policy for deciding whether to close lagging or leading timestamps. Release note: None 60753: kv: add TestStoreRangeSplitAndMergeWithGlobalReads r=nvanbenschoten a=nvanbenschoten Made possible by #60567. This commit adds a new test called TestStoreRangeSplitAndMergeWithGlobalReads that tests that a range configured to serve global reads can be split and merged. In essence, this tests whether the split and merge transactions can handle having their timestamp bumped by the closed timestamp on the ranges they're operating on. The test revealed that range merges did have issues in these cases, because SubsumeRequests assumed that the intent on the RHS's local descriptor was below the node's HLC clock. This is no longer always true, so we now perform the inconsistent scan at hlc.MaxTimestamp, just like QueryIntent requests do. 60772: sql: add parse_timestamp builtin r=RaduBerinde a=RaduBerinde Add a builtin that can be used to parse timestamp strings. This is like a cast, but it does not accept relative timestamps so it can be immutable. Only immutable expressions are allowed in computed column expressions or partial index predicates; unlike casts, the new function can be used in such expressions. Fixes #60578. Release notes (sql change): A new parse_timestamp function can be used to parse absolute timestamp strings in computed column expressions or partial index predicates. 60796: geo: replace GEOS with new WKT parser for EWKT parsing r=otan a=andyyang890 Previously, the `parseEWKT` function used GEOS to parse WKT strings. This was inadequate because GEOS has issues with parsing Z and M dimensions. To address this, a new WKT parser was implemented with goyacc and this patch integrates it into CockroachDB. Refs: #53091 Release note: None 60818: opt: add partition info to the opt catalog r=rytaft a=rytaft Prior to this commit, the opt catalog did not include zone information or prefixes specific to each partition of an index. This commit adds this information since it will be necessary to support locality optimized search in a future commit. Informs #55185 Release note: None Co-authored-by: Andrei Matei <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]> Co-authored-by: Radu Berinde <[email protected]> Co-authored-by: Andy Yang <[email protected]> Co-authored-by: Rebecca Taft <[email protected]>
58422: changefeedccl: support primary key changes r=ajwerner a=ajwerner This PR does a few things. One is that when errors occur due to unsupported schema changes during the execution of a changefeed, relatively poor handling ensues. Ideally we'd allow the changefeed to run its course right up to that unsupported schema change's timestamp and then ensure that we persist the fact that we've processed all of that data. That would permit a user to then restart a changefeed after the unsupported change. There are some edge cases here worth considering related to off-by-ones in the timestamp management. I probably should go through that exercise before merging this PR. The real feature this work is in support of is to allow for changefeeds to successfully navigate changes to a primary index. This PR works and support changes to the primary key of a table that also include column set changes. Release note (enterprise change): Support primary key changes in `CHANGEFEED`. 60825: kv/kvclient: fix ManualRefresh error handling r=ajwerner a=nvanbenschoten Fixes #60760. Fallout from #60567. The refreshed BatchRequest was nil on the error path, which was resulting in a nil-pointer exception. This commit fixes this by passing the original BatchRequest to updateStateLocked, like the TxnCoordSender normally does. 60908: Revert "vendor: bump pebble to 959663f8" r=nvanbenschoten a=nvanbenschoten Informs #60828. This reverts commit d8c3eef. Co-authored-by: Andrew Werner <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]>
`Txn.ManualRefresh` was added in 9135287 as part of cockroachdb#60567. In df1c620, we stopped using it in range merge transactions. This commit removes the API to eliminate complexity from the TxnCoordSender. Epic: None Release note: None
101778: kv: remove Txn.ManualRefresh r=arulajmani a=nvanbenschoten `Txn.ManualRefresh` was added in 9135287 as part of #60567. In df1c620, we stopped using it in range merge transactions. This commit removes the API to eliminate complexity from the TxnCoordSender. Epic: None Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]>
Historically we have not allowed the merge transaction to be pushed. The
reason we disabled this was because of the hazard due to attempting to
refresh reads on the RHS of the merge after the SubsumeRequest has been
sent. The
SubsumeRequest
effectively freezes the RHS until the mergecommits or aborts. In order to side-step this hazard, this change ensures
that nothing should prevent the merge transaction from either committing
or aborting.
kv/kvclient: add ManualRefresh support for transactions
This commit adds support for client-initiated refreshes of transactions.
The implementation is somewhat simplistic in that it hijacks existing
logic that occurs during the sending of a request. This makes the
implementation more uniform with the rest of the client library at
the cost of being somewhat awkward and implicit from a code-reading
perspective.
The motivation for this change is to provide the necessary tools to allow
the merge transaction to get pushed. The adoption follows in the next
commit.
Fixes #59308.
Release note: None