-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
jepsen-monotonic failure when using RootTxn on the gateway #41424
Comments
I confirm this bug is present in also I have written a simplified version of the jepsen test in Go suitable for monotonic_test.zip (place in Now also testing 2.1 for good measure. |
Confirming bug present in 2.1 too. |
What do you mean by the bug being present in older releases? Just that we're looking at the wrong Transaction object, or is the monotonic test failing there too? If the latter, do you understand why this test is failing while the jepsen test has never (to my knowledge) hit this before on older versions? |
Both
Yes I understand it. The Jepsen test separates the phases in a multi-statement txn, using standalone SELECTs first to find the max and cluster timestamp, and then a separate INSERT ... VALUES for the mutation. These standalone SELECTs are using LeafTxn objects since at least 2.1 (ever since we use distsql to serve them, which may have been 2.0), so were never able to auto-refresh (instead any conflict would have caused a client-side retry) In order to tickle the bug, you need a read that encounters a refresh error on a RootTxn object with a mutation that also gets a call to INSERT INTO t.t%d(v, ts) SELECT max(x)+1, if(pg_sleep(0.005*random()), cluster_logical_timestamp(), 0)
FROM (VALUES
((SELECT max(v) FROM t.t1)),
((SELECT max(v) FROM t.t2))
) AS x(x) This triggers the bug because in 2.1 19.1 and current I'd say client apps that only use INSERT ... VALUES are unaffected. However an UPDATE combined with cluster_logical_timestamp() is likely to be affected since 2.1 too. |
Andrei tells me that it's not only queries that use |
The theory about us looking at However, I think the real problem is that
|
Before this patch, the txn.CommitTimestamp() function (which powers the cluster_logical_timestamp(), among others) was failing to take into consideration possible refreshes that might have happened since the current epoch began (i.e. txn.RefreshedTimestamp). Fixes cockroachdb#41424 Release note (bug fix): Fix a bug causing the cluster_logical_timestamp() function to sometimes return incorrect results.
Ah, that makes sense. Originally refreshes were only possible at the end of the transaction, but early refreshes/pushes are now possible. |
Before this patch, the txn.CommitTimestamp() function (which powers the cluster_logical_timestamp(), among others) was failing to take into consideration possible refreshes that might have happened since the current epoch began (i.e. txn.RefreshedTimestamp). Fixes cockroachdb#41424 Release note (bug fix): Fix a bug causing the cluster_logical_timestamp() function to sometimes return incorrect results.
41323: sql: bevy of fixes for ActiveRecord compatibility r=jordanlewis a=jordanlewis - improve efficiency of `'foo'::REGCLASS` - improve efficiency of `pg_get_coldesc` - parse INTERVAL(6) as a no-op meaning INTERVAL - bugfix to generate_subscripts for oidvector and int2vector - add pg_available_extensions - fix `pg_get_indexdef` and `pg_attrdef.adbin` to be more Postgres compatible With these fixes, 90% of the fork in `activerecord-cockroachdb-adapter` can go away. 41438: kv: fix cluster_logical_timestamp() r=andreimatei a=andreimatei Before this patch, the txn.CommitTimestamp() function (which powers the cluster_logical_timestamp(), among others) was failing to take into consideration possible refreshes that might have happened since the current epoch began (i.e. txn.RefreshedTimestamp). Fixes #41424 Release note (bug fix): Fix a bug causing the cluster_logical_timestamp() function to sometimes return incorrect results. Co-authored-by: Jordan Lewis <[email protected]> Co-authored-by: Andrei Matei <[email protected]>
Before this patch, the txn.CommitTimestamp() function (which powers the cluster_logical_timestamp(), among others) was failing to take into consideration possible refreshes that might have happened since the current epoch began (i.e. txn.RefreshedTimestamp). Fixes cockroachdb#41424 Release note (bug fix): Fix a bug causing the cluster_logical_timestamp() function to sometimes return incorrect results.
Before this patch, the txn.CommitTimestamp() function (which powers the cluster_logical_timestamp(), among others) was failing to take into consideration possible refreshes that might have happened since the current epoch began (i.e. txn.RefreshedTimestamp). Fixes cockroachdb#41424 Release note (bug fix): Fix a bug causing the cluster_logical_timestamp() function to sometimes return incorrect results.
41439: release-19.2: kv: fix cluster_logical_timestamp() r=andreimatei a=andreimatei Backport 1/1 commits from #41438. /cc @cockroachdb/release --- Before this patch, the txn.CommitTimestamp() function (which powers the cluster_logical_timestamp(), among others) was failing to take into consideration possible refreshes that might have happened since the current epoch began (i.e. txn.RefreshedTimestamp). Fixes #41424 Release note (bug fix): Fix a bug causing the cluster_logical_timestamp() function to sometimes return incorrect results. Co-authored-by: Andrei Matei <[email protected]>
#41307 changed how DistSQL uses transactions. It's first commit 29e18f0 makes a seemingly innocuous change about using RootTxns on the gateway more often than before. This change causes the
monotonic
jepsen test to fail. It fails about 50% with no nemesis, and always with some of the nemeses (e.g.majority-ring
).The test essentially uses two tables and runs transactions that read the maximum from both tables, increment by one, and write to one of the tables that maximum and the
cluster_logical_timestamp()
. It then verifies that both the maximum values and the timestamps increase monotonically. But lo and behold, they're no longer monotonic.I have a theory about the cause: I believe it's because of this check intended to prevent refreshes when
cluster_logical_timestamp()
(or other things that lock a txn's timestamp) has been usedcockroach/pkg/roachpb/data.go
Line 1284 in e3346c1
I believe the check is looking at the wrong
txn
proto. It's looking at a proto coming from apErr
, but it's unlikely for that one to have theOrigTimestampWasObserved
bit set. That bit is probably only set on the client, not the server.If this theory is correct, this means that we've always had this bug, but #41307 exposed it more broadly. It used to be the case that only queries forced to be planned entirely on the gateway (e.g. mutations) would use the
RootTxn
, others would use theLeafTxn
. Leaves don't do refreshes, and so they're safe (and suboptimal when a refresh would help).Besides regular queries, schema changes used to use the
OrigTimestampWasObserved
bit too, so I think those probably are at risk in 19.1 and below.If this theory is correct, this use of the wrong txn object steams from the fact that we put transactions in
pErrs
, for maximal confusion. It's always been a pet peeve of mine:#18919 (comment)
The text was updated successfully, but these errors were encountered: