sql/tests: TestRandomSyntaxSQLSmith failed #63794
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
Comments
cockroach-teamcity
added
branch-master
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 3aad3f8a372090ddaedad0ad58a945fb399ddbc6: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Internal log
/cc @rafiss |
|
The latest failure is caused by this query: It caused this server panic: And these logs This is the schema However, when I tried to reproduce locally, I instead got a handled error instead of a panic cc @rytaft -- i'll put this on the SQL Queries board for further triage |
|
Thanks, @rafiss! I can reproduce locally using a logictest (the assertion is only triggered in test builds): |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 4dc05cceea254854317a9374a4b88df06c0946a6: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 66a1657247f8e1cd84ce22a32321fda6352d789f: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 5f40d691ef9caf910f97fbad53802000c0ca72eb: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 9e821aa456bfbe8b5ec3a8ef79ae3954d52cb675: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 3cd29547884feb24b150b67ca2a88073798a35fd: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 2d7f9a08bac49040a6d77c5465e2b4fd8fdac1f4: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 2dbd33ce319439bc424a52896a03e0a7ccff07f1: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ fd2ab675441d1928650a0e3ed7eb3aa0395f51b2: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 9ae5896d1944df379cc8005d597d0e89d710da81: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 9f198e190812bf301e8a2126cf43cd78e1cfb163: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 9f198e190812bf301e8a2126cf43cd78e1cfb163: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 9f198e190812bf301e8a2126cf43cd78e1cfb163: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 7d55af4a45bccf631c5afe1b7466523453e7cef8: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
|
I've reduced the reproduction to this: |
|
The bug was introduced in f5a6fb0 |
|
The |
|
Full optsteps output here: https://gist.github.com/mgartner/6597d7fc811960cb6fc1da15ed0996e1 This particular step looks suspicious. Before the rule is applied there is a filter with --------------------------------------------------------------------------------
HoistProjectFromInnerJoin (higher cost)
--------------------------------------------------------------------------------
project
├── columns: a:1!null
├── immutable
└── inner-join (hash)
├── columns: a:1!null b:4!null c:5!null b:8!null c:9!null b:12!null c:13!null
├── immutable
├── fd: (4)-->(5), (8)-->(9), (4)==(1,8,12,13), (8)==(1,4,12,13), (5)==(9), (9)==(5), (1)==(4,8,12,13), (12)==(1,4,8,13), (13)==(1,4,8,12)
├── project
│ ├── columns: c:5 b:4
│ ├── immutable
│ ├── fd: (4)-->(5)
│ ├── scan bc
│ │ ├── columns: b:4
│ │ └── computed column expressions
│ │ └── c:5
│ │ └── b:4 % 2
│ └── projections
│ └── b:4 % 2 [as=c:5, outer=(4), immutable]
- ├── inner-join (hash)
- │ ├── columns: a:1!null b:8!null c:9 b:12!null c:13!null
+ ├── project
+ │ ├── columns: c:9 a:1!null b:8!null b:12!null c:13!null
│ ├── immutable
│ ├── fd: (8)-->(9), (8)==(1,12,13), (12)==(1,8,13), (13)==(1,8,12), (1)==(8,12,13)
│ ├── inner-join (hash)
- │ │ ├── columns: b:8!null c:9 b:12!null c:13!null
+ │ │ ├── columns: a:1!null b:8!null b:12!null c:13!null
│ │ ├── immutable
- │ │ ├── fd: (8)-->(9), (8)==(12,13), (12)==(8,13), (13)==(8,12)
- │ │ ├── project
- │ │ │ ├── columns: c:9 b:8
+ │ │ ├── fd: (12)-->(13), (8)==(1,12), (12)==(1,8), (1)==(8,12)
+ │ │ ├── scan a
+ │ │ │ └── columns: a:1
+ │ │ ├── inner-join (merge)
+ │ │ │ ├── columns: b:8!null b:12!null c:13!null
+ │ │ │ ├── left ordering: +12
+ │ │ │ ├── right ordering: +8
│ │ │ ├── immutable
- │ │ │ ├── fd: (8)-->(9)
- │ │ │ ├── scan bc
+ │ │ │ ├── fd: (12)-->(13), (8)==(12), (12)==(8)
+ │ │ │ ├── project
+ │ │ │ │ ├── columns: c:13!null b:12!null
+ │ │ │ │ ├── immutable
+ │ │ │ │ ├── fd: (12)-->(13)
+ │ │ │ │ ├── ordering: +12
+ │ │ │ │ ├── select
+ │ │ │ │ │ ├── columns: b:12!null
+ │ │ │ │ │ ├── immutable
+ │ │ │ │ │ ├── ordering: +12
+ │ │ │ │ │ ├── scan bc@secondary
+ │ │ │ │ │ │ ├── columns: b:12!null
+ │ │ │ │ │ │ ├── constraint: /12/14: (/NULL - ]
+ │ │ │ │ │ │ └── ordering: +12
+ │ │ │ │ │ └── filters
+ │ │ │ │ │ └── b:12 = (b:12 % 2) [outer=(12), immutable, constraints=(/12: (/NULL - ])]
+ │ │ │ │ └── projections
+ │ │ │ │ └── b:12 % 2 [as=c:13, outer=(12), immutable]
+ │ │ │ ├── scan bc@secondary
│ │ │ │ ├── columns: b:8
- │ │ │ │ └── computed column expressions
- │ │ │ │ └── c:9
- │ │ │ │ └── b:8 % 2
- │ │ │ └── projections
- │ │ │ └── b:8 % 2 [as=c:9, outer=(8), immutable]
- │ │ ├── project
- │ │ │ ├── columns: c:13 b:12
- │ │ │ ├── immutable
- │ │ │ ├── fd: (12)-->(13)
- │ │ │ ├── scan bc
- │ │ │ │ ├── columns: b:12
- │ │ │ │ └── computed column expressions
- │ │ │ │ └── c:13
- │ │ │ │ └── b:12 % 2
- │ │ │ └── projections
- │ │ │ └── b:12 % 2 [as=c:13, outer=(12), immutable]
+ │ │ │ │ └── ordering: +8
+ │ │ │ └── filters (true)
│ │ └── filters
- │ │ ├── b:8 = c:13 [outer=(8,13), constraints=(/8: (/NULL - ]; /13: (/NULL - ]), fd=(8)==(13), (13)==(8)]
- │ │ └── b:8 = b:12 [outer=(8,12), constraints=(/8: (/NULL - ]; /12: (/NULL - ]), fd=(8)==(12), (12)==(8)]
- │ ├── scan a
- │ │ └── columns: a:1
- │ └── filters
- │ └── a:1 = b:8 [outer=(1,8), constraints=(/1: (/NULL - ]; /8: (/NULL - ]), fd=(1)==(8), (8)==(1)]
+ │ │ └── a:1 = b:8 [outer=(1,8), constraints=(/1: (/NULL - ]; /8: (/NULL - ]), fd=(1)==(8), (8)==(1)]
+ │ └── projections
+ │ └── b:8 % 2 [as=c:9, outer=(8), immutable]
└── filters
├── b:4 = b:8 [outer=(4,8), constraints=(/4: (/NULL - ]; /8: (/NULL - ]), fd=(4)==(8), (8)==(4)]
└── c:5 = c:9 [outer=(5,9), constraints=(/5: (/NULL - ]; /9: (/NULL - ]), fd=(5)==(9), (9)==(5)] |
|
This might be where the error comes from. Notice the new ordering --------------------------------------------------------------------------------
GenerateMergeJoins (higher cost)
--------------------------------------------------------------------------------
project
├── columns: a:1!null
├── immutable
└── inner-join (hash)
├── columns: a:1!null b:4!null c:5!null b:8!null c:9!null b:12!null c:13!null
├── immutable
├── fd: (4)-->(5), (8)-->(9), (4)==(1,8,12,13), (8)==(1,4,12,13), (5)==(9), (9)==(5), (1)==(4,8,12,13), (12)==(1,4,8,13), (13)==(1,4,8,12)
├── project
│ ├── columns: c:5 b:4
│ ├── immutable
│ ├── fd: (4)-->(5)
│ ├── scan bc
│ │ ├── columns: b:4
│ │ └── computed column expressions
│ │ └── c:5
│ │ └── b:4 % 2
│ └── projections
│ └── b:4 % 2 [as=c:5, outer=(4), immutable]
- ├── project
- │ ├── columns: c:9 a:1!null b:8!null b:12!null c:13!null
+ ├── inner-join (merge)
+ │ ├── columns: a:1!null b:8!null c:9 b:12!null c:13!null
+ │ ├── left ordering: +8
+ │ ├── right ordering: +1
│ ├── immutable
│ ├── fd: (8)-->(9), (8)==(1,12,13), (12)==(1,8,13), (13)==(1,8,12), (1)==(8,12,13)
- │ ├── inner-join (hash)
- │ │ ├── columns: a:1!null b:8!null b:12!null c:13!null
+ │ ├── project
+ │ │ ├── columns: c:9 b:8!null b:12!null c:13!null
│ │ ├── immutable
- │ │ ├── fd: (12)-->(13), (8)==(1,12), (12)==(1,8), (1)==(8,12)
- │ │ ├── scan a
- │ │ │ └── columns: a:1
+ │ │ ├── fd: (8)-->(9), (8)==(12,13), (12)==(8,13), (13)==(8,12)
+ │ │ ├── ordering: +(8|12|13) [actual: +12]
│ │ ├── inner-join (merge)
│ │ │ ├── columns: b:8!null b:12!null c:13!null
│ │ │ ├── left ordering: +12
│ │ │ ├── right ordering: +8
│ │ │ ├── immutable
│ │ │ ├── fd: (12)-->(13), (8)==(12), (12)==(8)
+ │ │ │ ├── ordering: +(8|12|13) [actual: +12]
│ │ │ ├── project
│ │ │ │ ├── columns: c:13!null b:12!null
│ │ │ │ ├── immutable
│ │ │ │ ├── fd: (12)-->(13)
│ │ │ │ ├── ordering: +12
│ │ │ │ ├── select
│ │ │ │ │ ├── columns: b:12!null
│ │ │ │ │ ├── immutable
│ │ │ │ │ ├── ordering: +12
│ │ │ │ │ ├── scan bc@secondary
│ │ │ │ │ │ ├── columns: b:12!null
│ │ │ │ │ │ ├── constraint: /12/14: (/NULL - ]
│ │ │ │ │ │ └── ordering: +12
│ │ │ │ │ └── filters
│ │ │ │ │ └── b:12 = (b:12 % 2) [outer=(12), immutable, constraints=(/12: (/NULL - ])]
│ │ │ │ └── projections
│ │ │ │ └── b:12 % 2 [as=c:13, outer=(12), immutable]
│ │ │ ├── scan bc@secondary
│ │ │ │ ├── columns: b:8
│ │ │ │ └── ordering: +8
│ │ │ └── filters (true)
- │ │ └── filters
- │ │ └── a:1 = b:8 [outer=(1,8), constraints=(/1: (/NULL - ]; /8: (/NULL - ]), fd=(1)==(8), (8)==(1)]
- │ └── projections
- │ └── b:8 % 2 [as=c:9, outer=(8), immutable]
+ │ │ └── projections
+ │ │ └── b:8 % 2 [as=c:9, outer=(8), immutable]
+ │ ├── sort
+ │ │ ├── columns: a:1
+ │ │ ├── ordering: +1
+ │ │ └── scan a
+ │ │ └── columns: a:1
+ │ └── filters (true)
└── filters
├── b:4 = b:8 [outer=(4,8), constraints=(/4: (/NULL - ]; /8: (/NULL - ]), fd=(4)==(8), (8)==(4)]
└── c:5 = c:9 [outer=(5,9), constraints=(/5: (/NULL - ]; /9: (/NULL - ]), fd=(5)==(9), (9)==(5)] |
|
It looks like |
|
I'm fairly stumped on this one, but still making some slow progress. @RaduBerinde @rytaft let me know if you have any ideas here that might help. It looks like the diff I posted above of It derives an interesting ordering of Later, I think It could be that |
|
The key issue is that we end up with a plan where we first Select with filter "b:12 = (b:12+1)" and then we Project "(b:12+1)". This is equivalent to first projecting and selecting (perhaps via a join condition) which is what the normalized plan does but we can't derive the same FDs. Unfortunately, there will always be differences like this because the FD determination can't be perfect. This is problematic given that we require the OrderingChoice groups to be equivalent. I think that (other than the assertion) in practice we would be ok because the columns are actually equal. I think we can fix by checking for this in --- a/pkg/sql/opt/props/physical/ordering_choice.go
+++ b/pkg/sql/opt/props/physical/ordering_choice.go
@@ -520,7 +520,8 @@ func (oc *OrderingChoice) CanSimplify(fdset *props.FuncDepSet) bool {
}
// Check whether new equivalent columns can be added by the FD set.
- equiv := fdset.ComputeEquivClosure(group.Group)
+ equiv := fdset.ComputeEquivGroup(group.AnyID())
if !equiv.Equals(group.Group) {
return true
}
@@ -575,7 +576,9 @@ func (oc *OrderingChoice) Simplify(fdset *props.FuncDepSet) {
}
// Expand group with equivalent columns from FD set.
- group.Group = fdset.ComputeEquivClosure(group.Group)
+ group.Group = fdset.ComputeEquivGroup(group.AnyID())
// Add this group's columns and find closure with the new columns.
closure = closure.Union(group.Group)We'd need to document that Simplify can "cut both ways" - it can reduce an equality group if it doesn't jibe with the FDs. |
Wow, that seems to work beautifully! Thanks for the help! I'll put up a PR. |
mgartner
added a commit
to mgartner/cockroach
that referenced
this issue
Apr 28, 2021
Previously, `OrderingChoice.Simplify` would add but never remove columns from ordering column groups based on equivalency in an FD. In rare cases, this could cause the optimizer to generate expressions which violated an invariant that all columns in an ordering column group are equivalent according to the expression's FD. Violation of this invariant only panics in test builds, and in the test cases found that trigger this panic, there is likely no correctness issues with the expression. Therefore, there was probably no impact in any release builds. This commit updates `OrderingChoice.Simplify` so that non-equivalent columns in an ordering column group are removed from the group, satisfying the invariant. Fixes cockroachdb#63794 Release note: None
|
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ 852e22a2ad78b1a19bd4f504f7079274c0c95fe5: Reproduce
To reproduce, try: make stressrace TESTS=TestRandomSyntaxSQLSmith PKG=./pkg/sql/tests TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1Same failure on other branches
Internal log
/cc @rafiss |
mgartner
added a commit
to mgartner/cockroach
that referenced
this issue
Apr 28, 2021
Previously, `OrderingChoice.Simplify` would add but never remove columns from ordering column groups based on equivalency in an FD. In rare cases, this could cause the optimizer to generate expressions which violated an invariant that all columns in an ordering column group are equivalent according to the expression's FD. Violation of this invariant only panics in test builds, and in the test cases found that trigger this panic, there is likely no correctness issues with the expression. Therefore, there was probably no impact in any release builds. This commit updates `OrderingChoice.Simplify` so that non-equivalent columns in an ordering column group are removed from the group, satisfying the invariant. Fixes cockroachdb#63794 Release note: None
craig bot
pushed a commit
that referenced
this issue
Apr 28, 2021
64253: builtins: implement ST_Envelope(box2d) r=sumeerbhola a=otan Release note (sql change): Implement ST_Envelope for Box2D. 64265: kvserver: speed up intent resolution for aborted txns r=sumeerbhola a=erikgrinaker ### kvserver: speed up intent resolution for aborted txns Cleaning up intents for aborted txns during `EndTxn` could be very slow for intents that had already been removed by a concurrent process, due to suboptimal iterator reuse. This patch changes this intent resolution path to instead create a new iterator with `Prefix:true` for each intent rather than seeking a reusable iterator. This reduces intent resolution time by two orders of magnitude (~200s → ~1s in tests). A simple performance regression test has been added for this. Resolves #64092. Release note (performance improvement): improved intent cleanup performance for aborted transactions. ### intentresolver: reduce ranged resolution batch size This patch adds the constants `intentResolverRangeBatchSize` and `intentResolverRangeRequestSize` to control the number of requests and number of intents per request for ranged intent resolution. It also reduces the number of range requests per batch from 100 to 10, since ranged requests can fan out to hit 200 intents each (via range scans) which is significantly more expensive than single-intent requests. Release note: None /cc @cockroachdb/kv 64324: kvserver: synchronize replica removal with read-only requests r=tbg,nvanbenschoten a=erikgrinaker Replica removal did not synchronize with in-flight read-only requests, which could cause them to be evaluated on a removed (empty) replica, returning an empty result. This patch fixes the problem by locking `Replica.readOnlyCmdMu` during replica removal, thus either waiting for read-only requests to complete or not evaluating them. Resolves #64325. Release note (bug fix): Fixed a race condition where read-only requests during replica removal (e.g. during range merges or rebalancing) could be evaluated on the removed replica, returning an empty result. /cc @cockroachdb/kv We lock `readOnlyCmdMu` during `removeUninitializedReplicaRaftMuLocked` and `tryGetOrCreateReplica` as well, even though I don't believe reads can be routed to these replicas, following review comments. 64342: opt: remove non-equivalent group columns in OrderingChoice.Simplify r=mgartner a=mgartner Previously, `OrderingChoice.Simplify` would add but never remove columns from ordering column groups based on equivalency in an FD. In rare cases, this could cause the optimizer to generate expressions which violated an invariant that all columns in an ordering column group are equivalent according to the expression's FD. Violation of this invariant only panics in test builds, and in the test cases found that trigger this panic, there is likely no correctness issues with the expression. Therefore, there was probably no impact in any release builds. This commit updates `OrderingChoice.Simplify` so that non-equivalent columns in an ordering column group are removed from the group, satisfying the invariant. Fixes #63794 Release note: None Co-authored-by: Oliver Tan <[email protected]> Co-authored-by: Erik Grinaker <[email protected]> Co-authored-by: Marcus Gartner <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
sql/tests.TestRandomSyntaxSQLSmith failed with artifacts on master @ e29d694755e4800f07df139137c71cf741829dd0:
Reproduce
To reproduce, try:
Internal log
/cc @rafiss
This test on roachdash | Improve this report!
The text was updated successfully, but these errors were encountered: