Skip to content

Commit

Permalink
Merge #66989 #67263
Browse files Browse the repository at this point in the history 
66989: roachtest: check metrics during multi-region TPC-C runs r=ajstorm a=otan

See individual commits for details.

Resolves #65911
Resolves #66953

67263: opt,ccl: eliminate Project inside GroupBy, add tests for validation of partial unique indexes r=rytaft a=rytaft

**opt: add a test-only check for `NullsAreDistinct` in `GroupingPrivate`**

This commit adds a check that `NullsAreDistinct` is true iff the grouping
operator is `UpsertDistinctOn` or `EnsureUpsertDistinctOn`.

Release note: None

**opt: add exploration rule to eliminate `Project` inside `GroupBy`**

This commit updates the exploration rule `EliminateIndexJoinInsideGroupBy`
and renames it to `EliminateIndexJoinOrProjectInsideGroupBy`. The rule now
removes either an `IndexJoin` or `Project` operator if it can be proven that
the removal does not affect the output of the parent grouping operator.

Removal of a `Project` is needed in cases where the partial index predicate
constrains some columns to be constant, and therefore provides those
columns as constant projections. If the projected columns are not actually
needed by the `GroupBy`, however, the `Project` is not necessary and interferes
with other rules matching, such as `SplitGroupByScanIntoUnionScans`.

Informs #65473

Release note (performance improvement): Improved the efficiency of validation
for some partial unique indexes in `REGIONAL BY ROW` tables by improving the
query plan to use all streaming operations.

**ccl: add tests for validation of partial unique indexes**

This commit adds tests to ensure that the expected query plan is used
to validate new partial unique indexes in `REGIONAL BY ROW` tables.

Informs #65473

Release note: None

Co-authored-by: Oliver Tan <[email protected]>
Co-authored-by: Rebecca Taft <[email protected]>
  • Loading branch information
@otan @rytaft
3 people committed Jul 6, 2021
3 parents 1d0eaa9 + 310b9fb + 6424f6b commit 4c39df8
Show file tree
Hide file tree
Showing 19 changed files with 1,512 additions and 280 deletions.
1 change: 1 addition & 0 deletions build/bazelutil/check.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ pkg/kv/kvclient/rangecache/range_cache.go://go:generate mockgen -package=rangeca
pkg/kv/kvclient/rangefeed/rangefeed.go://go:generate mockgen -package=rangefeed -source rangefeed.go -destination=mocks_generated.go .
pkg/kv/kvclient/kvcoord/transport.go://go:generate mockgen -package=kvcoord -destination=mocks_generated.go . Transport
pkg/roachpb/api.go://go:generate mockgen -package=roachpb -destination=mocks_generated.go . InternalClient,Internal_RangeFeedClient
pkg/cmd/roachtest/drt.go://go:generate mockgen -source drt.go -package main -destination drt_generated.go
pkg/security/securitytest/securitytest.go://go:generate go-bindata -mode 0600 -modtime 1400000000 -pkg securitytest -o embedded.go -ignore README.md -ignore regenerate.sh test_certs
pkg/security/securitytest/securitytest.go://go:generate gofmt -s -w embedded.go
pkg/security/securitytest/securitytest.go://go:generate goimports -w embedded.go
Expand Down
3 changes: 3 additions & 0 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -878,6 +878,7 @@ github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCV
github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
Expand Down Expand Up @@ -1043,9 +1044,11 @@ github.com/moby/moby v20.10.6+incompatible/go.mod h1:fDXVQ6+S340veQPv35CzDahGBmH
github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 h1:rzf0wL0CHVc8CEsgyygG0Mn9CNCCPZqOPaz8RiiHYQk=
github.com/moby/term v0.0.0-20201216013528-df9cb8a40635/go.mod h1:FBS0z0QWA44HXygs7VXDUOGoN/1TV3RuWkLO04am3wc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180320133207-05fbef0ca5da/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
github.com/montanaflynn/stats v0.6.3 h1:F8446DrvIF5V5smZfZ8K9nrmmix0AFgevPdLruGOmzk=
Expand Down
210 changes: 191 additions & 19 deletions pkg/ccl/logictestccl/testdata/logic_test/regional_by_row
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1412,67 +1412,239 @@ CREATE INDEX new_idx ON regional_by_row_table(a, b)
# The validation query to create the unique constraint should be efficient (see
# #56201).
statement ok
CREATE TABLE t56201 (a INT, b STRING) LOCALITY REGIONAL BY ROW;
CREATE TABLE t56201 (a INT, b STRING, c STRING NOT NULL) LOCALITY REGIONAL BY ROW;
ALTER TABLE t56201 INJECT STATISTICS '[
{
"columns": ["a"],
"distinct_count": 3,
"row_count": 1000000,
"created_at": "2018-01-01 1:00:00.00000+00:00"
},
{
"columns": ["b"],
"distinct_count": 999900,
"null_count": 5,
"row_count": 1000000,
"created_at": "2018-01-01 1:00:00.00000+00:00"
},
{
"columns": ["c"],
"distinct_count": 999999,
"row_count": 1000000,
"created_at": "2018-01-01 1:00:00.00000+00:00"
}
]';
ALTER TABLE t56201 ADD CONSTRAINT key_a_b UNIQUE (a, b);

query T
SELECT * FROM [EXPLAIN (VERBOSE) SELECT a, b
FROM t56201
WHERE a IS NOT NULL AND b IS NOT NULL
GROUP BY a, b
HAVING count(*) > 1
LIMIT 1] OFFSET 2
----
·
• project
│ columns: (a, b)
│ estimated row count: 1 (missing stats)
│ estimated row count: 1
│
└── • limit
│ columns: (a, b, count_rows)
│ estimated row count: 1 (missing stats)
│ estimated row count: 1
│ count: 1
│
└── • filter
│ columns: (a, b, count_rows)
│ estimated row count: 330 (missing stats)
│ estimated row count: 333,300
│ filter: count_rows > 1
│
└── • group
│ columns: (a, b, count_rows)
│ estimated row count: 991 (missing stats)
│ estimated row count: 999,900
│ aggregate 0: count_rows()
│ group by: a, b
│ ordered: +a,+b
│
└── • union all
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 30 (missing stats)
│ estimated row count: 30
│
├── • union all
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 20 (missing stats)
│ │ estimated row count: 20
│ │
│ ├── • filter
│ │ │ columns: (a, b)
│ │ │ ordering: +a,+b
│ │ │ estimated row count: 10
│ │ │ filter: b IS NOT NULL
│ │ │
│ │ └── • scan
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 10 (<0.01% of the table; stats collected <hidden> ago)
│ │ table: t56201@key_a_b
│ │ spans: /"@"/!NULL-/"@"/PrefixEnd
│ │
│ └── • filter
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 10
│ │ filter: b IS NOT NULL
│ │
│ └── • scan
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 10 (<0.01% of the table; stats collected <hidden> ago)
│ table: t56201@key_a_b
│ spans: /"\x80"/!NULL-/"\x80"/PrefixEnd
│
└── • filter
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 10
│ filter: b IS NOT NULL
│
└── • scan
columns: (a, b)
ordering: +a,+b
estimated row count: 10 (<0.01% of the table; stats collected <hidden> ago)
table: t56201@key_a_b
spans: /"\xc0"/!NULL-/"\xc0"/PrefixEnd

statement ok
CREATE UNIQUE INDEX key_b_partial ON t56201 (b) WHERE a > 0;

query T
SELECT * FROM [EXPLAIN (VERBOSE) SELECT b
FROM t56201@key_b_partial
WHERE b IS NOT NULL AND a > 0
GROUP BY b
HAVING count(*) > 1
LIMIT 1] OFFSET 2
----
·
• project
│ columns: (b)
│ estimated row count: 1
│
└── • limit
│ columns: (b, count_rows)
│ estimated row count: 1
│ count: 1
│
└── • filter
│ columns: (b, count_rows)
│ estimated row count: 111,111
│ filter: count_rows > 1
│
└── • group
│ columns: (b, count_rows)
│ estimated row count: 333,333
│ aggregate 0: count_rows()
│ group by: b
│ ordered: +b
│
└── • project
│ columns: (b)
│ ordering: +b
│
└── • union all
│ columns: (b, crdb_region, rowid)
│ ordering: +b
│ estimated row count: 10
│
├── • union all
│ │ columns: (b, crdb_region, rowid)
│ │ ordering: +b
│ │ estimated row count: 7
│ │
│ ├── • scan
│ │ columns: (b, crdb_region, rowid)
│ │ ordering: +b
│ │ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ │ table: t56201@key_b_partial (partial index)
│ │ spans: /"@"/!NULL-/"@"/PrefixEnd
│ │
│ └── • scan
│ columns: (b, crdb_region, rowid)
│ ordering: +b
│ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ table: t56201@key_b_partial (partial index)
│ spans: /"\x80"/!NULL-/"\x80"/PrefixEnd
│
└── • scan
columns: (b, crdb_region, rowid)
ordering: +b
estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
table: t56201@key_b_partial (partial index)
spans: /"\xc0"/!NULL-/"\xc0"/PrefixEnd

statement ok
CREATE UNIQUE INDEX key_c_partial ON t56201 (c) WHERE a = 1;

query T
SELECT * FROM [EXPLAIN (VERBOSE) SELECT c
FROM t56201
WHERE c IS NOT NULL AND a = 1
GROUP BY c
HAVING count(*) > 1
LIMIT 1] OFFSET 2
----
·
• project
│ columns: (c)
│ estimated row count: 1
│
└── • limit
│ columns: (c, count_rows)
│ estimated row count: 1
│ count: 1
│
└── • filter
│ columns: (c, count_rows)
│ estimated row count: 111,111
│ filter: count_rows > 1
│
└── • group
│ columns: (c, count_rows)
│ estimated row count: 333,333
│ aggregate 0: count_rows()
│ group by: c
│ ordered: +c
│
└── • union all
│ columns: (c)
│ ordering: +c
│ estimated row count: 10
│
├── • union all
│ │ columns: (c)
│ │ ordering: +c
│ │ estimated row count: 7
│ │
│ ├── • scan
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 10 (missing stats)
│ │ table: t56201@key_a_b
│ │ columns: (c)
│ │ ordering: +c
│ │ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ │ table: t56201@key_c_partial (partial index)
│ │ spans: /"@"-/"@"/PrefixEnd
│ │
│ └── • scan
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 10 (missing stats)
│ table: t56201@key_a_b
│ columns: (c)
│ ordering: +c
│ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ table: t56201@key_c_partial (partial index)
│ spans: /"\x80"-/"\x80"/PrefixEnd
│
└── • scan
columns: (a, b)
ordering: +a,+b
estimated row count: 10 (missing stats)
table: t56201@key_a_b
columns: (c)
ordering: +c
estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
table: t56201@key_c_partial (partial index)
spans: /"\xc0"-/"\xc0"/PrefixEnd

statement ok
Expand Down
10 changes: 10 additions & 0 deletions pkg/cmd/roachtest/BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ go_library(
"django.go",
"django_blocklist.go",
"drop.go",
"drt.go",
"drt_generated.go",
"encryption.go",
"engine_switch.go",
"event_log.go",
Expand Down Expand Up @@ -187,10 +189,14 @@ go_library(
"@com_github_cockroachdb_ttycolor//:ttycolor",
"@com_github_codahale_hdrhistogram//:hdrhistogram",
"@com_github_dustin_go_humanize//:go-humanize",
"@com_github_golang_mock//gomock",
"@com_github_kr_pretty//:pretty",
"@com_github_lib_pq//:pq",
"@com_github_nlopes_slack//:slack",
"@com_github_petermattis_goid//:goid",
"@com_github_prometheus_client_golang//api",
"@com_github_prometheus_client_golang//api/prometheus/v1:prometheus",
"@com_github_prometheus_common//model",
"@com_github_shopify_sarama//:sarama",
"@com_github_shopify_toxiproxy//client",
"@com_github_spf13_cobra//:cobra",
Expand All @@ -211,6 +217,7 @@ go_test(
size = "small",
srcs = [
"blocklist_test.go",
"drt_test.go",
"tpcc_test.go",
"z_cluster_test.go",
"z_test_registry_test.go",
Expand All @@ -222,6 +229,7 @@ go_test(
"//pkg/cmd/roachtest/cluster",
"//pkg/cmd/roachtest/logger",
"//pkg/cmd/roachtest/option",
"//pkg/cmd/roachtest/prometheus",
"//pkg/cmd/roachtest/registry",
"//pkg/cmd/roachtest/spec",
"//pkg/cmd/roachtest/test",
Expand All @@ -231,8 +239,10 @@ go_test(
"//pkg/util/timeutil",
"//pkg/util/version",
"@com_github_cockroachdb_errors//:errors",
"@com_github_golang_mock//gomock",
"@com_github_google_go_github//github",
"@com_github_kr_pretty//:pretty",
"@com_github_prometheus_common//model",
"@com_github_stretchr_testify//assert",
"@com_github_stretchr_testify//require",
"@org_golang_x_oauth2//:oauth2",
Expand Down
Loading

0 comments on commit 4c39df8

Please sign in to comment.