Skip to content

Commit

Permalink
Merge cockroachdb#67310 cockroachdb#67374 cockroachdb#67430
Browse files Browse the repository at this point in the history 
67310: backupccl: add exportrequest.delay.total metric r=pbardea a=pbardea

This commit adds a metric to track how long export requests are blocked
by the concurrency limiter.

Release note (ops change): Introduce a new metric
(exportrequest.delay.total) to track how long ExportRequests (issued by
BACKUP) are delayed by throttling mechansisms.

67374: opt: improve statistics and cardinality calculation for enum types r=rytaft a=rytaft

Prior to this commit, there were three different places in the code
where we calculated the number of distinct values in a span. This
commit unifies them to remove code duplication and allow all three
places to benefit from improvements to one.

As a result, this commit improves statistics estimation and cardinality
calculation for enum values. This commit also improves statistics estimation
for enum values in cases where table statistics are not available or are
stale. Finally, this commit adds logic to use the column type for
cardinality calculation when building logical properties.

Release note (performance improvement): Improved the optimizer's
cardinality estimations for enum columns, including the `crdb_region` column
in `REGIONAL BY ROW` tables, as well as all other columns with user-defined
types. This may result in the optimizer choosing a better query plan in some
cases.

67430: roachtest: update django blocklist r=rafiss a=RichardJCai

Release note: None

Co-authored-by: Paul Bardea <[email protected]>
Co-authored-by: Rebecca Taft <[email protected]>
Co-authored-by: richardjcai <[email protected]>
  • Loading branch information
@pbardea @rytaft @RichardJCai
4 people committed Jul 9, 2021
4 parents 90f4522 + 49a76ad + d2f8622 + c58c274 commit 64cedaf
Show file tree
Hide file tree
Showing 12 changed files with 245 additions and 119 deletions.
36 changes: 18 additions & 18 deletions pkg/ccl/logictestccl/testdata/logic_test/regional_by_row
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1469,49 +1469,49 @@ LIMIT 1] OFFSET 2
└── • union all
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 30
│ estimated row count: 999,995
│
├── • union all
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 20
│ │ estimated row count: 666,663
│ │
│ ├── • filter
│ │ │ columns: (a, b)
│ │ │ ordering: +a,+b
│ │ │ estimated row count: 10
│ │ │ estimated row count: 333,332
│ │ │ filter: b IS NOT NULL
│ │ │
│ │ └── • scan
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 10 (<0.01% of the table; stats collected <hidden> ago)
│ │ estimated row count: 333,333 (33% of the table; stats collected <hidden> ago)
│ │ table: t56201@key_a_b
│ │ spans: /"@"/!NULL-/"@"/PrefixEnd
│ │
│ └── • filter
│ │ columns: (a, b)
│ │ ordering: +a,+b
│ │ estimated row count: 10
│ │ estimated row count: 333,332
│ │ filter: b IS NOT NULL
│ │
│ └── • scan
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 10 (<0.01% of the table; stats collected <hidden> ago)
│ estimated row count: 333,333 (33% of the table; stats collected <hidden> ago)
│ table: t56201@key_a_b
│ spans: /"\x80"/!NULL-/"\x80"/PrefixEnd
│
└── • filter
│ columns: (a, b)
│ ordering: +a,+b
│ estimated row count: 10
│ estimated row count: 333,332
│ filter: b IS NOT NULL
│
└── • scan
columns: (a, b)
ordering: +a,+b
estimated row count: 10 (<0.01% of the table; stats collected <hidden> ago)
estimated row count: 333,333 (33% of the table; stats collected <hidden> ago)
table: t56201@key_a_b
spans: /"\xc0"/!NULL-/"\xc0"/PrefixEnd

Expand Down Expand Up @@ -1555,31 +1555,31 @@ LIMIT 1] OFFSET 2
└── • union all
│ columns: (b, crdb_region, rowid)
│ ordering: +b
│ estimated row count: 10
│ estimated row count: 333,333
│
├── • union all
│ │ columns: (b, crdb_region, rowid)
│ │ ordering: +b
│ │ estimated row count: 7
│ │ estimated row count: 222,222
│ │
│ ├── • scan
│ │ columns: (b, crdb_region, rowid)
│ │ ordering: +b
│ │ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ │ estimated row count: 111,111 (11% of the table; stats collected <hidden> ago)
│ │ table: t56201@key_b_partial (partial index)
│ │ spans: /"@"/!NULL-/"@"/PrefixEnd
│ │
│ └── • scan
│ columns: (b, crdb_region, rowid)
│ ordering: +b
│ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ estimated row count: 111,111 (11% of the table; stats collected <hidden> ago)
│ table: t56201@key_b_partial (partial index)
│ spans: /"\x80"/!NULL-/"\x80"/PrefixEnd
│
└── • scan
columns: (b, crdb_region, rowid)
ordering: +b
estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
estimated row count: 111,111 (11% of the table; stats collected <hidden> ago)
table: t56201@key_b_partial (partial index)
spans: /"\xc0"/!NULL-/"\xc0"/PrefixEnd

Expand Down Expand Up @@ -1619,31 +1619,31 @@ LIMIT 1] OFFSET 2
└── • union all
│ columns: (c)
│ ordering: +c
│ estimated row count: 10
│ estimated row count: 333,333
│
├── • union all
│ │ columns: (c)
│ │ ordering: +c
│ │ estimated row count: 7
│ │ estimated row count: 222,222
│ │
│ ├── • scan
│ │ columns: (c)
│ │ ordering: +c
│ │ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ │ estimated row count: 111,111 (11% of the table; stats collected <hidden> ago)
│ │ table: t56201@key_c_partial (partial index)
│ │ spans: /"@"-/"@"/PrefixEnd
│ │
│ └── • scan
│ columns: (c)
│ ordering: +c
│ estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
│ estimated row count: 111,111 (11% of the table; stats collected <hidden> ago)
│ table: t56201@key_c_partial (partial index)
│ spans: /"\x80"-/"\x80"/PrefixEnd
│
└── • scan
columns: (c)
ordering: +c
estimated row count: 3 (<0.01% of the table; stats collected <hidden> ago)
estimated row count: 111,111 (11% of the table; stats collected <hidden> ago)
table: t56201@key_c_partial (partial index)
spans: /"\xc0"-/"\xc0"/PrefixEnd

Expand Down
15 changes: 11 additions & 4 deletions pkg/cmd/roachtest/tests/django_blocklist.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -168,15 +168,22 @@ var djangoBlocklists = blocklistsForVersion{
}

// Maintain that this list is alphabetized.
var djangoBlocklist21_2 = blocklist{}
var djangoBlocklist21_2 = djangoBlocklist21_1

var djangoBlocklist21_1 = blocklist{}
var djangoBlocklist21_1 = djangoBlocklist20_2

var djangoBlocklist20_2 = blocklist{}

var djangoIgnoreList21_2 = djangoIgnoreList21_1
var djangoIgnoreList21_2 = blocklist{
"migrations.test_operations.OperationTests.test_alter_fk_non_fk": "will be fixed in django-cockroachdb v3.2.2",
"schema.tests.SchemaTests.test_alter_field_db_collation": "will be fixed in django-cockroachdb v3.2.2",
"schema.tests.SchemaTests.test_alter_field_type_and_db_collation": "will be fixed in django-cockroachdb v3.2.2",
}

var djangoIgnoreList21_1 = djangoIgnoreList20_2
var djangoIgnoreList21_1 = blocklist{
"schema.tests.SchemaTests.test_alter_field_db_collation": "will be fixed in django-cockroachdb v3.2.2",
"schema.tests.SchemaTests.test_alter_field_type_and_db_collation": "will be fixed in django-cockroachdb v3.2.2",
}

var djangoIgnoreList20_2 = blocklist{
"expressions.tests.BasicExpressionsTests.test_boolean_expression_combined": "unknown",
Expand Down
14 changes: 14 additions & 0 deletions pkg/kv/kvserver/metrics.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1051,6 +1051,14 @@ var (
Unit: metric.Unit_NANOSECONDS,
}

// Export request counter.
metaExportEvalTotalDelay = metric.Metadata{
Name: "exportrequest.delay.total",
Help: "Amount by which evaluation of Export requests was delayed",
Measurement: "Nanoseconds",
Unit: metric.Unit_NANOSECONDS,
}

// Encryption-at-rest metrics.
// TODO(mberhault): metrics for key age, per-key file/bytes counts.
metaEncryptionAlgorithm = metric.Metadata{
Expand Down Expand Up @@ -1282,6 +1290,9 @@ type StoreMetrics struct {
AddSSTableProposalTotalDelay *metric.Counter
AddSSTableProposalEngineDelay *metric.Counter

// Export request stats.
ExportRequestProposalTotalDelay *metric.Counter

// Encryption-at-rest stats.
// EncryptionAlgorithm is an enum representing the cipher in use, so we use a gauge.
EncryptionAlgorithm *metric.Gauge
Expand Down Expand Up @@ -1672,6 +1683,9 @@ func newStoreMetrics(histogramWindow time.Duration) *StoreMetrics {
AddSSTableProposalTotalDelay: metric.NewCounter(metaAddSSTableEvalTotalDelay),
AddSSTableProposalEngineDelay: metric.NewCounter(metaAddSSTableEvalEngineDelay),

// ExportRequest proposal.
ExportRequestProposalTotalDelay: metric.NewCounter(metaExportEvalTotalDelay),

// Encryption-at-rest.
EncryptionAlgorithm: metric.NewGauge(metaEncryptionAlgorithm),

Expand Down
1 change: 1 addition & 0 deletions pkg/kv/kvserver/store_send.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,7 @@ func (s *Store) maybeThrottleBatch(
}

waited := timeutil.Since(before)
s.metrics.ExportRequestProposalTotalDelay.Inc(waited.Nanoseconds())
if waited > time.Second {
log.Infof(ctx, "Export request was delayed by %v", waited)
}
Expand Down
49 changes: 5 additions & 44 deletions pkg/sql/opt/constraint/constraint.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ import (

"github.com/cockroachdb/cockroach/pkg/sql/opt"
"github.com/cockroachdb/cockroach/pkg/sql/sem/tree"
"github.com/cockroachdb/cockroach/pkg/sql/types"
"github.com/cockroachdb/errors"
)

Expand Down Expand Up @@ -703,54 +702,16 @@ func (c *Constraint) CalculateMaxResults(
return 0, false
}
if prefix == numCols-1 {
keyCtx := MakeKeyContext(&c.Columns, evalCtx)
// If the prefix does not include the last column, calculate the number of
// distinct values possible in the span. This is only supported for int
// and date types.
// distinct values possible in the span.
for i := 0; i < c.Spans.Count(); i++ {
sp := c.Spans.Get(i)
start := sp.StartKey()
end := sp.EndKey()

// Ensure that the keys specify the last column.
if start.Length() != numCols || end.Length() != numCols {
return 0, false
}

// TODO(asubiotto): This logic is very similar to
// updateDistinctCountsFromConstraint. It would be nice to extract this
// logic somewhere.
colIdx := numCols - 1
startVal := start.Value(colIdx)
endVal := end.Value(colIdx)
var startIntVal, endIntVal int64
if startVal.ResolvedType().Family() == types.IntFamily &&
endVal.ResolvedType().Family() == types.IntFamily {
startIntVal = int64(*startVal.(*tree.DInt))
endIntVal = int64(*endVal.(*tree.DInt))
} else if startVal.ResolvedType().Family() == types.DateFamily &&
endVal.ResolvedType().Family() == types.DateFamily {
startDate := startVal.(*tree.DDate)
endDate := endVal.(*tree.DDate)
if !startDate.IsFinite() || !endDate.IsFinite() {
// One of the boundaries is not finite, so we can't determine the
// distinct count for this column.
return 0, false
}
startIntVal = int64(startDate.PGEpochDays())
endIntVal = int64(endDate.PGEpochDays())
} else {
spanDistinctVals, ok := sp.KeyCount(&keyCtx, numCols)
if !ok {
return 0, false
}

if c.Columns.Get(colIdx).Ascending() {
distinctVals += uint64(endIntVal - startIntVal)
} else {
distinctVals += uint64(startIntVal - endIntVal)
}

// Add one since both start and end boundaries should be inclusive
// (due to Span.PreferInclusive).
distinctVals++
distinctVals += uint64(spanDistinctVals)
}
} else {
distinctVals = uint64(c.Spans.Count())
Expand Down
95 changes: 95 additions & 0 deletions pkg/sql/opt/exec/execbuilder/testdata/enums
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,21 @@ CREATE TYPE greeting AS ENUM ('hello', 'howdy', 'hi');
CREATE TABLE t (x greeting PRIMARY KEY, y greeting, INDEX i (y), FAMILY (x, y));
INSERT INTO t VALUES ('hello', 'howdy'), ('howdy', 'hi')

# Test that we calculate the correct stats and cardinality.
query T
EXPLAIN (OPT,VERBOSE) SELECT * FROM t
----
scan t
├── columns: x:1 y:2
├── check constraint expressions
│ └── x:1 IN ('hello', 'howdy', 'hi') [outer=(1), constraints=(/1: [/'hello' - /'hello'] [/'howdy' - /'howdy'] [/'hi' - /'hi']; tight)]
├── cardinality: [0 - 3]
├── stats: [rows=3]
├── cost: 18.05
├── key: (1)
├── fd: (1)-->(2)
└── prune: (1,2)

query T
EXPLAIN (OPT) SELECT * FROM t WHERE x = 'hello'
----
Expand Down Expand Up @@ -96,6 +111,26 @@ scan checks@checks_x_y_idx
├── [/'hi'/2 - /'hi'/2]
└── [/'cheers'/2 - /'cheers'/2]

# Test that we calculate the correct stats and cardinality.
query T
EXPLAIN (OPT,VERBOSE) SELECT DISTINCT x FROM checks
----
distinct-on
├── columns: x:1
├── grouping columns: x:1
├── internal-ordering: +1
├── cardinality: [0 - 4]
├── stats: [rows=4, distinct(1)=4, null(1)=0]
├── cost: 1115.67
├── key: (1)
└── scan checks@checks_x_y_idx
├── columns: x:1
├── stats: [rows=1000, distinct(1)=4, null(1)=0]
├── cost: 1105.61
├── ordering: +1
├── prune: (1)
└── interesting orderings: (+1)

# Test that a limited, ordered scan is efficient.
statement ok
CREATE TABLE composite_key (x greeting, y INT, PRIMARY KEY (x, y), FAMILY (x, y));
Expand All @@ -120,3 +155,63 @@ limit
│ ├── constraint: /26/27: [/'cheers' - /'cheers']
│ └── limit: 5
└── 5

statement ok
CREATE TABLE nulls (x greeting, y int, INDEX (x, y))

# Test that we calculate the correct stats and cardinality including null values.
query T
EXPLAIN (OPT,VERBOSE) SELECT x FROM nulls WHERE y < 0 UNION SELECT x FROM nulls WHERE y > 10
----
union
├── columns: x:11
├── left columns: nulls.x:1
├── right columns: nulls.x:6
├── internal-ordering: +11
├── cardinality: [0 - 5]
├── stats: [rows=5, distinct(11)=5, null(11)=1]
├── cost: 2278.80667
├── key: (11)
├── interesting orderings: (+11)
├── project
│ ├── columns: nulls.x:1
│ ├── stats: [rows=333.333333, distinct(1)=5, null(1)=3.33333333]
│ ├── cost: 1139.37333
│ ├── ordering: +1
│ ├── interesting orderings: (+1)
│ └── select
│ ├── columns: nulls.x:1 y:2
│ ├── stats: [rows=333.333333, distinct(1)=5, null(1)=3.33333333, distinct(2)=33.3333333, null(2)=0]
│ ├── cost: 1136.03
│ ├── ordering: +1
│ ├── interesting orderings: (+1,+2)
│ ├── scan nulls@nulls_x_y_idx
│ │ ├── columns: nulls.x:1 y:2
│ │ ├── stats: [rows=1000, distinct(1)=5, null(1)=10, distinct(2)=100, null(2)=10]
│ │ ├── cost: 1126.01
│ │ ├── ordering: +1
│ │ ├── prune: (1,2)
│ │ └── interesting orderings: (+1,+2)
│ └── filters
│ └── y:2 < 0 [outer=(2), constraints=(/2: (/NULL - /-1]; tight)]
└── project
├── columns: nulls.x:6
├── stats: [rows=333.333333, distinct(6)=5, null(6)=3.33333333]
├── cost: 1139.37333
├── ordering: +6
├── interesting orderings: (+6)
└── select
├── columns: nulls.x:6 y:7
├── stats: [rows=333.333333, distinct(6)=5, null(6)=3.33333333, distinct(7)=33.3333333, null(7)=0]
├── cost: 1136.03
├── ordering: +6
├── interesting orderings: (+6,+7)
├── scan nulls@nulls_x_y_idx
│ ├── columns: nulls.x:6 y:7
│ ├── stats: [rows=1000, distinct(6)=5, null(6)=10, distinct(7)=100, null(7)=10]
│ ├── cost: 1126.01
│ ├── ordering: +6
│ ├── prune: (6,7)
│ └── interesting orderings: (+6,+7)
└── filters
└── y:7 > 10 [outer=(7), constraints=(/7: [/11 - ]; tight)]
Loading

0 comments on commit 64cedaf

Please sign in to comment.