Skip to content

Commit

Permalink
Merge #60553 #60624
Browse files Browse the repository at this point in the history 
60553: opt: inline all filters with virtual columns r=mgartner a=mgartner

#### opt: inline all filters with virtual columns

Previously, `InlineSelectVirtualColumns` inlined virtual column
expressions in filters and pushed those filters below a Project only if
the virtual columns were indexed. Now, all virtual column expressions
are inlined. This allows exploration rules to generate expressions that
utilize partial indexes with predicates that reference virtual columns.

We considered extending `InlineSelectVirtualColumns` to also inline a
virtual columns if they are referenced in partial index predicates.
This would require addition metadata to keep track of the set of such
virtual columns. Virtual columns have already been inlined in the
predicates in the table metadata, so simply fetching the outer columns
of those predicates would not suffice. Because this rule already is
highly selective (only operating on virtual columns), we opted to
simplify the rule to inline all virtual columns instead.

There is no release note because virtual columns are gated behind an
experimental session setting.

Release note: None

#### opt: fix optsteps panic from InlineSelectVirtualColumns

Previously, running the `optsteps` test command panicked with "could not
find path to expr" for queries when `InlineSelectVirtualColumns` was
matched. This commit fixes the panic by eliminating the anti-pattern of
calling a constructor during the pattern-match phase of a normalization
rule.

The `optsteps` test command displays step `n` of optimization by running
the optimizer twice and halting after `n` and `n - 1` rules have been
applied. The difference in the two resulting expressions represents the
change in step `n`. `InlineSelectVirtualColumns` previously called
`ConstructFiltersItem` during the pattern-match phase of the rule, but
only added the constructed filters item to the memo in the replace
phase of the rule. When some normalization rule applied during the call
to `ConstructFiltersItem` was the last rule applied in a step, the
constructed item was not added to the memo because the limit on the
number of rules prevented the replace phase of
`InlineSelectVirtualColumns`. As a result, `optsteps` panicked when it
could not find the normalized `FiltersItem` in the memo.

Release note: None


60624: tree: correctly redact value of into_db r=dt a=dt

Release note (bug fix): fix a bug that would cause the value of the optional into_db parameter to RESTORE to be included in anonymizeds crash reports.

Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: David Taylor <[email protected]>
  • Loading branch information
@mgartner @dt
3 people committed Feb 16, 2021
3 parents 3c4aaab + 2ac4d8c + a5fa284 commit dc90a0a
Show file tree
Hide file tree
Showing 7 changed files with 180 additions and 325 deletions.
12 changes: 12 additions & 0 deletions pkg/sql/opt/norm/general_funcs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -675,6 +675,11 @@ func (c *CustomFuncs) IsFilterFalse(filters memo.FiltersExpr) bool {
return filters.IsFalse()
}

// IsFilterEmpty returns true if filters is empty.
func (c *CustomFuncs) IsFilterEmpty(filters memo.FiltersExpr) bool {
return len(filters) == 0
}

// IsContradiction returns true if the given filter item contains a
// contradiction constraint.
func (c *CustomFuncs) IsContradiction(item *memo.FiltersItem) bool {
Expand All @@ -692,6 +697,13 @@ func (c *CustomFuncs) ConcatFilters(left, right memo.FiltersExpr) memo.FiltersEx
return newFilters
}

// DiffFilters creates new Filters that contains all conditions in left that do
// not exist in right. If right is empty, the original left filters are
// returned.
func (c *CustomFuncs) DiffFilters(left, right memo.FiltersExpr) memo.FiltersExpr {
return left.Difference(right)
}

// RemoveFiltersItem returns a new list that is a copy of the given list, except
// that it does not contain the given search item. If the list contains the item
// multiple times, then only the first instance is removed. If the list does not
Expand Down
172 changes: 20 additions & 152 deletions pkg/sql/opt/norm/inline_funcs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -216,172 +216,39 @@ func (c *CustomFuncs) CanInline(scalar opt.ScalarExpr) bool {
return false
}

// IndexedVirtualColumns returns the set of column IDs in the scanPrivate's
// table that are both virtual computed columns and key columns of a secondary
// index.
//
// TODO(mgartner): Include virtual computed columns that are referenced in
// partial index predicates.
func (c *CustomFuncs) IndexedVirtualColumns(scanPrivate *memo.ScanPrivate) opt.ColSet {
md := c.mem.Metadata()
tab := md.Table(scanPrivate.Table)
tabMeta := md.TableMeta(scanPrivate.Table)

virtualCols := tabMeta.VirtualComputedColumns()
if virtualCols.Empty() {
// Return early if there are no virtual columns to avoid unnecessarily
// calculating the index key columns.
return opt.ColSet{}
}

var indexCols opt.ColSet
for i, n := 0, tab.IndexCount(); i < n; i++ {
indexCols.UnionWith(tabMeta.IndexKeyColumnsMapVirtual(i))
}

return virtualCols.Intersection(indexCols)
// VirtualColumns returns the set of columns in the scanPrivate's table that are
// virtual computed columns.
func (c *CustomFuncs) VirtualColumns(scanPrivate *memo.ScanPrivate) opt.ColSet {
tabMeta := c.mem.Metadata().TableMeta(scanPrivate.Table)
return tabMeta.VirtualComputedColumns()
}

// TryInlineSelectVirtualColumns is used by the InlineSelectVirtualColumns rule
// to find filters on virtual computed columns that can be pushed below the
// Project that produces the virtual column. The filter is pushed-down by
// inlining the virtual column expression. Only the columns in
// indexedVirtualColumns are eligible for inlining.
//
// If successful, a pair of filters is returned; the first containing filters in
// which virtual columns have been inlined, and the second containing filters
// that were unaffected by inlining. If unsuccessful, an empty
// InlinedVirtualColumnFiltersPair is returned, indicating that there is nothing
// to inline.
//
// Two separate filters are returned as a pair so that the
// InlineSelectVirtualColumns rule can push down eligible filters, but not the
// rest.
//
// For example:
//
// CREATE TABLE t (
// a INT,
// b INT,
// v INT AS (abs(a)),
// w INT AS (abs(b)),
// INDEX (v)
// )
//
// SELECT * FROM t WHERE v = 5 AND w = 10
// InlinableVirtualColumnFilters returns a new filters expression containing any
// of the given filters that meet the criteria:
//
// The (v = 5) and (w = 10) filters are returned in separate filters. In the
// first, v is inlined because it is indexed, so the filter is transformed to
// (abs(a) = 5). This filter is no longer dependent on the projection of v, so
// it can be pushed below the Project. The w variable is not indexed, so the
// (w = 10) filter is unchanged and remains above the Project.
// 1. The filter has references to any of the columns in virtualColumns.
// 2. The filter is not a correlated subquery.
//
// See the InlineSelectVirtualColumns rule for more details.
func (c *CustomFuncs) TryInlineSelectVirtualColumns(
filters memo.FiltersExpr, projections memo.ProjectionsExpr, indexedVirtualColumns opt.ColSet,
) InlinedVirtualColumnFiltersPair {
// Collect the projections for the indexed virtual columns.
var inlinableCols opt.ColSet
var inlinableProjections memo.ProjectionsExpr
for i := range projections {
col := projections[i].Col

// If the projected expression is volatile, it cannot be inlined. This
// should be impossible (computed column expressions cannot be
// volatile), but adds additional safety in case this function is
// expanded to inline non-virtual columns in the future.
if projections[i].ScalarProps().VolatilitySet.HasVolatile() {
continue
}

if indexedVirtualColumns.Contains(col) {
// Initialize inlinableProjections lazily.
if inlinableProjections == nil {
inlinableProjections = make(memo.ProjectionsExpr, 0, len(projections)-i)
}
inlinableCols.Add(col)
inlinableProjections = append(inlinableProjections, projections[i])
}
}

if len(inlinableProjections) == 0 {
// None of the projections are eligible for inlining in filters.
return InlinedVirtualColumnFiltersPair{}
}

// For each filter, determine if there are any virtual columns to inline. If
// so, inline the virtual column expressions in the filter. We keep track of
// which filters have inlined projections in inlinedFilterIdxs so that we
// can build a list of non-inlined filters below.
var inlined memo.FiltersExpr
var inlinedFilterIdxs util.FastIntSet
func (c *CustomFuncs) InlinableVirtualColumnFilters(
filters memo.FiltersExpr, virtualColumns opt.ColSet,
) (inlinableFilters memo.FiltersExpr) {
for i := range filters {
item := &filters[i]

// Do not inline a filter if it has a correlated subquery or it does not
// reference an inlinable virtual column.
if item.ScalarProps().HasCorrelatedSubquery || !item.ScalarProps().OuterCols.Intersects(inlinableCols) {
// reference a virtual column.
if item.ScalarProps().HasCorrelatedSubquery || !item.ScalarProps().OuterCols.Intersects(virtualColumns) {
continue
}

// Initialize inlined lazily.
if inlined == nil {
inlined = make(memo.FiltersExpr, 0, len(filters)-i)
}

// Inline the virtual column expressions in the filter.
filter := c.f.ConstructFiltersItem(
c.inlineProjections(item.Condition, inlinableProjections).(opt.ScalarExpr),
)
inlined = append(inlined, filter)
inlinedFilterIdxs.Add(i)
}

if len(inlined) == 0 {
// No projections were inlined in filters.
return InlinedVirtualColumnFiltersPair{}
}

// Collect the filters which did not have projections inlined.
notInlined := make(memo.FiltersExpr, 0, len(filters)-inlinedFilterIdxs.Len())
for i := range filters {
if !inlinedFilterIdxs.Contains(i) {
notInlined = append(notInlined, filters[i])
// Initialize inlinableFilters lazily.
if inlinableFilters == nil {
inlinableFilters = make(memo.FiltersExpr, 0, len(filters)-i)
}
}

return InlinedVirtualColumnFiltersPair{
inlined: inlined,
notInlined: notInlined,
inlinableFilters = append(inlinableFilters, *item)
}
}

// InlinedVirtualColumnFiltersPair is the result of
// TryInlineSelectVirtualColumns. See that function for more details.
type InlinedVirtualColumnFiltersPair struct {
inlined memo.FiltersExpr
notInlined memo.FiltersExpr
}

// InlinedFilters returns the filters in the pair where virtual columns were
// inlined.
func (c *CustomFuncs) InlinedFilters(pair InlinedVirtualColumnFiltersPair) memo.FiltersExpr {
return pair.inlined
}

// NotInlinedFilters returns the filters in the pair where virtual columns were
// not inlined.
func (c *CustomFuncs) NotInlinedFilters(pair InlinedVirtualColumnFiltersPair) memo.FiltersExpr {
return pair.notInlined
}

// InlineSelectVirtualColumnsSucceeded returns true if the pair is not nil,
// indicating that InlineSelectVirtualColumns successfully inlined virtual
// columns into some of the filters.
func (c *CustomFuncs) InlineSelectVirtualColumnsSucceeded(
pair InlinedVirtualColumnFiltersPair,
) bool {
return len(pair.inlined) > 0
return inlinableFilters
}

// InlineSelectProject searches the filter conditions for any variable
Expand Down Expand Up @@ -412,6 +279,7 @@ func (c *CustomFuncs) InlineProjectProject(
newProjections := make(memo.ProjectionsExpr, len(projections))
for i := range projections {
item := &projections[i]

newProjections[i] = c.f.ConstructProjectionsItem(
c.inlineProjections(item.Element, innerProjections).(opt.ScalarExpr),
item.Col,
Expand Down
116 changes: 58 additions & 58 deletions pkg/sql/opt/norm/rules/inline.opt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -159,76 +159,76 @@

# InlineSelectVirtualColumns pushes Select filters referencing virtual columns
# into a Project by inlining the virtual column expressions. This makes the
# Select independent of the Project. Filters with virtual columns are only
# pushed below the Project if those virtual columns are indexed by a secondary
# index. This allows exploration rules to generate plans that use those indexes.
# Non-indexed virtual column filters are not inlined because the virtual column
# Select independent of the Project. Because these filters are pushed below the
# Project, exploration rules that match on the (Select (Scan)) pattern can
# generate plans that use indexes on virtual columns.
#
# Filters on non-virtual projected columns are not inlined because the
# expression would be executed twice (once in the filter and once in the
# projection), adding overhead without any chance of a secondary index being
# used in the optimized plan.
# projection), adding overhead without any chance of a secondary index on a
# virtual column being used in the optimized plan.
#
# Notice that this rule is similar to PushSelectIntoInlinableProject. The key
# difference is that PushSelectIntoInlinableProject only inlines simple
# expressions that will add negligible overhead when computing twice.
# Conversely, InlineSelectVirtualColumns does not discriminate by the type of
# expression. It will inline all virtual columns that are indexed in the hopes
# that inling will lead to a query plan that uses a virtual column index.
# expression. It will inline all virtual columns in the hopes that inlining will
# lead to a query plan that uses a virtual column index.
#
# Also, PushSelectIntoInlinableProject will inline filters if and only if each
# filter is inlinable (by its definition), whereas InlineSelectVirtualColumns
# will split the input filters into two groups: one to inline below the Project,
# and one to leave above the Project.
# Also, PushSelectIntoInlinableProject will inline filters if and only if all of
# the filter items are inlinable (by its definition), whereas
# InlineSelectVirtualColumns will split the input filters into two groups: one
# to inline below the Project, and one to leave above the Project. This allows
# filters on virtual columns to be pushed down in more cases.
#
# For example, consider the table and query:
#
# CREATE TABLE t (
# a INT,
# b INT,
# v INT AS (abs(a)),
# w INT AS (abs(b)),
# v INT AS (abs(a)) VIRTUAL,
# INDEX (v)
# )
# SELECT * FROM t WHERE v = 5 AND w = 10
# SELECT v, w FROM (
# SELECT v, abs(b) AS w FROM t
# ) WHERE v = 5 AND w = 10
#
# The partially normalized expression for the SELECT query before
# InlineSelectVirtualColumns is:
# InlineSelectVirtualColumns is applied is:
#
# project
# ├── columns: a:1 b:2 v:3 w:4
# └── select
# ├── columns: a:1 b:2 v:3 w:4
# ├── project
# │ ├── columns: v:3 w:4 a:1 b:2
# │ ├── scan t
# │ │ └── columns: a:1 b:2
# │ └── projections
# │ ├── abs(a:1) [as=v:3]
# │ └── abs(b:2) [as=w:4]
# └── filters
# ├── v:3 = 5
# └── w:4 = 10
# select
# ├── columns: v:3 w:6
# ├── project
# │ ├── columns: w:6 v:3
# │ ├── scan t
# │ │ └── columns: a:1 b:2
# │ └── projections
# │ ├── abs(b:2) [as=w:6]
# │ └── abs(a:1) [as=v:3]
# └── filters
# ├── v:3 = 5
# └── w:6 = 10
#
# InlineSelectVirtualColumns will push only the (v = 5) filter below the Project
# as (abs(a) = 5) because v is indexed. The (w = 10) filter remains above the
# Project.
# as (abs(a) = 5) because v is a virtual column. The (w = 10) filter remains
# above the Project. Notice the (Select (Scan)) pattern that will allow a
# constrained scan over the secondary index to be generated.
#
# project
# ├── columns: a:1 b:2 v:3 w:4
# └── select
# ├── columns: a:1 b:2 v:3 w:4!null
# ├── project
# │ ├── columns: v:3 w:4 a:1 b:2
# │ ├── select
# │ │ ├── columns: a:1 b:2
# │ │ ├── scan t
# │ │ │ └── columns: a:1 b:2
# │ │ └── filters
# │ │ └── abs(a:1) = 5
# │ └── projections
# │ ├── abs(a:1) [as=v:3]
# │ └── abs(b:2) [as=w:4]
# └── filters
# └── w:4 = 10
# select
# ├── columns: v:3 w:6
# ├── project
# │ ├── columns: w:6 v:3
# │ ├── select
# │ │ ├── columns: a:1 b:2
# │ │ ├── scan t
# │ │ │ └── columns: a:1 b:2
# │ │ └── filters
# │ │ └── abs(a:1) = 5
# │ └── projections
# │ ├── abs(b:2) [as=w:6]
# │ └── abs(a:1) [as=v:3]
# └── filters
# └── w:6 = 10
#
# This rule has no explicit priority so that it runs before
# PushSelectIntoInlinableProject (which is low priority). It must run before
Expand All @@ -244,26 +244,26 @@
)
$filters:* &
^(ColsAreEmpty
$indexedVirtualColumns:(IndexedVirtualColumns
$scanPrivate
)
$virtualColumns:(VirtualColumns $scanPrivate)
) &
(InlineSelectVirtualColumnsSucceeded
$result:(TryInlineSelectVirtualColumns
^(IsFilterEmpty
$inlinableFilters:(InlinableVirtualColumnFilters
$filters
$projections
$indexedVirtualColumns
$virtualColumns
)
)
)
=>
(Select
(Project
(Select $scan (InlinedFilters $result))
(Select
$scan
(InlineSelectProject $inlinableFilters $projections)
)
$projections
$passthrough
)
(NotInlinedFilters $result)
(DiffFilters $filters $inlinableFilters)
)

# InlineProjectInProject folds an inner Project operator into an outer Project
Expand Down
Loading

0 comments on commit dc90a0a

Please sign in to comment.