Skip to content

Commit

Permalink
opt: fix optsteps panic from InlineSelectVirtualColumns
Browse files Browse the repository at this point in the history 
Previously, running the `optsteps` test command panicked with "could not
find path to expr" for queries when `InlineSelectVirtualColumns` was
matched. This commit fixes the panic by eliminating the anti-pattern of
calling a constructor during the pattern-match phase of a normalization
rule.

The `optsteps` test command displays step `n` of optimization by running
the optimizer twice and halting after `n` and `n - 1` rules have been
applied. The difference in the two resulting expressions represents the
change in step `n`. `InlineSelectVirtualColumns` previously called
`ConstructFiltersItem` during the pattern-match phase of the rule, but
only added the constructed filters item to the memo in the replace
phase of the rule. When some normalization rule applied during the call
to `ConstructFiltersItem` was the last rule applied in a step, the
constructed item was not added to the memo because the limit on the
number of rules prevented the replace phase of
`InlineSelectVirtualColumns`. As a result, `optsteps` panicked when it
could not find the normalized `FiltersItem` in the memo.

Release note: None
  • Loading branch information
@mgartner
mgartner committed Feb 16, 2021
1 parent 6596e1d commit 2ac4d8c
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 134 deletions.
12 changes: 12 additions & 0 deletions pkg/sql/opt/norm/general_funcs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -675,6 +675,11 @@ func (c *CustomFuncs) IsFilterFalse(filters memo.FiltersExpr) bool {
return filters.IsFalse()
}

// IsFilterEmpty returns true if filters is empty.
func (c *CustomFuncs) IsFilterEmpty(filters memo.FiltersExpr) bool {
return len(filters) == 0
}

// IsContradiction returns true if the given filter item contains a
// contradiction constraint.
func (c *CustomFuncs) IsContradiction(item *memo.FiltersItem) bool {
Expand All @@ -692,6 +697,13 @@ func (c *CustomFuncs) ConcatFilters(left, right memo.FiltersExpr) memo.FiltersEx
return newFilters
}

// DiffFilters creates new Filters that contains all conditions in left that do
// not exist in right. If right is empty, the original left filters are
// returned.
func (c *CustomFuncs) DiffFilters(left, right memo.FiltersExpr) memo.FiltersExpr {
return left.Difference(right)
}

// RemoveFiltersItem returns a new list that is a copy of the given list, except
// that it does not contain the given search item. If the list contains the item
// multiple times, then only the first instance is removed. If the list does not
Expand Down
143 changes: 14 additions & 129 deletions pkg/sql/opt/norm/inline_funcs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -223,148 +223,32 @@ func (c *CustomFuncs) VirtualColumns(scanPrivate *memo.ScanPrivate) opt.ColSet {
return tabMeta.VirtualComputedColumns()
}

// TryInlineSelectVirtualColumns is used by the InlineSelectVirtualColumns rule
// to find filters on virtual computed columns that can be pushed below the
// Project that produces the virtual column. The filter is pushed-down by
// inlining the virtual column expression. Only the columns in virtualColumns
// are eligible for inlining. It is useful to inline filters with virtual
// columns in order to generate expressions that utilize indexes on virtual
// columns during exploration. See the InlineSelectVirtualColumns rule for more
// details.
// InlinableVirtualColumnFilters returns a new filters expression containing any
// of the given filters that meet the criteria:
//
// If successful, a pair of filters is returned; the first containing filters in
// which virtual columns have been inlined, and the second containing filters
// that were unaffected by inlining. If unsuccessful, an empty
// InlinedVirtualColumnFiltersPair is returned, indicating that there is nothing
// to inline.
// 1. The filter has references to any of the columns in virtualColumns.
// 2. The filter is not a correlated subquery.
//
// Two separate filters are returned as a pair so that the
// InlineSelectVirtualColumns rule can push down eligible filters, but not the
// rest.
//
// For example:
//
// CREATE TABLE t (
// a INT,
// b INT,
// v INT AS (abs(a))
// )
//
// SELECT v, w FROM (
// SELECT v, abs(b) AS w FROM t
// ) WHERE v = 5 AND w = 10
//
// The (v = 5) and (w = 10) filters are returned in separate filters. In the
// first v is inlined because it is a virtual column, and the filter is
// transformed to (abs(a) = 5). This filter is no longer dependent on the
// projection of v, so it can be pushed below the Project. The w variable is not
// a virtual column, so the (w = 10) filter is unchanged and remains above the
// Project.
func (c *CustomFuncs) TryInlineSelectVirtualColumns(
filters memo.FiltersExpr, projections memo.ProjectionsExpr, virtualColumns opt.ColSet,
) InlinedVirtualColumnFiltersPair {
// Collect the projections for the virtual columns.
var inlinableCols opt.ColSet
var inlinableProjections memo.ProjectionsExpr
for i := range projections {
col := projections[i].Col

// If the projected expression is volatile, it cannot be inlined. This
// should be impossible (computed column expressions cannot be
// volatile), but adds additional safety in case this function is
// expanded to inline non-virtual columns in the future.
if projections[i].ScalarProps().VolatilitySet.HasVolatile() {
continue
}

if virtualColumns.Contains(col) {
// Initialize inlinableProjections lazily.
if inlinableProjections == nil {
inlinableProjections = make(memo.ProjectionsExpr, 0, len(projections)-i)
}
inlinableCols.Add(col)
inlinableProjections = append(inlinableProjections, projections[i])
}
}

if len(inlinableProjections) == 0 {
// None of the projections are eligible for inlining in filters.
return InlinedVirtualColumnFiltersPair{}
}

// For each filter, determine if there are any virtual columns to inline. If
// so, inline the virtual column expressions in the filter. We keep track of
// which filters have inlined projections in inlinedFilterIdxs so that we
// can build a list of non-inlined filters below.
var inlined memo.FiltersExpr
var inlinedFilterIdxs util.FastIntSet
func (c *CustomFuncs) InlinableVirtualColumnFilters(
filters memo.FiltersExpr, virtualColumns opt.ColSet,
) (inlinableFilters memo.FiltersExpr) {
for i := range filters {
item := &filters[i]

// Do not inline a filter if it has a correlated subquery or it does not
// reference a virtual column.
if item.ScalarProps().HasCorrelatedSubquery || !item.ScalarProps().OuterCols.Intersects(inlinableCols) {
if item.ScalarProps().HasCorrelatedSubquery || !item.ScalarProps().OuterCols.Intersects(virtualColumns) {
continue
}

// Initialize inlined lazily.
if inlined == nil {
inlined = make(memo.FiltersExpr, 0, len(filters)-i)
// Initialize inlinableFilters lazily.
if inlinableFilters == nil {
inlinableFilters = make(memo.FiltersExpr, 0, len(filters)-i)
}

// Inline the virtual column expressions in the filter.
filter := c.f.ConstructFiltersItem(
c.inlineProjections(item.Condition, inlinableProjections).(opt.ScalarExpr),
)
inlined = append(inlined, filter)
inlinedFilterIdxs.Add(i)
inlinableFilters = append(inlinableFilters, *item)
}

if len(inlined) == 0 {
// No projections were inlined in filters.
return InlinedVirtualColumnFiltersPair{}
}

// Collect the filters which did not have projections inlined.
notInlined := make(memo.FiltersExpr, 0, len(filters)-inlinedFilterIdxs.Len())
for i := range filters {
if !inlinedFilterIdxs.Contains(i) {
notInlined = append(notInlined, filters[i])
}
}

return InlinedVirtualColumnFiltersPair{
inlined: inlined,
notInlined: notInlined,
}
}

// InlinedVirtualColumnFiltersPair is the result of
// TryInlineSelectVirtualColumns. See that function for more details.
type InlinedVirtualColumnFiltersPair struct {
inlined memo.FiltersExpr
notInlined memo.FiltersExpr
}

// InlinedFilters returns the filters in the pair where virtual columns were
// inlined.
func (c *CustomFuncs) InlinedFilters(pair InlinedVirtualColumnFiltersPair) memo.FiltersExpr {
return pair.inlined
}

// NotInlinedFilters returns the filters in the pair where virtual columns were
// not inlined.
func (c *CustomFuncs) NotInlinedFilters(pair InlinedVirtualColumnFiltersPair) memo.FiltersExpr {
return pair.notInlined
}

// InlineSelectVirtualColumnsSucceeded returns true if the inlined filters in
// the pair are not empty, indicating that InlineSelectVirtualColumns
// successfully inlined virtual columns into some of the filters.
func (c *CustomFuncs) InlineSelectVirtualColumnsSucceeded(
pair InlinedVirtualColumnFiltersPair,
) bool {
return len(pair.inlined) > 0
return inlinableFilters
}

// InlineSelectProject searches the filter conditions for any variable
Expand Down Expand Up @@ -395,6 +279,7 @@ func (c *CustomFuncs) InlineProjectProject(
newProjections := make(memo.ProjectionsExpr, len(projections))
for i := range projections {
item := &projections[i]

newProjections[i] = c.f.ConstructProjectionsItem(
c.inlineProjections(item.Element, innerProjections).(opt.ScalarExpr),
item.Col,
Expand Down
12 changes: 7 additions & 5 deletions pkg/sql/opt/norm/rules/inline.opt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -246,22 +246,24 @@
^(ColsAreEmpty
$virtualColumns:(VirtualColumns $scanPrivate)
) &
(InlineSelectVirtualColumnsSucceeded
$result:(TryInlineSelectVirtualColumns
^(IsFilterEmpty
$inlinableFilters:(InlinableVirtualColumnFilters
$filters
$projections
$virtualColumns
)
)
)
=>
(Select
(Project
(Select $scan (InlinedFilters $result))
(Select
$scan
(InlineSelectProject $inlinableFilters $projections)
)
$projections
$passthrough
)
(NotInlinedFilters $result)
(DiffFilters $filters $inlinableFilters)
)

# InlineProjectInProject folds an inner Project operator into an outer Project
Expand Down

0 comments on commit 2ac4d8c

Please sign in to comment.