Skip to content

Commit

Permalink
Merge cockroachdb#52755
Browse files Browse the repository at this point in the history 
52755: opt: pull join commutation logic into join_order_builder.go r=DrewKimball a=DrewKimball

Previously, inner joins and full joins were commuted by the
CommuteJoin rule. This PR brings commutation into the addJoin
function of JoinOrderBuilder, so that joins can be commuted at
the same time as they are reordered.

Note that CommuteLeftJoin remains because commuting a left join
requires a change in the type of join operator.

Also note that setting ReorderJoinsLimit to 0 now disallows
commutation, while setting ReorderJoinsLimit to 1 allows
commutation, but no other reordering.

Release note: None

Co-authored-by: Drew Kimball <[email protected]>
  • Loading branch information
@DrewKimball
craig[bot] and DrewKimball committed Aug 14, 2020
2 parents b289a79 + f5b3d4f commit dd49b04
Show file tree
Hide file tree
Showing 14 changed files with 951 additions and 556 deletions.
24 changes: 12 additions & 12 deletions pkg/sql/opt/memo/testdata/logprops/join
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2465,18 +2465,28 @@ limit
├── interesting orderings: (+6) (+8) (+10) (+14)
├── inner-join (hash)
│ ├── columns: t_st_id:1(int!null) t_tt_id:2(int!null) t_s_symb:3(int!null) st_id:6(int!null) tt_id:8(int!null) s_symb:10(int!null) s_st_id:11(int!null) s_ex_id:12(int!null) ex_id:14(int!null)
│ ├── multiplicity: left-rows(zero-or-more), right-rows(zero-or-one)
│ ├── multiplicity: left-rows(zero-or-one), right-rows(zero-or-more)
│ ├── fd: (10)-->(11,12), (12)==(14), (14)==(12), (1)==(6), (6)==(1), (2)==(8), (8)==(2), (3)==(10), (10)==(3)
│ ├── limit hint: 50.00
│ ├── prune: (11)
│ ├── interesting orderings: (+6) (+8) (+10) (+14)
│ ├── scan trade
│ │ ├── columns: t_st_id:1(int!null) t_tt_id:2(int!null) t_s_symb:3(int!null)
│ │ ├── prune: (1-3)
│ │ └── unfiltered-cols: (1-5)
│ ├── inner-join (cross)
│ │ ├── columns: st_id:6(int!null) tt_id:8(int!null) s_symb:10(int!null) s_st_id:11(int!null) s_ex_id:12(int!null) ex_id:14(int!null)
│ │ ├── multiplicity: left-rows(one-or-more), right-rows(zero-or-more)
│ │ ├── multiplicity: left-rows(zero-or-more), right-rows(one-or-more)
│ │ ├── key: (6,8,10)
│ │ ├── fd: (10)-->(11,12), (12)==(14), (14)==(12)
│ │ ├── prune: (6,8,10,11)
│ │ ├── interesting orderings: (+6) (+8) (+10) (+14)
│ │ ├── scan status_type
│ │ │ ├── columns: st_id:6(int!null)
│ │ │ ├── key: (6)
│ │ │ ├── prune: (6)
│ │ │ ├── interesting orderings: (+6)
│ │ │ └── unfiltered-cols: (6,7)
│ │ ├── inner-join (cross)
│ │ │ ├── columns: tt_id:8(int!null) s_symb:10(int!null) s_st_id:11(int!null) s_ex_id:12(int!null) ex_id:14(int!null)
│ │ │ ├── key: (8,10)
Expand Down Expand Up @@ -2515,17 +2525,7 @@ limit
│ │ │ │ ├── variable: ex_id:14 [type=int]
│ │ │ │ └── variable: s_ex_id:12 [type=int]
│ │ │ └── filters (true)
│ │ ├── scan status_type
│ │ │ ├── columns: st_id:6(int!null)
│ │ │ ├── key: (6)
│ │ │ ├── prune: (6)
│ │ │ ├── interesting orderings: (+6)
│ │ │ └── unfiltered-cols: (6,7)
│ │ └── filters (true)
│ ├── scan trade
│ │ ├── columns: t_st_id:1(int!null) t_tt_id:2(int!null) t_s_symb:3(int!null)
│ │ ├── prune: (1-3)
│ │ └── unfiltered-cols: (1-5)
│ └── filters
│ ├── eq [type=bool, outer=(1,6), constraints=(/1: (/NULL - ]; /6: (/NULL - ]), fd=(1)==(6), (6)==(1)]
│ │ ├── variable: st_id:6 [type=int]
Expand Down
262 changes: 128 additions & 134 deletions pkg/sql/opt/memo/testdata/stats_quality/tpch/q20
View file

Large diffs are not rendered by default.

14 changes: 1 addition & 13 deletions pkg/sql/opt/norm/testdata/rules/combo
View file
Original file line number Diff line number Diff line change
Expand Up @@ -308,11 +308,8 @@ GenerateConstrainedScans (no changes)
--------------------------------------------------------------------------------
GenerateIndexScans (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
ReorderJoins (no changes)
--------------------------------------------------------------------------------
================================================================================
CommuteJoin
ReorderJoins
Cost: 2132.97
================================================================================
project
Expand Down Expand Up @@ -411,9 +408,6 @@ GenerateLookupJoins
- │ └── ordering: +7
└── filters (true)
--------------------------------------------------------------------------------
CommuteJoin (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
GenerateMergeJoins (higher cost)
--------------------------------------------------------------------------------
project
Expand Down Expand Up @@ -1036,15 +1030,9 @@ GenerateStreamingGroupBy (no changes)
ReorderJoins (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
CommuteJoin (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
GenerateMergeJoins (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
CommuteJoin (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
GenerateMergeJoins (no changes)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Expand Down
75 changes: 73 additions & 2 deletions pkg/sql/opt/testutils/opttester/testdata/explore-trace
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,37 @@ SELECT * FROM
ORDER BY pid
----
----
================================================================================
ReorderJoins
================================================================================
Source expression:
sort
└── project
└── inner-join (hash)
├── inner-join (hash)
│ ├── scan grandchild
│ ├── scan child
│ └── filters
│ ├── grandchild.pid = child.pid
│ └── grandchild.cid = child.cid
├── scan parent
└── filters
└── grandchild.pid = parent.pid

New expression 1 of 1:
sort
└── project
└── inner-join (hash)
├── inner-join (hash)
│ ├── scan child
│ ├── scan grandchild
│ └── filters
│ ├── grandchild.pid = child.pid
│ └── grandchild.cid = child.cid
├── scan parent
└── filters
└── grandchild.pid = parent.pid

================================================================================
ReorderJoins
================================================================================
Expand All @@ -148,7 +179,7 @@ Source expression:
└── filters
└── grandchild.pid = parent.pid

New expression 1 of 2:
New expression 1 of 5:
sort
└── project
└── inner-join (hash)
Expand All @@ -162,7 +193,21 @@ New expression 1 of 2:
├── grandchild.pid = child.pid
└── grandchild.cid = child.cid

New expression 2 of 2:
New expression 2 of 5:
sort
└── project
└── inner-join (hash)
├── inner-join (hash)
│ ├── scan child
│ ├── scan parent
│ └── filters
│ └── child.pid = parent.pid
├── scan grandchild
└── filters
├── grandchild.pid = child.pid
└── grandchild.cid = child.cid

New expression 3 of 5:
sort
└── project
└── inner-join (hash)
Expand All @@ -175,5 +220,31 @@ New expression 2 of 2:
└── filters
├── grandchild.pid = child.pid
└── grandchild.cid = child.cid

New expression 4 of 5:
sort
└── project
└── inner-join (hash)
├── inner-join (hash)
│ ├── scan grandchild
│ ├── scan parent
│ └── filters
│ └── grandchild.pid = parent.pid
├── scan child
└── filters
├── grandchild.pid = child.pid
└── grandchild.cid = child.cid

New expression 5 of 5:
sort
└── project
└── inner-join (hash)
├── scan parent
├── inner-join (merge)
│ ├── scan grandchild
│ ├── scan child
│ └── filters (true)
└── filters
└── grandchild.pid = parent.pid
----
----
38 changes: 9 additions & 29 deletions pkg/sql/opt/xform/custom_funcs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1532,33 +1532,6 @@ func (c *CustomFuncs) getKnownScanConstraint(
//
// ----------------------------------------------------------------------

// CommuteJoinFlags returns a join private for the commuted join (where the left
// and right sides are swapped). It adjusts any join flags that are specific to
// one side.
func (c *CustomFuncs) CommuteJoinFlags(p *memo.JoinPrivate) *memo.JoinPrivate {
if p.Flags.Empty() {
return p
}

// swap is a helper function which swaps the values of two (single-bit) flags.
swap := func(f, a, b memo.JoinFlags) memo.JoinFlags {
// If the bits are different, flip them both.
if f.Has(a) != f.Has(b) {
f ^= (a | b)
}
return f
}
f := p.Flags
f = swap(f, memo.AllowLookupJoinIntoLeft, memo.AllowLookupJoinIntoRight)
f = swap(f, memo.AllowHashJoinStoreLeft, memo.AllowHashJoinStoreRight)
if p.Flags == f {
return p
}
res := *p
res.Flags = f
return &res
}

// GenerateMergeJoins spawns MergeJoinOps, based on any interesting orderings.
func (c *CustomFuncs) GenerateMergeJoins(
grp memo.RelExpr,
Expand Down Expand Up @@ -1588,8 +1561,15 @@ func (c *CustomFuncs) GenerateMergeJoins(
orders := DeriveInterestingOrderings(left).Copy()
orders.RestrictToCols(leftEq.ToSet())

if !joinPrivate.Flags.Has(memo.AllowHashJoinStoreLeft) &&
!joinPrivate.Flags.Has(memo.AllowHashJoinStoreRight) {
if (!joinPrivate.Flags.Has(memo.AllowHashJoinStoreLeft) &&
!joinPrivate.Flags.Has(memo.AllowHashJoinStoreRight)) ||
c.e.evalCtx.SessionData.ReorderJoinsLimit == 0 {
// If we are using a hint, or the join limit is set to zero, the join won't
// be commuted. Add the orderings from the right side.
rightOrders := DeriveInterestingOrderings(right).Copy()
rightOrders.RestrictToCols(leftEq.ToSet())
orders = append(orders, rightOrders...)

// If we don't allow hash join, we must do our best to generate a merge
// join, even if it means sorting both sides. We append an arbitrary
// ordering, in case the interesting orderings don't result in any merge
Expand Down
27 changes: 20 additions & 7 deletions pkg/sql/opt/xform/join_order_builder.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -309,8 +309,7 @@ func (jb *JoinOrderBuilder) Init(f *norm.Factory, evalCtx *tree.EvalContext) {
jb.joinCount = 0
}

// Reorder adds all valid (non-commutative) orderings of the given join to the
// memo.
// Reorder adds all valid orderings of the given join to the memo.
func (jb *JoinOrderBuilder) Reorder(join memo.RelExpr) {
switch t := join.(type) {
case *memo.InnerJoinExpr, *memo.SemiJoinExpr, *memo.AntiJoinExpr,
Expand Down Expand Up @@ -519,11 +518,9 @@ func (jb *JoinOrderBuilder) addJoins(s1, s2 vertexSet) {
}
if e.checkNonInnerJoin(s2, s1) {
// If joining s1, s2 is not valid, try s2, s1. We only do this if the
// s1, s2 join fails, because commutation (for full joins) is handled by
// the CommuteJoin rule.
//
// This is necessary because we only iterate s1 up to subset / 2 in
// DPSube(). Take this transformation as an example:
// s1, s2 join fails, because commutation is handled by the addJoin
// function. This is necessary because we only iterate s1 up to subset / 2
// in DPSube(). Take this transformation as an example:
//
// SELECT *
// FROM (SELECT * FROM xy LEFT JOIN ab ON x = a)
Expand Down Expand Up @@ -662,6 +659,7 @@ func (jb *JoinOrderBuilder) addJoin(
// Hook for testing purposes.
jb.callOnAddJoinFunc(s1, s2, joinFilters, op)
}

left := jb.plans[s1]
right := jb.plans[s2]
union := s1.union(s2)
Expand All @@ -670,6 +668,16 @@ func (jb *JoinOrderBuilder) addJoin(
} else {
jb.plans[union] = jb.memoize(op, left, right, joinFilters, selectFilters)
}

if commute(op) {
// Also add the commuted version of the join to the memo.
jb.addToGroup(op, right, left, joinFilters, selectFilters, jb.plans[union])

if jb.onAddJoinFunc != nil {
// Hook for testing purposes.
jb.callOnAddJoinFunc(s2, s1, joinFilters, op)
}
}
}

// areFiltersRedundant returns true if the given FiltersExpr contains a single
Expand Down Expand Up @@ -1262,6 +1270,11 @@ func (e *edge) checkRules(s1, s2 vertexSet) bool {
return true
}

// commute returns true if the given join operator type is commutable.
func commute(op opt.Operator) bool {
return op == opt.InnerJoinOp || op == opt.FullJoinOp
}

// assoc returns true if two joins with the operator types and filters described
// by the given edges are associative with each other. An example of an
// application of the associative property:
Expand Down
18 changes: 4 additions & 14 deletions pkg/sql/opt/xform/rules/join.opt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@
# join.opt contains exploration rules for the Join operator.
# =============================================================================

# ReorderJoins matches the first expression of an inner join group and adds to
# the memo all valid join orderings that do not introduce cross joins. If the
# join has join hints or is the result of a previous join reordering, the join
# tree is not reordered. For more information, see the comment in
# join_order_builder.go.
# ReorderJoins matches the first expression of a join group and adds to the memo
# all valid join orderings that do not introduce cross joins. If the join has
# join hints or is the result of a previous join reordering, the join tree is
# not reordered. For more information, see the comment in join_order_builder.go.
#
# Citations: [8]
[ReorderJoins, Explore]
Expand All @@ -16,15 +15,6 @@
=>
(ReorderJoins)

# CommuteJoin creates a Join with the left and right inputs swapped. This is
# useful for other rules that convert joins to other operators (like merge
# join).
# If any join hints are specified, we keep the order in the query.
[CommuteJoin, Explore]
(InnerJoin | FullJoin $left:* $right:* $on:* $private:*)
=>
((OpName) $right $left $on (CommuteJoinFlags $private))

# CommuteLeftJoin creates a Join with the left and right inputs swapped.
# This is symmetric with the CommuteRightJoin normalization rule.
[CommuteLeftJoin, Explore]
Expand Down
Loading

0 comments on commit dd49b04

Please sign in to comment.