sql: a crash with a lookup join #40562

yuzefovich · 2019-09-06T21:26:04Z

I slightly modified query 21 of TPCH benchmark, and it crashed the server with:

panic while executing 1 statements: SELECT _, count(*) AS _ FROM _, _ AS _, _, _ WHERE (((((((_ = _._) AND (_ = _._)) AND (_ = _)) AND (_._ > _._)) AND EXISTS (SELECT * FROM _ AS _ WHERE (_._ = _._) AND (_._ != _._))) AND (NOT EXISTS (SELECT * FROM _ AS _ WHERE (_._ = _._) AND (_._ > _._)))) AND (_ = _)) AND (_ = _) GROUP BY _ ORDER BY _ DESC, _ LIMIT _

goroutine 6034 [running]:
runtime/debug.Stack(0x8786980, 0xc00789a300, 0x3)
	/usr/local/opt/go/libexec/src/runtime/debug/stack.go:24 +0x9d
github.com/cockroachdb/cockroach/pkg/util/log.ReportPanic(0x8786980, 0xc00789a300, 0xc000278000, 0x7bef120, 0xc003e23170, 0x1)
	/Users/yahoryuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:217 +0xb6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc007d0c000, 0x8786980, 0xc00789a300, 0x78a6ec0, 0xb1cdc40)
	/Users/yahoryuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:723 +0x2a0
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc007d0c000, 0x8786980, 0xc00789a300)
	/Users/yahoryuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:442 +0x61
panic(0x78a6ec0, 0xb1cdc40)
	/usr/local/opt/go/libexec/src/runtime/panic.go:522 +0x1b5
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc007f10120, 0x8788500, 0xc007ce8f70, 0x7dad612, 0xb)
	/Users/yahoryuzefovich/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:242 +0x64d6
...

Here is the modified query:

EXPLAIN SELECT s_name, count(*) AS numwait
    FROM supplier, lineitem AS l1, orders, nation
   WHERE s_suppkey = l1.l_suppkey
     AND o_orderkey = l1.l_orderkey
     AND o_orderstatus = 'F'
     AND l1.l_receiptdate > l1.l_commitdate
     AND EXISTS(
            SELECT *
              FROM lineitem AS l2
             WHERE l2.l_orderkey = l1.l_orderkey
               AND l2.l_suppkey != l1.l_suppkey
         )
     AND NOT EXISTS(
                SELECT *
                  FROM lineitem AS l3
                 WHERE l3.l_orderkey = l1.l_orderkey
                   AND l3.l_receiptdate > l3.l_commitdate
             )
     AND s_nationkey = n_nationkey
     AND n_name = 'SAUDI ARABIA'
GROUP BY s_name
ORDER BY numwait DESC, s_name
   LIMIT 100;

(I simply removed AND l3.l_suppkey != l1.l_suppkey condition from within NOT EXISTS subquery.)

The text was updated successfully, but these errors were encountered:

jordanlewis · 2019-09-06T21:36:55Z

More accurately it's a crash in explain, right?

E190906 21:35:45.824605 673 util/log/crash_reporting.go:207  [n1,client=[::1]:65421,user=root] a panic has occurred!
panic: runtime error: index out of range
	panic: runtime error: index out of range [recovered]
	panic: panic while executing 1 statements: EXPLAIN SELECT _, count(*) AS _ FROM _, _ AS _, _, _ WHERE (((((((_ = _._) AND (_ = _._)) AND (_ = _)) AND (_._ > _._)) AND EXISTS (SELECT * FROM _ AS _ WHERE (_._ = _._) AND (_._ != _._))) AND (NOT EXISTS (SELECT * FROM _ AS _ WHERE (_._ = _._) AND (_._ > _._)))) AND (_ = _)) AND (_ = _) GROUP BY _ ORDER BY _ DESC, _ LIMIT _; caused by runtime error: index out of range

goroutine 673 [running]:
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc003519000, 0x85acb60, 0xc0034cc840, 0x76f24a0, 0xaf5fc20)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:730 +0x330
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc003519000, 0x85acb60, 0xc0034cc840)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:442 +0x61
panic(0x76f24a0, 0xaf5fc20)
	/usr/local/Cellar/go/1.12.4/libexec/src/runtime/panic.go:522 +0x1b5
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85ae6e0, 0xc003f900d0, 0x7c0bd89, 0xb)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:242 +0x64d6
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85ae6e0, 0xc003f900d0, 0x7bf96a9, 0x4)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85ae6e0, 0xc003f901a0, 0x7c0bd89, 0xb)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:269 +0x56d1
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85ae6e0, 0xc003f901a0, 0x7bfeabd, 0x6)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85ae9e0, 0xc003f82900, 0x7bfeabd, 0x6)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:221 +0x4c17
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85ae9e0, 0xc003f82900, 0x8, 0xc003ebc030)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85ae560, 0xc003eeec00, 0x7bfb71a, 0x5)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:429 +0x131a
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85ae560, 0xc003eeec00, 0x5, 0xc003f03800)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85aede0, 0xc003f5a8c0, 0x7bf9829, 0x4)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:390 +0x23d7
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85aede0, 0xc003f5a8c0, 0x7bfe87d, 0x6)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85ae6a0, 0xc003f5a910, 0x7bfb913, 0x5)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:342 +0x4201
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85ae6a0, 0xc003f5a910, 0x7c0df07, 0xc)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc003f77a40, 0x85ae460, 0xc003f47680, 0x7c0df07, 0xc)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:619 +0x1778
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc003f77a40, 0x85ae460, 0xc003f47680, 0xc003c56678, 0x400b3ac)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:72
github.com/cockroachdb/cockroach/pkg/sql.populateEntriesForObserver(0x85acc20, 0xc003061cb0, 0x85ae460, 0xc003f47680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/explain_plan.go:281 +0x2a1
github.com/cockroachdb/cockroach/pkg/sql.planToTree(0x85acc20, 0xc003061cb0, 0xc003519678, 0x0)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/explain_tree.go:112 +0x27e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).maybeSavePlan(0xc003519000, 0x85acc20, 0xc003061cb0, 0xc003519358, 0xc003c56bc8)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor_statement_metrics.go:109 +0xa8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine.func1(0x85acc20, 0xc003061cb0, 0x0)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:634 +0x47
github.com/cockroachdb/cockroach/pkg/sql.(*planTop).close(0xc003519678, 0x85acc20, 0xc003061cb0)
	/Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:297 +0x246
panic(0x76f24a0, 0xaf5fc20)

It fails on the last line here:

	case *lookupJoinNode:
		if v.observer.attr != nil {
			v.observer.attr(name, "table", fmt.Sprintf("%s@%s", n.table.desc.Name, n.table.index.Name))
			v.observer.attr(name, "type", joinTypeStr(n.joinType))
		}
		var b bytes.Buffer
		b.WriteByte('(')
		inputCols := planColumns(n.input)
		for i, c := range n.eqCols {
			if i > 0 {
				b.WriteString(", ")
			}
			b.WriteString(inputCols[c].Name)
		}

Which says to me that a column in n.eqCols isn't present in inputCols. Not sure what that would mean though.

yuzefovich · 2019-09-06T21:39:33Z

It will also crash without explain, on distsql physical planning step.

yuzefovich · 2019-09-06T21:40:54Z

Debugger says that n.eqCols = {8}, but len(inputCols) == 8, so somehow the indices of the columns are screwed up.

jordanlewis · 2019-09-06T21:41:29Z

It sounds like this is an optimizer or execbuilder bug then, if we're getting malformed planNodes.

yuzefovich · 2019-09-06T21:44:59Z

Yeah, maybe. I wasn't sure about that, so I assigned it to Solon for triage.

jordanlewis · 2019-09-07T15:54:32Z

I think that this is a release blocker, so I'm adding it to #40447.

jordanlewis · 2019-09-10T13:01:28Z

Explain(opt):

                                          text
+--------------------------------------------------------------------------------------+
  limit
   ├── sort
   │    └── group-by
   │         ├── inner-join (lookup orders)
   │         │    ├── inner-join (lookup nation)
   │         │    │    ├── inner-join (lookup supplier)
   │         │    │    │    ├── semi-join (lookup lineitem)
   │         │    │    │    │    ├── anti-join (merge)
   │         │    │    │    │    │    ├── select
   │         │    │    │    │    │    │    ├── scan l1
   │         │    │    │    │    │    │    └── filters
   │         │    │    │    │    │    │         └── l1.l_receiptdate > l1.l_commitdate
   │         │    │    │    │    │    ├── select
   │         │    │    │    │    │    │    ├── scan l3
   │         │    │    │    │    │    │    └── filters
   │         │    │    │    │    │    │         └── l3.l_receiptdate > l3.l_commitdate
   │         │    │    │    │    │    └── filters (true)
   │         │    │    │    │    └── filters
   │         │    │    │    │         └── l2.l_suppkey != l1.l_suppkey
   │         │    │    │    └── filters (true)
   │         │    │    └── filters
   │         │    │         └── n_name = 'SAUDI ARABIA'
   │         │    └── filters
   │         │         └── o_orderstatus = 'F'
   │         └── aggregations
   │              └── count-rows
   └── const: 100
(27 rows)

Time: 20.006ms

Verbose:

                                                                                                      text
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  limit
   ├── columns: s_name:2 numwait:69
   ├── internal-ordering: -69,+2
   ├── cardinality: [0 - 100]
   ├── stats: [rows=1.68010753e-15]
   ├── cost: 14535822.2
   ├── key: (2)
   ├── fd: (2)-->(69)
   ├── ordering: -69,+2
   ├── interesting orderings: (-69,+2)
   ├── sort
   │    ├── columns: s_name:2 count_rows:69
   │    ├── stats: [rows=1.68010753e-15, distinct(2)=1.68010753e-15, null(2)=0]
   │    ├── cost: 14535822.2
   │    ├── key: (2)
   │    ├── fd: (2)-->(69)
   │    ├── ordering: -69,+2
   │    ├── prune: (69)
   │    └── group-by
   │         ├── columns: s_name:2 count_rows:69
   │         ├── grouping columns: s_name:2
   │         ├── stats: [rows=1.68010753e-15, distinct(2)=1.68010753e-15, null(2)=0]
   │         ├── cost: 14535822.2
   │         ├── key: (2)
   │         ├── fd: (2)-->(69)
   │         ├── prune: (69)
   │         ├── inner-join (lookup orders)
   │         │    ├── columns: s_suppkey:1 s_name:2 s_nationkey:4 l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20 o_orderkey:24 o_orderstatus:26 n_nationkey:33 n_name:34
   │         │    ├── key columns: [8] = [24]
   │         │    ├── stats: [rows=1.68010753e-15, distinct(1)=3.36021505e-25, null(1)=0, distinct(2)=1.68010753e-15, null(2)=0, distinct(4)=1.68010753e-19, null(4)=0, distinct(8)=3.36021505e-25, null(8)=0, distinct(10)=3.36021505e-25, null(10)=0, distinct(19)=3.36021505e-25, null(19)=0, distinct(20)=3.36021505e-25, null(20)=0, distinct(24)=3.36021505e-25, null(24)=0, distinct(26)=1.68010753e-19, null(26)=0, distinct(33)=1.68010753e-19, null(33)=0, distinct(34)=1.68010753e-19, null(34)=0]
   │         │    ├── cost: 14535822.2
   │         │    ├── fd: ()-->(26,34), (1)-->(2,4), (8)==(24), (24)==(8), (1)==(10), (10)==(1), (4)==(33), (33)==(4)
   │         │    ├── prune: (2)
   │         │    ├── interesting orderings: (+1) (+4,+1) (+8) (+10,+8) (+19,+8) (+20,+8) (+24) (+33)
   │         │    ├── inner-join (lookup nation)
   │         │    │    ├── columns: s_suppkey:1 s_name:2 s_nationkey:4 l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20 n_nationkey:33 n_name:34
   │         │    │    ├── key columns: [4] = [33]
   │         │    │    ├── stats: [rows=3.36021505e-21, distinct(1)=3.36021505e-25, null(1)=0, distinct(2)=3.36021505e-21, null(2)=0, distinct(4)=3.36021505e-25, null(4)=0, distinct(8)=3.36021505e-25, null(8)=0, distinct(10)=3.36021505e-25, null(10)=0, distinct(19)=3.36021505e-25, null(19)=0, distinct(20)=3.36021505e-25, null(20)=0, distinct(33)=3.36021505e-25, null(33)=0, distinct(34)=3.36021505e-25, null(34)=0]
   │         │    │    ├── cost: 14535822.2
   │         │    │    ├── fd: ()-->(34), (1)-->(2,4), (1)==(10), (10)==(1), (4)==(33), (33)==(4)
   │         │    │    ├── interesting orderings: (+33) (+8) (+10,+8) (+19,+8) (+20,+8) (+1) (+4,+1)
   │         │    │    ├── join-size: 3
   │         │    │    ├── inner-join (lookup supplier)
   │         │    │    │    ├── columns: s_suppkey:1 s_name:2 s_nationkey:4 l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20
   │         │    │    │    ├── key columns: [10] = [1]
   │         │    │    │    ├── stats: [rows=3.36021505e-21, distinct(1)=3.36021505e-25, null(1)=0, distinct(2)=3.36021505e-21, null(2)=0, distinct(4)=3.36021505e-21, null(4)=0, distinct(8)=3.36021505e-25, null(8)=0, distinct(10)=3.36021505e-25, null(10)=0, distinct(19)=3.36021505e-25, null(19)=0, distinct(20)=3.36021505e-25, null(20)=0]
   │         │    │    │    ├── cost: 14535822.2
   │         │    │    │    ├── fd: (1)-->(2,4), (1)==(10), (10)==(1)
   │         │    │    │    ├── interesting orderings: (+8) (+10,+8) (+19,+8) (+20,+8) (+1) (+4,+1)
   │         │    │    │    ├── join-size: 2
   │         │    │    │    ├── semi-join (lookup lineitem)
   │         │    │    │    │    ├── columns: l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20
   │         │    │    │    │    ├── key columns: [8] = [37]
   │         │    │    │    │    ├── stats: [rows=3.36021505e-25, distinct(8)=3.36021505e-25, null(8)=0, distinct(10)=3.36021505e-25, null(10)=0, distinct(19)=3.36021505e-25, null(19)=0, distinct(20)=3.36021505e-25, null(20)=0]
   │         │    │    │    │    ├── cost: 14535822.1
   │         │    │    │    │    ├── interesting orderings: (+8) (+10,+8) (+19,+8) (+20,+8)
   │         │    │    │    │    ├── anti-join (merge)
   │         │    │    │    │    │    ├── columns: l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20
   │         │    │    │    │    │    ├── left ordering: +8
   │         │    │    │    │    │    ├── right ordering: +53
   │         │    │    │    │    │    ├── stats: [rows=1e-10, distinct(8)=1e-10, null(8)=0, distinct(10)=1e-10, null(10)=0, distinct(19)=1e-10, null(19)=0, distinct(20)=1e-10, null(20)=0]
   │         │    │    │    │    │    ├── cost: 14535822.1
   │         │    │    │    │    │    ├── prune: (10)
   │         │    │    │    │    │    ├── interesting orderings: (+8) (+10,+8) (+19,+8) (+20,+8)
   │         │    │    │    │    │    ├── select
   │         │    │    │    │    │    │    ├── columns: l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20
   │         │    │    │    │    │    │    ├── stats: [rows=2000405, distinct(8)=1216823.04, null(8)=0, distinct(10)=9920, null(10)=0, distinct(19)=2466, null(19)=0, distinct(20)=2554, null(20)=0]
   │         │    │    │    │    │    │    ├── cost: 7261470.18
   │         │    │    │    │    │    │    ├── ordering: +8
   │         │    │    │    │    │    │    ├── prune: (8,10)
   │         │    │    │    │    │    │    ├── interesting orderings: (+8) (+10,+8) (+19,+8) (+20,+8)
   │         │    │    │    │    │    │    ├── scan l1
   │         │    │    │    │    │    │    │    ├── columns: l1.l_orderkey:8 l1.l_suppkey:10 l1.l_commitdate:19 l1.l_receiptdate:20
   │         │    │    │    │    │    │    │    ├── stats: [rows=6001215, distinct(8)=1527270, null(8)=0, distinct(10)=9920, null(10)=0, distinct(19)=2466, null(19)=0, distinct(20)=2554, null(20)=0]
   │         │    │    │    │    │    │    │    │   histogram(8)=  0  600  29405   600   29405   600   29405   600   29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600
   │         │    │    │    │    │    │    │    │                <--- 259 ------- 26467 ------- 59207 ------- 92198 ------- 119297 ------- 152835 ------- 192930 ------- 216613 ------- 250049 ------- 283009 ------- 316674 ------- 343492 ------- 370081 ------- 398978 ------- 427136 ------- 451940 ------- 484640 ------- 516327 ------- 543776 ------- 574273 ------- 607682 ------- 635431 ------- 670788 ------- 699265 ------- 719654 ------- 742917 ------- 771012 ------- 796770 ------- 826112 ------- 850951 ------- 877543 ------- 909604 ------- 942918 ------- 970532 ------- 1000391 ------- 1030214 ------- 1059907 ------- 1092641 ------- 1120068 ------- 1154695 ------- 1185507 ------- 1209666 ------- 1243234 ------- 1275557 ------- 1307142 ------- 1331973 ------- 1368615 ------- 1396323 ------- 1423010 ------- 1456164 ------- 1486917 ------- 1516550 ------- 1552549 ------- 1579585 ------- 1612455 ------- 1642531 ------- 1676580 ------- 1705603 ------- 1737379 ------- 1763591 ------- 1790245 ------- 1819335 ------- 1844710 ------- 1864612 ------- 1889926 ------- 1928295 ------- 1965473 ------- 1998983 ------- 2027527 ------- 2048551 ------- 2078854 ------- 2106083 ------- 2135556 ------- 2164289 ------- 2206340 ------- 2233472 ------- 2260064 ------- 2300770 ------- 2330023 ------- 2357092 ------- 2388230 ------- 2426433 ------- 2463365 ------- 2496390 ------- 2520805 ------- 2544741 ------- 2574656 ------- 2601922 ------- 2630337 ------- 2661090 ------- 2688837 ------- 2717091 ------- 2753248 ------- 2784514 ------- 2815654 ------- 2843492 ------- 2865159 ------- 2890210 ------- 2918375 ------- 2954336 ------- 2980678 ------- 3005669 ------- 3038340 ------- 3070563 ------- 3100709 ------- 3132295 ------- 3162914 ------- 3196743 ------- 3234852 ------- 3260454 ------- 3295427 ------- 3323842 ------- 3345668 ------- 3372196 ------- 3402884 ------- 3440258 ------- 3464801 ------- 3492229 ------- 3520742 ------- 3554529 ------- 3587654 ------- 3613602 ------- 3645282 ------- 3674690 ------- 3709539 ------- 3737734 ------- 3769639 ------- 3803009 ------- 3828197 ------- 3859718 ------- 3887847 ------- 3921028 ------- 3953764 ------- 3991430 ------- 4024257 ------- 4051394 ------- 4089348 ------- 4119110 ------- 4146818 ------- 4179524 ------- 4202784 ------- 4233312 ------- 4264738 ------- 4294821 ------- 4315171 ------- 4335330 ------- 4365093 ------- 4393541 ------- 4418692 ------- 4444096 ------- 4476134 ------- 4514599 ------- 4541538 ------- 4572485 ------- 4601888 ------- 4631015 ------- 4667013 ------- 4692935 ------- 4731204 ------- 4757383 ------- 4786823 ------- 4816707 ------- 4844615 ------- 4875205 ------- 4906914 ------- 4935015 ------- 4965024 ------- 4989671 ------- 5022531 ------- 5056038 ------- 5086177 ------- 5123200 ------- 5151108 ------- 5184928 ------- 5214022 ------- 5244417 ------- 5272033 ------- 5303910 ------- 5347521 ------- 5375653 ------- 5404514 ------- 5429543 ------- 5460707 ------- 5496513 ------- 5527555 ------- 5555686 ------- 5592868 ------- 5635456 ------- 5661925 ------- 5692225 ------- 5722148 ------- 5751811 ------- 5790023 ------- 5824256 ------- 5854533 ------- 5885762 ------- 5915648 ------- 5947713 ------- 5972262 ------- 5999617
   │         │    │    │    │    │    │    │    │   histogram(10)=  0 1800 29405 1200 29405  600  29405 1200  29405  600  27605 2400  29405  600  29405  600  29405  600  29405 1200  29405 1800  29405 1800  28805 1800  28805 1800  27005 3000  29405  600  29405  600  29405  600  28805 1200  29405 1200  28805 1200  29405  1200  28805  1200  29405  600   29405  600   29405  1200  29405  600   28805  2400  29405  1200  29405  1200  29405  1800  29405  1200  29405  600   29405  600   29405  600   29405  2400  29405  600   27605  2400  29405  1200  29405  600   29405  600   28205  1800  29405  600   29405  1200  29405  1200  28805  1200  28805  1200  29405  1800  29405  600   29405  1800  29405  1800  29405  1200  29405  600   28205  1800  28805  1200  29405  600   29405  600   29405  600   29405  600   29405  600   29405  1200  28805  1800  29405  600   27605  2400  28805  1800  29405  600   29405  600   29405  600   28205  1800  29405  1200  29405  600   27605  2400  28805  1200  29405  600   29405  600   29405  600   29405  600   28805  1800  29405  1200  28805  1200  29405  600   29405  600   28805  1800  28805  1800  27605  2400  28805  1800  28205  1800  29405  1800  29405  600   29405  1800  29405  1200  28805  1200  28805  1200  29405  600   28805  1800  29405  1200  28805  600   29405  600   29405  600   29405  1200  28805  600   28805  1200  29405  1200  27605  1800  29405  600   29405  1200  28805  600   27005  3000  28805  1800  27005  3000  28805  1800  28805  1200  28205  1800  28805  1200  28805  600   28805  600   28805  1200  28805  1800  27605  1800  27605  2400  29405  600   29405  1200  28205  3600  28205  1200  27605  2400  28205  3000  28205  1800  28805  1200  28805  600   28205  1800  28205  1200  27005  2400  28205  1200  28805  1200  28805  1200  28805  600   29405  600   29405  600   29405  1200  28805  600   28205  1800  28205  1800  29405  600   28805  1800  28805  600   29405  600   29405  600   29405  1800  28805  1800  28805  1200  28805  600   28805  600   28805  600   28805  1200  29405  1200  28205  1200  29405  600   29405  1800  28205  1200  28805  600   27605  2400  29405  600   29405  600   29405  600   29405  600   29405  600   29405  600   28205  1800  29405  600   29405  1200  28805  600   28805  1200  29405  600   29405  2400  28205  2400  28805  600   28205  1200  27605  2400  28805  600   28205  1200  29405  600   29405  600   29405  600   29405  600   28805  2400  28205  1200  28805  600   28805  1800  28805  1200  28805  1800  28205  1200  28205  1200  29405  600   29405  600   29405  1200  28205  1200  29405  600   28805  1200  29405  600   28805  1200
   │         │    │    │    │    │    │    │    │                 <--- 1 -------- 53 ------- 117 ------- 154 ------- 205 ------- 258 ------- 299 ------- 343 ------- 384 ------- 439 ------- 491 ------- 546 ------- 589 ------- 652 ------- 689 ------- 740 ------- 792 ------- 841 ------- 887 ------- 932 ------- 979 ------- 1026 ------- 1078 ------- 1130 ------- 1187 ------- 1237 ------- 1291 ------- 1356 ------- 1402 ------- 1455 ------- 1506 ------- 1560 ------- 1601 ------- 1668 ------- 1712 ------- 1770 ------- 1818 ------- 1855 ------- 1901 ------- 1952 ------- 2001 ------- 2046 ------- 2093 ------- 2132 ------- 2183 ------- 2232 ------- 2286 ------- 2339 ------- 2391 ------- 2446 ------- 2494 ------- 2546 ------- 2596 ------- 2654 ------- 2695 ------- 2737 ------- 2802 ------- 2862 ------- 2915 ------- 2962 ------- 3002 ------- 3055 ------- 3119 ------- 3178 ------- 3228 ------- 3285 ------- 3339 ------- 3386 ------- 3434 ------- 3491 ------- 3556 ------- 3596 ------- 3629 ------- 3689 ------- 3766 ------- 3808 ------- 3858 ------- 3897 ------- 3945 ------- 3984 ------- 4030 ------- 4084 ------- 4139 ------- 4189 ------- 4239 ------- 4282 ------- 4341 ------- 4398 ------- 4445 ------- 4490 ------- 4548 ------- 4602 ------- 4649 ------- 4700 ------- 4751 ------- 4810 ------- 4860 ------- 4915 ------- 4954 ------- 5004 ------- 5059 ------- 5107 ------- 5145 ------- 5185 ------- 5237 ------- 5284 ------- 5336 ------- 5388 ------- 5438 ------- 5481 ------- 5527 ------- 5579 ------- 5628 ------- 5673 ------- 5719 ------- 5779 ------- 5823 ------- 5870 ------- 5919 ------- 5974 ------- 6021 ------- 6063 ------- 6121 ------- 6167 ------- 6214 ------- 6260 ------- 6310 ------- 6367 ------- 6415 ------- 6479 ------- 6517 ------- 6564 ------- 6611 ------- 6658 ------- 6712 ------- 6762 ------- 6813 ------- 6865 ------- 6922 ------- 6973 ------- 7026 ------- 7071 ------- 7118 ------- 7166 ------- 7214 ------- 7274 ------- 7331 ------- 7387 ------- 7423 ------- 7464 ------- 7514 ------- 7557 ------- 7605 ------- 7665 ------- 7712 ------- 7761 ------- 7825 ------- 7876 ------- 7922 ------- 7985 ------- 8025 ------- 8078 ------- 8139 ------- 8197 ------- 8249 ------- 8297 ------- 8341 ------- 8389 ------- 8450 ------- 8490 ------- 8540 ------- 8598 ------- 8661 ------- 8712 ------- 8762 ------- 8818 ------- 8871 ------- 8918 ------- 8958 ------- 8999 ------- 9044 ------- 9123 ------- 9178 ------- 9228 ------- 9267 ------- 9309 ------- 9367 ------- 9413 ------- 9454 ------- 9505 ------- 9567 ------- 9610 ------- 9657 ------- 9704 ------- 9762 ------- 9806 ------- 9859 ------- 9913 ------- 9951 ------- 9995
   │         │    │    │    │    │    │    │    │   histogram(19)=  0      600       27605      2400      28205      1800      27605      3000      25805      4200      27605      2400      27605      3600      28805      2400      28805      3600      28805      3000      28205      2400      29405      4200      28805      3000      28805      2400      29405      1200      27605      4800      28805      2400      28805      2400      28805      4800      28205      3000      26405      3600      24604      5401      27605      3000      28805      2400      29405      1800      28205      1800      24004      6001      28805      3600      25805      3600      27005      3000      26405      4200      28805      3000      27005      2400      27005      2400      28205      4800      28205      2400      25805      4200      25805      3600      27605      3000      27005      3000      27605      4200      27605      2400      27005      4200      27605      4800      27605      1800      28805      2400      28205      3000      27605      3600      28205      1200      24604      4800      28205      1800      28805      2400      28805      2400      27605      1800      28205      3600      25805      3600      26405      6001      28205      3000      27605      1800      27005      2400      28205      2400      28205      3000      28805      3600      28205      1800      27005      2400      27005      4800      28805      2400      27605      4200      25205      4200      28805      1200      27605      4200      27605      3000      27005      4800      27605      2400      28205      1800      28805      4200      28205      1800      26405      3000      28205      1200      26405      5401      28205      3600      28205      1800      27005      3000      26405      3600      25205      4200      25205      6001      26405      4200      27005      2400      26405      4800      25805      4200      28205      4200      27605      3600      28205      3000      27005      4200      28805      2400      24004      5401      28805      3600      26405      3000      27005      3000      27605      3000      24604      6001      28805      3000      25805      4200      27605      3600      28805      1200      26405      3600      28805      3000      27605      3600      23404      6601      25805      3600      27605      3600      26405      3000      26405      3000      28805      3600      26405      4800      26405      3000      27005      3000      24604      4800      27005      3600      28205      3600      27605      2400      25805      3000      26405      3000      26405      2400      28205      3600      27005      5401      27605      2400      25205      4800      27605      2400      28205      2400      27605      1800      25805      3600      27605      3600      26405      2400      27605      3000      24004      4800      25805      4200      28205      3000      25205      4200      28205      4200      27605      3000      26405      4800      28205      3000      27005      3600      25205      3600      27005      1800      24604      4200      27605      3600      27005      3000      26405      2400      24604      4800      27605      1800      25205      3000      25805      3000      27005      3600      26405      1800      25205      3600      27605      600       26405      3000      26405      4800      27605      3600      27605      2400      27005      3000      26405      3600      27605      2400      26405      3000      25805      4200      25805      3000      25805      3000      27005      3600      27005      1800      25805      3000      26405      2400      25805      4200      26405      4200      26405      1800      25805      2400      26405      2400      24604      3600      27005      2400      25805      3000      25805      2400      27005      3600      24004      3600      25205      3000      25205      2400      25805      3000      27005      1200      25205      3600      24604      2400      27005      3600      24604      3000      26405      4800      25805      3000      24004      1800      25805      2400      22804      4800      23404      3000      24004      1800      23404      600
   │         │    │    │    │    │    │    │    │                 <--- '1992-02-05' ------- '1992-03-10' ------- '1992-03-27' ------- '1992-04-07' ------- '1992-04-18' ------- '1992-05-02' ------- '1992-05-14' ------- '1992-05-25' ------- '1992-06-08' ------- '1992-06-21' ------- '1992-07-04' ------- '1992-07-15' ------- '1992-07-29' ------- '1992-08-09' ------- '1992-08-21' ------- '1992-09-03' ------- '1992-09-16' ------- '1992-10-01' ------- '1992-10-15' ------- '1992-10-29' ------- '1992-11-06' ------- '1992-11-18' ------- '1992-11-28' ------- '1992-12-09' ------- '1992-12-22' ------- '1993-01-03' ------- '1993-01-13' ------- '1993-01-27' ------- '1993-02-08' ------- '1993-02-20' ------- '1993-03-04' ------- '1993-03-17' ------- '1993-03-27' ------- '1993-04-10' ------- '1993-04-22' ------- '1993-05-05' ------- '1993-05-16' ------- '1993-05-28' ------- '1993-06-07' ------- '1993-06-22' ------- '1993-07-02' ------- '1993-07-19' ------- '1993-08-02' ------- '1993-08-14' ------- '1993-08-24' ------- '1993-09-08' ------- '1993-09-20' ------- '1993-10-03' ------- '1993-10-16' ------- '1993-10-27' ------- '1993-11-09' ------- '1993-11-21' ------- '1993-12-10' ------- '1993-12-22' ------- '1994-01-05' ------- '1994-01-17' ------- '1994-01-27' ------- '1994-02-10' ------- '1994-02-20' ------- '1994-03-05' ------- '1994-03-15' ------- '1994-03-28' ------- '1994-04-08' ------- '1994-04-23' ------- '1994-05-05' ------- '1994-05-16' ------- '1994-05-29' ------- '1994-06-10' ------- '1994-06-22' ------- '1994-07-04' ------- '1994-07-18' ------- '1994-08-01' ------- '1994-08-10' ------- '1994-08-19' ------- '1994-09-03' ------- '1994-09-14' ------- '1994-09-24' ------- '1994-10-02' ------- '1994-10-16' ------- '1994-10-30' ------- '1994-11-11' ------- '1994-11-26' ------- '1994-12-10' ------- '1994-12-22' ------- '1995-01-03' ------- '1995-01-16' ------- '1995-01-26' ------- '1995-02-06' ------- '1995-02-16' ------- '1995-02-28' ------- '1995-03-10' ------- '1995-03-21' ------- '1995-04-02' ------- '1995-04-12' ------- '1995-04-24' ------- '1995-05-07' ------- '1995-05-22' ------- '1995-06-04' ------- '1995-06-17' ------- '1995-06-30' ------- '1995-07-10' ------- '1995-07-25' ------- '1995-08-05' ------- '1995-08-16' ------- '1995-08-28' ------- '1995-09-07' ------- '1995-09-21' ------- '1995-10-04' ------- '1995-10-13' ------- '1995-10-27' ------- '1995-11-09' ------- '1995-11-21' ------- '1995-12-02' ------- '1995-12-16' ------- '1995-12-28' ------- '1996-01-08' ------- '1996-01-20' ------- '1996-01-31' ------- '1996-02-13' ------- '1996-02-26' ------- '1996-03-10' ------- '1996-03-20' ------- '1996-04-03' ------- '1996-04-17' ------- '1996-04-29' ------- '1996-05-11' ------- '1996-05-25' ------- '1996-06-03' ------- '1996-06-13' ------- '1996-06-26' ------- '1996-07-10' ------- '1996-07-19' ------- '1996-07-31' ------- '1996-08-15' ------- '1996-08-27' ------- '1996-09-05' ------- '1996-09-16' ------- '1996-09-26' ------- '1996-10-05' ------- '1996-10-20' ------- '1996-11-02' ------- '1996-11-13' ------- '1996-11-25' ------- '1996-12-08' ------- '1996-12-21' ------- '1997-01-01' ------- '1997-01-13' ------- '1997-01-26' ------- '1997-02-05' ------- '1997-02-16' ------- '1997-02-28' ------- '1997-03-11' ------- '1997-03-24' ------- '1997-04-05' ------- '1997-04-16' ------- '1997-05-02' ------- '1997-05-13' ------- '1997-05-24' ------- '1997-06-07' ------- '1997-06-19' ------- '1997-07-03' ------- '1997-07-13' ------- '1997-07-23' ------- '1997-08-04' ------- '1997-08-17' ------- '1997-08-28' ------- '1997-09-10' ------- '1997-09-21' ------- '1997-10-04' ------- '1997-10-17' ------- '1997-10-28' ------- '1997-11-08' ------- '1997-11-16' ------- '1997-11-29' ------- '1997-12-11' ------- '1997-12-24' ------- '1998-01-04' ------- '1998-01-16' ------- '1998-01-27' ------- '1998-02-05' ------- '1998-02-17' ------- '1998-03-04' ------- '1998-03-19' ------- '1998-03-31' ------- '1998-04-09' ------- '1998-04-21' ------- '1998-05-02' ------- '1998-05-15' ------- '1998-05-25' ------- '1998-06-07' ------- '1998-06-20' ------- '1998-07-01' ------- '1998-07-12' ------- '1998-07-24' ------- '1998-08-06' ------- '1998-08-18' ------- '1998-08-30' ------- '1998-09-13' ------- '1998-09-26' ------- '1998-10-29'
   │         │    │    │    │    │    │    │    │   histogram(20)=  0      600       28805      1800      27605      2400      28805      2400      28205      1800      27605      2400      29405      600       29405      2400      28805      4200      27605      2400      27605      4200      28805      3000      29405      1200      27605      2400      29405      4200      28205      1800      26405      4200      27605      2400      28205      3000      27605      2400      27605      3000      28805      6001      29405      4200      25805      5401      25805      6001      29405      3000      28805      1800      27005      3000      28805      3000      27605      2400      28205      4800      28805      3600      28205      1800      28205      1200      27005      3000      28805      1800      28205      1800      28805      1200      27005      3600      27605      2400      27605      4800      26405      3600      28805      1200      28205      2400      28205      1800      28805      600       28805      1800      27605      1800      28805      1800      27605      1800      27605      3000      28805      2400      28805      2400      27605      3000      28805      1800      27005      4200      28805      3000      27605      2400      28205      5401      28205      4800      27605      3000      26405      5401      28205      6001      25205      4200      25805      3600      27005      3000      27605      2400      28805      1800      28805      3000      28205      4800      27005      2400      26405      3000      26405      3000      28805      2400      28205      3000      28805      2400      25805      4200      26405      3600      26405      3000      27605      1800      28805      2400      27005      3600      28205      3000      28205      1200      27005      2400      25805      4200      28205      2400      28205      3000      28205      1800      27005      5401      28205      3600      25205      4200      27605      3600      27605      1800      25805      3600      28805      1800      28805      1800      25205      4200      28205      1200      28205      1800      28805      3000      27005      2400      25805      4800      27605      1800      25805      4200      27605      1800      27605      4200      28205      1800      27005      3000      27605      2400      28205      3600      27605      1800      28805      1200      27605      3600      28805      1200      27005      4200      27605      5401      27005      3600      26405      3600      26405      3000      25805      4800      28205      2400      27005      3000      27605      3600      28205      1800      26405      2400      27605      1800      25205      3600      28205      1800      25805      3000      27605      1800      24604      4800      27605      2400      28205      1800      28205      600       26405      2400      26405      3600      27005      3000      28205      4200      27605      2400      28205      1200      27005      1800      27005      2400      27005      3000      21604      7801      28205      2400      25805      3000      27605      4200      26405      6001      28205      2400      26405      4800      25805      5401      27605      3600      28205      600       28205      1800      25805      3000      27005      1800      28205      1800      23404      5401      27605      600       27005      3600      24604      3600      26405      3600      27005      3000      25805      2400      27005      3000      24604      3600      25205      3000      27005      4800      23404      7201      26405      3000      27605      2400      27605      3600      26405      5401      27605      1800      27005      3000      26405      2400      27005      4800      27005      3000      27005      2400      25805      1800      25805      3600      27005      1200      24604      3000      25205      1800      25805      1800      26405      1200      26405      2400      25205      4800      24604      3000      23404      5401      25805      2400      24004      2400      26405      4800      21604      4200      22204      4200      25205      3000      25205      600       24604      1800      24604      600
   │         │    │    │    │    │    │    │    │                 <--- '1992-01-21' ------- '1992-03-15' ------- '1992-04-05' ------- '1992-04-20' ------- '1992-05-05' ------- '1992-05-18' ------- '1992-06-02' ------- '1992-06-17' ------- '1992-06-30' ------- '1992-07-11' ------- '1992-07-20' ------- '1992-08-01' ------- '1992-08-14' ------- '1992-08-27' ------- '1992-09-08' ------- '1992-09-21' ------- '1992-10-02' ------- '1992-10-13' ------- '1992-10-25' ------- '1992-11-05' ------- '1992-11-19' ------- '1992-12-03' ------- '1992-12-17' ------- '1992-12-27' ------- '1993-01-07' ------- '1993-01-18' ------- '1993-02-01' ------- '1993-02-10' ------- '1993-02-23' ------- '1993-03-05' ------- '1993-03-19' ------- '1993-04-01' ------- '1993-04-14' ------- '1993-04-28' ------- '1993-05-09' ------- '1993-05-22' ------- '1993-06-06' ------- '1993-06-17' ------- '1993-07-02' ------- '1993-07-13' ------- '1993-07-24' ------- '1993-08-04' ------- '1993-08-19' ------- '1993-08-29' ------- '1993-09-10' ------- '1993-09-23' ------- '1993-10-04' ------- '1993-10-18' ------- '1993-10-31' ------- '1993-11-14' ------- '1993-11-26' ------- '1993-12-10' ------- '1993-12-25' ------- '1994-01-09' ------- '1994-01-23' ------- '1994-02-02' ------- '1994-02-15' ------- '1994-02-27' ------- '1994-03-12' ------- '1994-03-22' ------- '1994-04-02' ------- '1994-04-12' ------- '1994-04-24' ------- '1994-05-08' ------- '1994-05-21' ------- '1994-06-03' ------- '1994-06-16' ------- '1994-06-27' ------- '1994-07-09' ------- '1994-07-25' ------- '1994-08-05' ------- '1994-08-14' ------- '1994-08-26' ------- '1994-09-08' ------- '1994-09-18' ------- '1994-09-29' ------- '1994-10-11' ------- '1994-10-25' ------- '1994-11-07' ------- '1994-11-20' ------- '1994-12-02' ------- '1994-12-13' ------- '1994-12-25' ------- '1995-01-08' ------- '1995-01-20' ------- '1995-01-31' ------- '1995-02-12' ------- '1995-02-22' ------- '1995-03-04' ------- '1995-03-16' ------- '1995-03-28' ------- '1995-04-07' ------- '1995-04-21' ------- '1995-05-02' ------- '1995-05-12' ------- '1995-05-27' ------- '1995-06-10' ------- '1995-06-22' ------- '1995-07-03' ------- '1995-07-16' ------- '1995-07-29' ------- '1995-08-08' ------- '1995-08-22' ------- '1995-09-03' ------- '1995-09-13' ------- '1995-09-24' ------- '1995-10-09' ------- '1995-10-19' ------- '1995-10-31' ------- '1995-11-13' ------- '1995-11-23' ------- '1995-12-03' ------- '1995-12-15' ------- '1995-12-29' ------- '1996-01-11' ------- '1996-01-21' ------- '1996-02-06' ------- '1996-02-17' ------- '1996-02-29' ------- '1996-03-15' ------- '1996-03-28' ------- '1996-04-09' ------- '1996-04-19' ------- '1996-05-04' ------- '1996-05-16' ------- '1996-05-27' ------- '1996-06-07' ------- '1996-06-17' ------- '1996-06-29' ------- '1996-07-09' ------- '1996-07-22' ------- '1996-08-01' ------- '1996-08-12' ------- '1996-08-23' ------- '1996-09-03' ------- '1996-09-18' ------- '1996-09-28' ------- '1996-10-12' ------- '1996-10-24' ------- '1996-11-04' ------- '1996-11-17' ------- '1996-11-27' ------- '1996-12-08' ------- '1996-12-17' ------- '1996-12-26' ------- '1997-01-08' ------- '1997-01-22' ------- '1997-02-04' ------- '1997-02-15' ------- '1997-03-01' ------- '1997-03-16' ------- '1997-03-27' ------- '1997-04-08' ------- '1997-04-20' ------- '1997-05-03' ------- '1997-05-18' ------- '1997-05-29' ------- '1997-06-10' ------- '1997-06-22' ------- '1997-07-05' ------- '1997-07-17' ------- '1997-07-31' ------- '1997-08-12' ------- '1997-08-23' ------- '1997-09-02' ------- '1997-09-12' ------- '1997-09-24' ------- '1997-10-05' ------- '1997-10-14' ------- '1997-10-26' ------- '1997-11-06' ------- '1997-11-19' ------- '1997-12-01' ------- '1997-12-15' ------- '1997-12-27' ------- '1998-01-07' ------- '1998-01-18' ------- '1998-02-02' ------- '1998-02-19' ------- '1998-03-04' ------- '1998-03-18' ------- '1998-03-29' ------- '1998-04-10' ------- '1998-04-20' ------- '1998-05-02' ------- '1998-05-11' ------- '1998-05-24' ------- '1998-06-04' ------- '1998-06-16' ------- '1998-06-27' ------- '1998-07-08' ------- '1998-07-20' ------- '1998-07-31' ------- '1998-08-10' ------- '1998-08-21' ------- '1998-09-04' ------- '1998-09-20' ------- '1998-10-06' ------- '1998-10-28' ------- '1998-12-11'
   │         │    │    │    │    │    │    │    ├── cost: 7201458.02
   │         │    │    │    │    │    │    │    ├── ordering: +8
   │         │    │    │    │    │    │    │    ├── prune: (8,10,19,20)
   │         │    │    │    │    │    │    │    └── interesting orderings: (+8) (+10,+8) (+19,+8) (+20,+8)
   │         │    │    │    │    │    │    └── filters
   │         │    │    │    │    │    │         └── l1.l_receiptdate > l1.l_commitdate [outer=(19,20), constraints=(/19: (/NULL - ]; /20: (/NULL - ])]
   │         │    │    │    │    │    ├── select
   │         │    │    │    │    │    │    ├── columns: l3.l_orderkey:53 l3.l_commitdate:64 l3.l_receiptdate:65
   │         │    │    │    │    │    │    ├── stats: [rows=2000405, distinct(53)=1216823.04, null(53)=0, distinct(64)=2466, null(64)=0, distinct(65)=2554, null(65)=0]
   │         │    │    │    │    │    │    ├── cost: 7201458.03
   │         │    │    │    │    │    │    ├── ordering: +53
   │         │    │    │    │    │    │    ├── prune: (53)
   │         │    │    │    │    │    │    ├── interesting orderings: (+53) (+64,+53) (+65,+53)
   │         │    │    │    │    │    │    ├── scan l3
   │         │    │    │    │    │    │    │    ├── columns: l3.l_orderkey:53 l3.l_commitdate:64 l3.l_receiptdate:65
   │         │    │    │    │    │    │    │    ├── stats: [rows=6001215, distinct(53)=1527270, null(53)=0, distinct(64)=2466, null(64)=0, distinct(65)=2554, null(65)=0]
   │         │    │    │    │    │    │    │    │   histogram(53)=  0  600  29405   600   29405   600   29405   600   29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405   600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    29405    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600    30006    600
   │         │    │    │    │    │    │    │    │                 <--- 259 ------- 26467 ------- 59207 ------- 92198 ------- 119297 ------- 152835 ------- 192930 ------- 216613 ------- 250049 ------- 283009 ------- 316674 ------- 343492 ------- 370081 ------- 398978 ------- 427136 ------- 451940 ------- 484640 ------- 516327 ------- 543776 ------- 574273 ------- 607682 ------- 635431 ------- 670788 ------- 699265 ------- 719654 ------- 742917 ------- 771012 ------- 796770 ------- 826112 ------- 850951 ------- 877543 ------- 909604 ------- 942918 ------- 970532 ------- 1000391 ------- 1030214 ------- 1059907 ------- 1092641 ------- 1120068 ------- 1154695 ------- 1185507 ------- 1209666 ------- 1243234 ------- 1275557 ------- 1307142 ------- 1331973 ------- 1368615 ------- 1396323 ------- 1423010 ------- 1456164 ------- 1486917 ------- 1516550 ------- 1552549 ------- 1579585 ------- 1612455 ------- 1642531 ------- 1676580 ------- 1705603 ------- 1737379 ------- 1763591 ------- 1790245 ------- 1819335 ------- 1844710 ------- 1864612 ------- 1889926 ------- 1928295 ------- 1965473 ------- 1998983 ------- 2027527 ------- 2048551 ------- 2078854 ------- 2106083 ------- 2135556 ------- 2164289 ------- 2206340 ------- 2233472 ------- 2260064 ------- 2300770 ------- 2330023 ------- 2357092 ------- 2388230 ------- 2426433 ------- 2463365 ------- 2496390 ------- 2520805 ------- 2544741 ------- 2574656 ------- 2601922 ------- 2630337 ------- 2661090 ------- 2688837 ------- 2717091 ------- 2753248 ------- 2784514 ------- 2815654 ------- 2843492 ------- 2865159 ------- 2890210 ------- 2918375 ------- 2954336 ------- 2980678 ------- 3005669 ------- 3038340 ------- 3070563 ------- 3100709 ------- 3132295 ------- 3162914 ------- 3196743 ------- 3234852 ------- 3260454 ------- 3295427 ------- 3323842 ------- 3345668 ------- 3372196 ------- 3402884 ------- 3440258 ------- 3464801 ------- 3492229 ------- 3520742 ------- 3554529 ------- 3587654 ------- 3613602 ------- 3645282 ------- 3674690 ------- 3709539 ------- 3737734 ------- 3769639 ------- 3803009 ------- 3828197 ------- 3859718 ------- 3887847 ------- 3921028 ------- 3953764 ------- 3991430 ------- 4024257 ------- 4051394 ------- 4089348 ------- 4119110 ------- 4146818 ------- 4179524 ------- 4202784 ------- 4233312 ------- 4264738 ------- 4294821 ------- 4315171 ------- 4335330 ------- 4365093 ------- 4393541 ------- 4418692 ------- 4444096 ------- 4476134 ------- 4514599 ------- 4541538 ------- 4572485 ------- 4601888 ------- 4631015 ------- 4667013 ------- 4692935 ------- 4731204 ------- 4757383 ------- 4786823 ------- 4816707 ------- 4844615 ------- 4875205 ------- 4906914 ------- 4935015 ------- 4965024 ------- 4989671 ------- 5022531 ------- 5056038 ------- 5086177 ------- 5123200 ------- 5151108 ------- 5184928 ------- 5214022 ------- 5244417 ------- 5272033 ------- 5303910 ------- 5347521 ------- 5375653 ------- 5404514 ------- 5429543 ------- 5460707 ------- 5496513 ------- 5527555 ------- 5555686 ------- 5592868 ------- 5635456 ------- 5661925 ------- 5692225 ------- 5722148 ------- 5751811 ------- 5790023 ------- 5824256 ------- 5854533 ------- 5885762 ------- 5915648 ------- 5947713 ------- 5972262 ------- 5999617
   │         │    │    │    │    │    │    │    │   histogram(64)=  0      600       27605      2400      28205      1800      27605      3000      25805      4200      27605      2400      27605      3600      28805      2400      28805      3600      28805      3000      28205      2400      29405      4200      28805      3000      28805      2400      29405      1200      27605      4800      28805      2400      28805      2400      28805      4800      28205      3000      26405      3600      24604      5401      27605      3000      28805      2400      29405      1800      28205      1800      24004      6001      28805      3600      25805      3600      27005      3000      26405      4200      28805      3000      27005      2400      27005      2400      28205      4800      28205      2400      25805      4200      25805      3600      27605      3000      27005      3000      27605      4200      27605      2400      27005      4200      27605      4800      27605      1800      28805      2400      28205      3000      27605      3600      28205      1200      24604      4800      28205      1800      28805      2400      28805      2400      27605      1800      28205      3600      25805      3600      26405      6001      28205      3000      27605      1800      27005      2400      28205      2400      28205      3000      28805      3600      28205      1800      27005      2400      27005      4800      28805      2400      27605      4200      25205      4200      28805      1200      27605      4200      27605      3000      27005      4800      27605      2400      28205      1800      28805      4200      28205      1800      26405      3000      28205      1200      26405      5401      28205      3600      28205      1800      27005      3000      26405      3600      25205      4200      25205      6001      26405      4200      27005      2400      26405      4800      25805      4200      28205      4200      27605      3600      28205      3000      27005      4200      28805      2400      24004      5401      28805      3600      26405      3000      27005      3000      27605      3000      24604      6001      28805      3000      25805      4200      27605      3600      28805      1200      26405      3600      28805      3000      27605      3600      23404      6601      25805      3600      27605      3600      26405      3000      26405      3000      28805      3600      26405      4800      26405      3000      27005      3000      24604      4800      27005      3600      28205      3600      27605      2400      25805      3000      26405      3000      26405      2400      28205      3600      27005      5401      27605      2400      25205      4800      27605      2400      28205      2400      27605      1800      25805      3600      27605      3600      26405      2400      27605      3000      24004      4800      25805      4200      28205      3000      25205      4200      28205      4200      27605      3000      26405      4800      28205      3000      27005      3600      25205      3600      27005      1800      24604      4200      27605      3600      27005      3000      26405      2400      24604      4800      27605      1800      25205      3000      25805      3000      27005      3600      26405      1800      25205      3600      27605      600       26405      3000      26405      4800      27605      3600      27605      2400      27005      3000      26405      3600      27605      2400      26405      3000      25805      4200      25805      3000      25805      3000      27005      3600      27005      1800      25805      3000      26405      2400      25805      4200      26405      4200      26405      1800      25805      2400      26405      2400      24604      3600      27005      2400      25805      3000      25805      2400      27005      3600      24004      3600      25205      3000      25205      2400      25805      3000      27005      1200      25205      3600      24604      2400      27005      3600      24604      3000      26405      4800      25805      3000      24004      1800      25805      2400      22804      4800      23404      3000      24004      1800      23404      600
   │         │    │    │    │    │    │    │    │                 <--- '1992-02-05' ------- '1992-03-10' ------- '1992-03-27' ------- '1992-04-07' ------- '1992-04-18' ------- '1992-05-02' ------- '1992-05-14' ------- '1992-05-25' ------- '1992-06-08' ------- '1992-06-21' ------- '1992-07-04' ------- '1992-07-15' ------- '1992-07-29' ------- '1992-08-09' ------- '1992-08-21' ------- '1992-09-03' ------- '1992-09-16' ------- '1992-10-01' ------- '1992-10-15' ------- '1992-10-29' ------- '1992-11-06' ------- '1992-11-18' ------- '1992-11-28' ------- '1992-12-09' ------- '1992-12-22' ------- '1993-01-03' ------- '1993-01-13' ------- '1993-01-27' ------- '1993-02-08' ------- '1993-02-20' ------- '1993-03-04' ------- '1993-03-17' ------- '1993-03-27' ------- '1993-04-10' ------- '1993-04-22' ------- '1993-05-05' ------- '1993-05-16' ------- '1993-05-28' ------- '1993-06-07' ------- '1993-06-22' ------- '1993-07-02' ------- '1993-07-19' ------- '1993-08-02' ------- '1993-08-14' ------- '1993-08-24' ------- '1993-09-08' ------- '1993-09-20' ------- '1993-10-03' ------- '1993-10-16' ------- '1993-10-27' ------- '1993-11-09' ------- '1993-11-21' ------- '1993-12-10' ------- '1993-12-22' ------- '1994-01-05' ------- '1994-01-17' ------- '1994-01-27' ------- '1994-02-10' ------- '1994-02-20' ------- '1994-03-05' ------- '1994-03-15' ------- '1994-03-28' ------- '1994-04-08' ------- '1994-04-23' ------- '1994-05-05' ------- '1994-05-16' ------- '1994-05-29' ------- '1994-06-10' ------- '1994-06-22' ------- '1994-07-04' ------- '1994-07-18' ------- '1994-08-01' ------- '1994-08-10' ------- '1994-08-19' ------- '1994-09-03' ------- '1994-09-14' ------- '1994-09-24' ------- '1994-10-02' ------- '1994-10-16' ------- '1994-10-30' ------- '1994-11-11' ------- '1994-11-26' ------- '1994-12-10' ------- '1994-12-22' ------- '1995-01-03' ------- '1995-01-16' ------- '1995-01-26' ------- '1995-02-06' ------- '1995-02-16' ------- '1995-02-28' ------- '1995-03-10' ------- '1995-03-21' ------- '1995-04-02' ------- '1995-04-12' ------- '1995-04-24' ------- '1995-05-07' ------- '1995-05-22' ------- '1995-06-04' ------- '1995-06-17' ------- '1995-06-30' ------- '1995-07-10' ------- '1995-07-25' ------- '1995-08-05' ------- '1995-08-16' ------- '1995-08-28' ------- '1995-09-07' ------- '1995-09-21' ------- '1995-10-04' ------- '1995-10-13' ------- '1995-10-27' ------- '1995-11-09' ------- '1995-11-21' ------- '1995-12-02' ------- '1995-12-16' ------- '1995-12-28' ------- '1996-01-08' ------- '1996-01-20' ------- '1996-01-31' ------- '1996-02-13' ------- '1996-02-26' ------- '1996-03-10' ------- '1996-03-20' ------- '1996-04-03' ------- '1996-04-17' ------- '1996-04-29' ------- '1996-05-11' ------- '1996-05-25' ------- '1996-06-03' ------- '1996-06-13' ------- '1996-06-26' ------- '1996-07-10' ------- '1996-07-19' ------- '1996-07-31' ------- '1996-08-15' ------- '1996-08-27' ------- '1996-09-05' ------- '1996-09-16' ------- '1996-09-26' ------- '1996-10-05' ------- '1996-10-20' ------- '1996-11-02' ------- '1996-11-13' ------- '1996-11-25' ------- '1996-12-08' ------- '1996-12-21' ------- '1997-01-01' ------- '1997-01-13' ------- '1997-01-26' ------- '1997-02-05' ------- '1997-02-16' ------- '1997-02-28' ------- '1997-03-11' ------- '1997-03-24' ------- '1997-04-05' ------- '1997-04-16' ------- '1997-05-02' ------- '1997-05-13' ------- '1997-05-24' ------- '1997-06-07' ------- '1997-06-19' ------- '1997-07-03' ------- '1997-07-13' ------- '1997-07-23' ------- '1997-08-04' ------- '1997-08-17' ------- '1997-08-28' ------- '1997-09-10' ------- '1997-09-21' ------- '1997-10-04' ------- '1997-10-17' ------- '1997-10-28' ------- '1997-11-08' ------- '1997-11-16' ------- '1997-11-29' ------- '1997-12-11' ------- '1997-12-24' ------- '1998-01-04' ------- '1998-01-16' ------- '1998-01-27' ------- '1998-02-05' ------- '1998-02-17' ------- '1998-03-04' ------- '1998-03-19' ------- '1998-03-31' ------- '1998-04-09' ------- '1998-04-21' ------- '1998-05-02' ------- '1998-05-15' ------- '1998-05-25' ------- '1998-06-07' ------- '1998-06-20' ------- '1998-07-01' ------- '1998-07-12' ------- '1998-07-24' ------- '1998-08-06' ------- '1998-08-18' ------- '1998-08-30' ------- '1998-09-13' ------- '1998-09-26' ------- '1998-10-29'
   │         │    │    │    │    │    │    │    │   histogram(65)=  0      600       28805      1800      27605      2400      28805      2400      28205      1800      27605      2400      29405      600       29405      2400      28805      4200      27605      2400      27605      4200      28805      3000      29405      1200      27605      2400      29405      4200      28205      1800      26405      4200      27605      2400      28205      3000      27605      2400      27605      3000      28805      6001      29405      4200      25805      5401      25805      6001      29405      3000      28805      1800      27005      3000      28805      3000      27605      2400      28205      4800      28805      3600      28205      1800      28205      1200      27005      3000      28805      1800      28205      1800      28805      1200      27005      3600      27605      2400      27605      4800      26405      3600      28805      1200      28205      2400      28205      1800      28805      600       28805      1800      27605      1800      28805      1800      27605      1800      27605      3000      28805      2400      28805      2400      27605      3000      28805      1800      27005      4200      28805      3000      27605      2400      28205      5401      28205      4800      27605      3000      26405      5401      28205      6001      25205      4200      25805      3600      27005      3000      27605      2400      28805      1800      28805      3000      28205      4800      27005      2400      26405      3000      26405      3000      28805      2400      28205      3000      28805      2400      25805      4200      26405      3600      26405      3000      27605      1800      28805      2400      27005      3600      28205      3000      28205      1200      27005      2400      25805      4200      28205      2400      28205      3000      28205      1800      27005      5401      28205      3600      25205      4200      27605      3600      27605      1800      25805      3600      28805      1800      28805      1800      25205      4200      28205      1200      28205      1800      28805      3000      27005      2400      25805      4800      27605      1800      25805      4200      27605      1800      27605      4200      28205      1800      27005      3000      27605      2400      28205      3600      27605      1800      28805      1200      27605      3600      28805      1200      27005      4200      27605      5401      27005      3600      26405      3600      26405      3000      25805      4800      28205      2400      27005      3000      27605      3600      28205      1800      26405      2400      27605      1800      25205      3600      28205      1800      25805      3000      27605      1800      24604      4800      27605      2400      28205      1800      28205      600       26405      2400      26405      3600      27005      3000      28205      4200      27605      2400      28205      1200      27005      1800      27005      2400      27005      3000      21604      7801      28205      2400      25805      3000      27605      4200      26405      6001      28205      2400      26405      4800      25805      5401      27605      3600      28205      600       28205      1800      25805      3000      27005      1800      28205      1800      23404      5401      27605      600       27005      3600      24604      3600      26405      3600      27005      3000      25805      2400      27005      3000      24604      3600      25205      3000      27005      4800      23404      7201      26405      3000      27605      2400      27605      3600      26405      5401      27605      1800      27005      3000      26405      2400      27005      4800      27005      3000      27005      2400      25805      1800      25805      3600      27005      1200      24604      3000      25205      1800      25805      1800      26405      1200      26405      2400      25205      4800      24604      3000      23404      5401      25805      2400      24004      2400      26405      4800      21604      4200      22204      4200      25205      3000      25205      600       24604      1800      24604      600
   │         │    │    │    │    │    │    │    │                 <--- '1992-01-21' ------- '1992-03-15' ------- '1992-04-05' ------- '1992-04-20' ------- '1992-05-05' ------- '1992-05-18' ------- '1992-06-02' ------- '1992-06-17' ------- '1992-06-30' ------- '1992-07-11' ------- '1992-07-20' ------- '1992-08-01' ------- '1992-08-14' ------- '1992-08-27' ------- '1992-09-08' ------- '1992-09-21' ------- '1992-10-02' ------- '1992-10-13' ------- '1992-10-25' ------- '1992-11-05' ------- '1992-11-19' ------- '1992-12-03' ------- '1992-12-17' ------- '1992-12-27' ------- '1993-01-07' ------- '1993-01-18' ------- '1993-02-01' ------- '1993-02-10' ------- '1993-02-23' ------- '1993-03-05' ------- '1993-03-19' ------- '1993-04-01' ------- '1993-04-14' ------- '1993-04-28' ------- '1993-05-09' ------- '1993-05-22' ------- '1993-06-06' ------- '1993-06-17' ------- '1993-07-02' ------- '1993-07-13' ------- '1993-07-24' ------- '1993-08-04' ------- '1993-08-19' ------- '1993-08-29' ------- '1993-09-10' ------- '1993-09-23' ------- '1993-10-04' ------- '1993-10-18' ------- '1993-10-31' ------- '1993-11-14' ------- '1993-11-26' ------- '1993-12-10' ------- '1993-12-25' ------- '1994-01-09' ------- '1994-01-23' ------- '1994-02-02' ------- '1994-02-15' ------- '1994-02-27' ------- '1994-03-12' ------- '1994-03-22' ------- '1994-04-02' ------- '1994-04-12' ------- '1994-04-24' ------- '1994-05-08' ------- '1994-05-21' ------- '1994-06-03' ------- '1994-06-16' ------- '1994-06-27' ------- '1994-07-09' ------- '1994-07-25' ------- '1994-08-05' ------- '1994-08-14' ------- '1994-08-26' ------- '1994-09-08' ------- '1994-09-18' ------- '1994-09-29' ------- '1994-10-11' ------- '1994-10-25' ------- '1994-11-07' ------- '1994-11-20' ------- '1994-12-02' ------- '1994-12-13' ------- '1994-12-25' ------- '1995-01-08' ------- '1995-01-20' ------- '1995-01-31' ------- '1995-02-12' ------- '1995-02-22' ------- '1995-03-04' ------- '1995-03-16' ------- '1995-03-28' ------- '1995-04-07' ------- '1995-04-21' ------- '1995-05-02' ------- '1995-05-12' ------- '1995-05-27' ------- '1995-06-10' ------- '1995-06-22' ------- '1995-07-03' ------- '1995-07-16' ------- '1995-07-29' ------- '1995-08-08' ------- '1995-08-22' ------- '1995-09-03' ------- '1995-09-13' ------- '1995-09-24' ------- '1995-10-09' ------- '1995-10-19' ------- '1995-10-31' ------- '1995-11-13' ------- '1995-11-23' ------- '1995-12-03' ------- '1995-12-15' ------- '1995-12-29' ------- '1996-01-11' ------- '1996-01-21' ------- '1996-02-06' ------- '1996-02-17' ------- '1996-02-29' ------- '1996-03-15' ------- '1996-03-28' ------- '1996-04-09' ------- '1996-04-19' ------- '1996-05-04' ------- '1996-05-16' ------- '1996-05-27' ------- '1996-06-07' ------- '1996-06-17' ------- '1996-06-29' ------- '1996-07-09' ------- '1996-07-22' ------- '1996-08-01' ------- '1996-08-12' ------- '1996-08-23' ------- '1996-09-03' ------- '1996-09-18' ------- '1996-09-28' ------- '1996-10-12' ------- '1996-10-24' ------- '1996-11-04' ------- '1996-11-17' ------- '1996-11-27' ------- '1996-12-08' ------- '1996-12-17' ------- '1996-12-26' ------- '1997-01-08' ------- '1997-01-22' ------- '1997-02-04' ------- '1997-02-15' ------- '1997-03-01' ------- '1997-03-16' ------- '1997-03-27' ------- '1997-04-08' ------- '1997-04-20' ------- '1997-05-03' ------- '1997-05-18' ------- '1997-05-29' ------- '1997-06-10' ------- '1997-06-22' ------- '1997-07-05' ------- '1997-07-17' ------- '1997-07-31' ------- '1997-08-12' ------- '1997-08-23' ------- '1997-09-02' ------- '1997-09-12' ------- '1997-09-24' ------- '1997-10-05' ------- '1997-10-14' ------- '1997-10-26' ------- '1997-11-06' ------- '1997-11-19' ------- '1997-12-01' ------- '1997-12-15' ------- '1997-12-27' ------- '1998-01-07' ------- '1998-01-18' ------- '1998-02-02' ------- '1998-02-19' ------- '1998-03-04' ------- '1998-03-18' ------- '1998-03-29' ------- '1998-04-10' ------- '1998-04-20' ------- '1998-05-02' ------- '1998-05-11' ------- '1998-05-24' ------- '1998-06-04' ------- '1998-06-16' ------- '1998-06-27' ------- '1998-07-08' ------- '1998-07-20' ------- '1998-07-31' ------- '1998-08-10' ------- '1998-08-21' ------- '1998-09-04' ------- '1998-09-20' ------- '1998-10-06' ------- '1998-10-28' ------- '1998-12-11'
   │         │    │    │    │    │    │    │    ├── cost: 7141445.87
   │         │    │    │    │    │    │    │    ├── ordering: +53
   │         │    │    │    │    │    │    │    ├── prune: (53,64,65)
   │         │    │    │    │    │    │    │    └── interesting orderings: (+53) (+64,+53) (+65,+53)
   │         │    │    │    │    │    │    └── filters
   │         │    │    │    │    │    │         └── l3.l_receiptdate > l3.l_commitdate [outer=(64,65), constraints=(/64: (/NULL - ]; /65: (/NULL - ])]
   │         │    │    │    │    │    └── filters (true)
   │         │    │    │    │    └── filters
   │         │    │    │    │         └── l2.l_suppkey != l1.l_suppkey [outer=(10,39), constraints=(/10: (/NULL - ]; /39: (/NULL - ])]
   │         │    │    │    └── filters (true)
   │         │    │    └── filters
   │         │    │         └── n_name = 'SAUDI ARABIA' [outer=(34), constraints=(/34: [/'SAUDI ARABIA' - /'SAUDI ARABIA']; tight), fd=()-->(34)]
   │         │    └── filters
   │         │         └── o_orderstatus = 'F' [outer=(26), constraints=(/26: [/'F' - /'F']; tight), fd=()-->(26)]
   │         └── aggregations
   │              └── count-rows
   └── const: 100
(121 rows)

Time: 14.15ms

jordanlewis · 2019-09-10T19:52:29Z

I tried to investigate this but I got kind of lost figuring out where in the chain the optimizer got confused. It seems like the execbuilder is faithfully doing what the optimizer tells it to, which results in an index that doesn't match the input columns.

So I'm going to hand this off to @RaduBerinde's team to take a look.

jordanlewis · 2019-09-10T19:59:25Z

If you comment out the PruneSemiAntiJoinCols rule, the following error is produced:

pq: internal error: invalid var index 21 (columns: 11)
DETAIL: stack trace:
github.com/cockroachdb/cockroach/pkg/sql/sem/tree/indexed_vars.go:182: checkIndex()
github.com/cockroachdb/cockroach/pkg/sql/sem/tree/indexed_vars.go:196: IndexedVar()
github.com/cockroachdb/cockroach/pkg/sql/opt_exec_factory.go:214: ConstructSimpleProject()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:1035: buildGroupByInput()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:861: buildGroupBy()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:198: buildRelational()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:1145: buildSort()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:207: buildRelational()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:1122: buildLimitOffset()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:204: buildRelational()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/statement.go:116: buildExplain()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/relational.go:261: buildRelational()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/builder.go:117: build()
github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder/builder.go:105: Build()
github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:169: makeOptimizerPlan()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:734: makeExecPlan()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:616: dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:417: execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:98: execStmt()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1238: execCmd()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167: run()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444: ServeConn()
github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584: func1()
runtime/asm_amd64.s:1337: goexit()

Thanks to @rytaft for volunteering to take this one on

40583: sqlmigrations: remove ensureMaxPrivileges migration r=mberhault a=ajwerner This migration has baked. Furthermore it uses table descriptors in ways I'm working to change and I want to see it gone. Release note: None 40669: opt: fix execbuilder output column count for lookup joins r=rytaft a=rytaft This commit fixes a bug in the `execbuilder`, which was determining the number of output columns incorrectly for lookup joins. In particular, if the join was a semi- or anti-join, the `execbuilder` was incorrectly including the right side columns in the column count. This commit fixes the code so it only includes left side columns for semi- and anti-joins. This commit also fixes a bug in the DistSQL planner that was masked by the bug in the `execbuilder`. In particular, the output types slice for lookup semi- and anti-joins incorrectly included the types from the right side columns. This commit fixes it to only include types for the left side columns. Fixes #40562 Release note (bug fix): Fixed an error that could occur when the optimizer created a plan with a lookup semi- or anti-join nested inside of another join. Co-authored-by: Andrew Werner <[email protected]> Co-authored-by: Rebecca Taft <[email protected]>

yuzefovich added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. labels Sep 6, 2019

yuzefovich assigned solongordon Sep 6, 2019

andy-kimball mentioned this issue Sep 7, 2019

19.2 release blockers list #40447

Closed

53 tasks

jordanlewis assigned RaduBerinde and unassigned solongordon Sep 10, 2019

jordanlewis assigned rytaft and unassigned RaduBerinde Sep 10, 2019

rytaft mentioned this issue Sep 11, 2019

opt: fix execbuilder output column count for lookup joins #40669

Merged

craig bot closed this as completed in 63b31df Sep 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql: a crash with a lookup join #40562

sql: a crash with a lookup join #40562

yuzefovich commented Sep 6, 2019

jordanlewis commented Sep 6, 2019

yuzefovich commented Sep 6, 2019

yuzefovich commented Sep 6, 2019

jordanlewis commented Sep 6, 2019

yuzefovich commented Sep 6, 2019

jordanlewis commented Sep 7, 2019

jordanlewis commented Sep 10, 2019

jordanlewis commented Sep 10, 2019

jordanlewis commented Sep 10, 2019

sql: a crash with a lookup join #40562

sql: a crash with a lookup join #40562

Comments

yuzefovich commented Sep 6, 2019

jordanlewis commented Sep 6, 2019

yuzefovich commented Sep 6, 2019

yuzefovich commented Sep 6, 2019

jordanlewis commented Sep 6, 2019

yuzefovich commented Sep 6, 2019

jordanlewis commented Sep 7, 2019

jordanlewis commented Sep 10, 2019

jordanlewis commented Sep 10, 2019

jordanlewis commented Sep 10, 2019