opt: use partial indexes for lookup joins

[mgartner]

Lookup joins are now generated when the ON filters of a join or the
right-side Select filters imply the partial index predicate.

In several cases, the filters are reduced once implication is proven.
This currently prevents optimal plans from being generated in several
cases, either because unnecessary joins remaining in the expression
tree, or because lookup semi- and anti-joins that should be created are
not. I've left TODOs to denote these cases.

Fixes #50227

Release note (performance improvement): The optimizer can now use
partial indexes for lookup join operations. This results in potentially
more efficient query plans for joins on tables with partial indexes.

[cockroach-teamcity]

This change is 

[rytaft]

Reviewed 4 of 4 files at r1.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @mgartner and @RaduBerinde)

---

pkg/sql/logictest/testdata/logic_test/partial_index, line 1005 at r1 (raw file):

CREATE TABLE join_small (m INT, n INT);
CREATE TABLE join_large (i INT, s STRING, INDEX (i) WHERE s IN ('foo', 'bar', 'baz'));
ALTER TABLE join_small INJECT STATISTICS '[

in case we ever decide to re-enable automatic stats for the logic tests... I'd add SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false here. You could also just force the lookup join using a LOOKUP JOIN hint.

---

pkg/sql/opt/xform/testdata/rules/join, line 3030 at r1 (raw file):

# --------------------------------------------------
# GenerateLookupJoinsWithFilter + Partial Indexes

how about adding at least one test where the filters don't imply the index predicate?

[mgartner]

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @RaduBerinde and @rytaft)

---

pkg/sql/logictest/testdata/logic_test/partial_index, line 1005 at r1 (raw file):

Previously, rytaft (Rebecca Taft) wrote…

in case we ever decide to re-enable automatic stats for the logic tests... I'd add SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false here. You could also just force the lookup join using a LOOKUP JOIN hint.

I've disabled automatic stats collection because for the last two test cases (which I removed but will re-add when they perform lookup joins) they cannot use join hints AFAIK.

---

pkg/sql/opt/xform/testdata/rules/join, line 3030 at r1 (raw file):

Previously, rytaft (Rebecca Taft) wrote…

how about adding at least one test where the filters don't imply the index predicate?

good idea. done.

[rytaft]

Reviewed 2 of 2 files at r2.
Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @RaduBerinde)

[RaduBerinde]

Reviewable status: complete! 1 of 0 LGTMs obtained (and 1 stale) (waiting on @RaduBerinde)

[mgartner]

TFTRs!

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)