Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

staging-v23.1.20: opt/memo: improve zigzag join cost and selectivity estimation with multi-column stats #123152

Merged

Conversation

rytaft
Copy link
Collaborator

@rytaft rytaft commented Apr 26, 2024

Backport 4/4 commits from #120805.

/cc @cockroachdb/release


opt: update seek and distribution cost of zigzag join to match scan

Prior to this commit, the optimizer could prefer a zigzag join over a
scan even if they produced the same number of rows. This was because scans
always included the cost of at least one seek (involving random I/O) and
some distribution cost, while zigzag joins did not. This commit updates
the cost of zigzag joins to include seek and distribution costs so they
will never be chosen over scans unless they produce fewer rows.

This change is behind the setting optimizer_use_improved_zigzag_join_costing.

Release note (performance improvement): Added a new setting
optimizer_use_improved_zigzag_join_costing. When enabled, the cost of zigzag
joins is updated so they will be never be chosen over scans unless they
produce fewer rows. This change only matters if the setting enable_zigzag_join
is also true.

opt/memo: improve selectivity estimation with multi-column stats

This commit updates correlationFromMultiColDistinctCounts in statisticsBuilder
to use a tighter lower bound for the multi-column selectivity. This avoids
cases where we significantly over-estimate the selectivity of a multi-column
predicate.

Fixes #121397

Release note (performance improvement): Improved the selectivity estimation of
multi-column filters when the multi-column distinct count is high. This avoids
cases where we significantly over-estimate the selectivity of a multi-column
predicate and as a result can prevent the optimizer from choosing a bad query
plan.

sql: add setting optimizer_use_improved_multi_column_selectivity_estimate

Informs #121397

Release note (sql change): Added a setting
optimizer_use_improved_multi_column_selectivity_estimate, which if enabled,
causes the optimizer to use an improved selectivity estimate for multi-column
predicates. This setting will default to true on versions 24.2+, and false
on prior versions.

opt: improve variable names in selectivityFromMultiColDistinctCounts

This commit improves the variable names in
selectivityFromMultiColDistinctCounts in statisticsBuilder to be more
self-documenting.

Release note: None


Release justification: low-risk, high benefit change to existing functionality to unblock a customer

rytaft added 4 commits April 26, 2024 14:27
Prior to this commit, the optimizer could prefer a zigzag join over a
scan even if they produced the same number of rows. This was because scans
always included the cost of at least one seek (involving random I/O) and
some distribution cost, while zigzag joins did not. This commit updates
the cost of zigzag joins to include seek and distribution costs so they
will never be chosen over scans unless they produce fewer rows.

This change is behind the setting optimizer_use_improved_zigzag_join_costing.

Release note (performance improvement): Added a new setting
optimizer_use_improved_zigzag_join_costing. When enabled, the cost of zigzag
joins is updated so they will be never be chosen over scans unless they
produce fewer rows. This change only matters if the setting enable_zigzag_join
is also true.
This commit updates correlationFromMultiColDistinctCounts in statisticsBuilder
to use a tighter lower bound for the multi-column selectivity. This avoids
cases where we significantly over-estimate the selectivity of a multi-column
predicate.

Fixes cockroachdb#121397

Release note (performance improvement): Improved the selectivity estimation of
multi-column filters when the multi-column distinct count is high. This avoids
cases where we significantly over-estimate the selectivity of a multi-column
predicate and as a result can prevent the optimizer from choosing a bad query
plan.
…mate

Informs cockroachdb#121397

Release note (sql change): Added a setting
optimizer_use_improved_multi_column_selectivity_estimate, which if enabled,
causes the optimizer to use an improved selectivity estimate for multi-column
predicates. This setting will default to true on versions 24.2+, and false
on prior versions.
This commit improves the variable names in
selectivityFromMultiColDistinctCounts in statisticsBuilder to be more
self-documenting.

Release note: None
@rytaft rytaft requested review from a team as code owners April 26, 2024 19:32
@rytaft rytaft requested review from dt and michae2 and removed request for a team April 26, 2024 19:32
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@rytaft rytaft requested a review from mgartner April 26, 2024 19:38
Copy link
Collaborator

@michae2 michae2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @dt and @mgartner)

@rytaft rytaft merged commit 057f6d8 into cockroachdb:staging-v23.1.20 Apr 26, 2024
6 checks passed
@rytaft rytaft deleted the backportstaging-v23.1.20-123083 branch April 26, 2024 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants