planner: fix the inappropriate heuristic rule to estimate the EQ selectivity when out of range

[qw4990]

What problem does this PR solve?

Issue Number: close #18461

Problem Summary: If the estimated value is out of range, an inappropriate heuristic rule sel = 1/NDV*(modifyRows/totalRows) is used, which may cause unexpected low sel when a few rows are modified.

What is changed and how it works?

Change this rule to:

func outOfRangeEQSelectivity(ndv, modifyRows, totalRows int64) float64 {
	if modifyRows == 0 {
		return 0 // it must be 0 since the histogram contains the whole data
	}
	if ndv < outOfRangeBetweenRate {
		ndv = outOfRangeBetweenRate // avoid inaccurate selectivity caused by small NDV
	}
	selectivity := 1 / float64(ndv) // TODO: After extracting TopN from histograms, we can minus the TopN fraction here.
	if selectivity*float64(totalRows) > float64(modifyRows) {
		selectivity = float64(modifyRows) / float64(totalRows)
	}
	return selectivity
}

Check List

Tests

• Unit test

Release note

• planner: fix the inappropriate heuristic rule to estimate the EQ selectivity when out of range

[sre-bot]

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

[sre-bot]

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

[codecov]

Codecov Report

Merging #18543 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #18543   +/-   ##
===========================================
  Coverage   79.4399%   79.4399%           
===========================================
  Files           546        546           
  Lines        148098     148098           
===========================================
  Hits         117649     117649           
  Misses        20971      20971           
  Partials       9478       9478           

[qw4990]

PTAL @eurekaka @winoros

[eurekaka]

I don't understand the rationale behind this new formula. For the old formula, selectivity = 1 / NDV * (ModifyRows / TotalRows), it assumes that the un-analyzed data rows are uniformly distributed and has similar distribution with analyzed data rows, this assumption sounds reasonable for me, while the new formula 1 / ndv indicates that the analyzed data rows contain the target value as well, which is contradictive with the truth that the target value is out of range / not in CMSketch.

[qw4990]

I don't understand the rationale behind this new formula. For the old formula, selectivity = 1 / NDV * (ModifyRows / TotalRows), it assumes that the un-analyzed data rows are uniformly distributed and has similar distribution with analyzed data rows, this assumption sounds reasonable for me, while the new formula 1 / ndv indicates that the analyzed data rows contain the target value as well, which is contradictive with the truth that the target value is out of range / not in CMSketch.

Actually which formula is more rational depends on what specific case it is.

If all ModifyRows rows are new-inserted and all of them are out-of-range, the formula 1/NDV is the correct formula of your assumption the un-analyzed data rows are uniformly distributed and has similar distribution with analyzed data rows.
Because for every new-inserted unique value, there should be Tot/NDV rows with the same value on it.

Since I can't find any specific case that makes 1/NDV * (Modi/Tot) be more rational than 1/NDV, and according to other DB's strategies(#18461), can we think 1/NDV is more robust than 1/NDV * (Modi/Tot)?

[eurekaka]

LGTM

[winoros]

lgtm

[qw4990]

/run-all-tests

[ti-srebot]

cherry pick to release-3.0 in PR #18994

[ti-srebot]

cherry pick to release-3.1 in PR #18995

[ti-srebot]

cherry pick to release-4.0 in PR #18997