statistics: fix init topn not sorted

[time-and-fate]

What problem does this PR solve?

Issue Number: close #34216 ref #32758

Problem Summary: Please see the issue.

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (please see the comments below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• qw4990
• winoros

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/94c4d91a76397f5efffe41ac8543a22f8b6cdb2a

[time-and-fate]

The behavior of InitStats() is different in unit tests and in real TiDB. In unit tests, the batch size for SQL execution is 1, which makes the bug not reproducible in unit tests. And the internal structure of stats is not exposed to the outside. This bug needs tidb to be restarted or join a new cluster to trigger. So it's also not convenient to add integration tests. So I did 2 manual tests:

1. I used the debugger to verify that the TopN is sorted after the PR.
2. I used an internal tool to test the p-error[1] distribution of cardinality estimation of JOB workload + IMDB dataset (that's also how we found this bug). The test result shows that cardinality estimation is improved:

[1]: We use the p-error definition from "Preventing bad plans by bounding the impact of cardinality estimation errors."

	Label	Count	P50	P90	P95	P99	Max
Overall	before	2686	126.000	1068187.000	2625728.000	39921236.000	1538186904.000
	after	2659	4.000	1104.685	4765.634	290965.000	7313673.000
Exact estimation	before	226	-	-	-	-	-
	after	216	-	-	-	-	-
Overestimation	before	721	29.430	6083.111	62117.131	733610.889	1275947.000
	after	1491	17.099	3614.055	28894.973	962682.273	7313673.000
Underestimation	before	1739	2736.000	1934074.000	6158732.000	47732163.000	1538186904.000
	after	952	1.591	56.523	237.000	2678.568	44896.225

Note that there are some bad overestimations cases after this PR. They are caused by:

1. The bug makes some estimations become 0 wrongly.
2. Actual row counts of some join results happen to be 0.
3. Then the estimations for those cases become worse after this change.

But I think it's OK to accept that and fix this bug, because:

1. It's just wrong, it just coincides with the correct row count.
2. It's not reliable, it only happens when a tidb is restarted or a new tidb joins the cluster. After an ANALYZE, the estimation becomes as expected again.
3. For most cases, the estimation becomes better.

[qw4990]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 6f70fc8

[sre-bot]

TiDB MergeCI notify

✅ Well Done! New fixed [1] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/common-test	🔴 failed 1, success 11, total 12	6 min 8 sec	Existing failure
idc-jenkins-ci-tidb/integration-ddl-test	✅ all 6 tests passed	7 min 5 sec	Fixed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 11 tests passed	10 min	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	5 min 29 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	4 min 58 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	4 min 48 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	4 min 18 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 31 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

[winoros]

/run-cherry-picker release-5.4
/run-cherry-picker release-6.0

[ti-srebot]

cherry pick to release-6.0 in PR #34316

[ti-srebot]

cherry pick to release-5.4 in PR #34317