statistics: enables global-level stats to be generated in fast analyze mode #22931

Reminiscent · 2021-02-25T02:57:12Z

What problem does this PR solve?

sub PR for #22554

Problem Summary:

Generate the global-level stats when we analyze the partition table in fast analyze mode.
Fix an issue for globalStats.Count.

What is changed and how it works?

Same as described above.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

No release note

Reminiscent · 2021-02-25T09:52:20Z

/run-all-tests

2.make fast analyze work in dynamic mode

qw4990 · 2021-02-26T03:38:10Z

executor/analyze.go

@@ -171,6 +171,12 @@ func (e *AnalyzeExec) Next(ctx context.Context, req *chunk.Chunk) error {
 			sc := e.ctx.GetSessionVars().StmtCtx
 			globalStats, err := statsHandle.MergePartitionStats2GlobalStats(sc, infoschema.GetInfoSchema(e.ctx), globalStatsID.tableID, info.isIndex, info.idxID)
 			if err != nil {
+				errMessage := err.Error()
+				if errMessage == "[stats] build global-level stats failed due to missing partition-level stats" {


Judging an error by comparing its error message is dangerous.
You'd better create a new kind of error and use errors.Cause(err).
Here is an example:

if _, ok := errors.Cause(err).(*tikv.PDError); !ok { break }

rebelice · 2021-02-26T04:09:46Z

executor/analyze.go

@@ -809,8 +815,7 @@ func (e *AnalyzeFastExec) calculateEstimateSampleStep() (err error) {
 		sql := new(strings.Builder)
 		sqlexec.MustFormatSQL(sql, "select count(*) from %n.%n", dbInfo.Name.L, e.tblInfo.Name.L)

-		pruneMode := variable.PartitionPruneMode(e.ctx.GetSessionVars().PartitionPruneMode.Load())
-		if pruneMode != variable.DynamicOnly && e.tblInfo.ID != e.tableID.GetStatisticsID() {
+		if e.tblInfo.ID != e.tableID.GetStatisticsID() {


Why don't we check PartitionPruneMode

The AnalyzeFastExec structure is based on partition. The content recorded in it corresponds to the content of the partition, not the whole table.

ti-chi-bot · 2021-02-26T06:14:50Z

@qw4990: Please use /LGTM instead of LGTM when you want to approve the pull request by comment.
If you use the GitHub review feature, please approve the PR directly, the comment will not take effect in the GitHub review feature.
If you have any qustions please refer to lgtm command help or lgtm plugin design.

If you have approved this PR, please ignore this reply. This reply is being used as a temporary reply during the migration of the new bot and will be removed on April 1.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

qw4990

LGTM

qw4990

/LGTM

Reminiscent · 2021-03-01T06:07:58Z

/merge

ti-chi-bot · 2021-03-01T06:07:59Z

@Reminiscent: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-03-01T06:08:01Z

@Reminiscent: /merge in this pull request requires 2 /lgtm.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…lyze � Conflicts: � executor/analyze.go � executor/analyze_test.go � planner/core/planbuilder.go

…lyze

Reminiscent · 2021-03-01T08:07:46Z

/merge

ti-chi-bot · 2021-03-01T08:07:47Z

@Reminiscent: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-03-01T08:07:50Z

@Reminiscent: /merge in this pull request requires 2 /lgtm.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Reminiscent · 2021-03-01T08:13:03Z

/run-tics-test

Reminiscent · 2021-03-01T08:13:51Z

/run-integration-common-test

Reminiscent · 2021-03-01T08:22:44Z

/merge

ti-chi-bot · 2021-03-01T08:22:45Z

@Reminiscent: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-03-01T08:22:46Z

@Reminiscent: /merge in this pull request requires 2 /lgtm.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

qw4990 · 2021-03-01T08:24:46Z

/merge

ti-chi-bot · 2021-03-01T08:24:47Z

@qw4990: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-03-01T08:24:48Z

@qw4990: /merge in this pull request requires 2 /lgtm.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

qw4990

/LGTM

Reminiscent requested review from a team as code owners February 25, 2021 02:57

Reminiscent requested review from lzmhhh123 and removed request for a team February 25, 2021 02:57

ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 25, 2021

github-actions bot added component/statistics sig/execution SIG execution labels Feb 25, 2021

Reminiscent closed this Feb 25, 2021

Reminiscent force-pushed the fastAnalyze branch from 2919ae4 to 541efe8 Compare February 25, 2021 08:12

ti-chi-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 25, 2021

Reminiscent reopened this Feb 25, 2021

ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 25, 2021

Reminiscent added 5 commits February 25, 2021 19:03

1.fix the wrong result for the stats.Count

089124e

2.make fast analyze work in dynamic mode

fix ut

adc8c9a

fix ut

a0fce3a

add some comments

be6f8f9

update some comments

8e42276

Reminiscent force-pushed the fastAnalyze branch from 1395565 to 8e42276 Compare February 25, 2021 11:04

qw4990 reviewed Feb 26, 2021

View reviewed changes

rebelice reviewed Feb 26, 2021

View reviewed changes

address comments

50d7396

qw4990 reviewed Feb 26, 2021

View reviewed changes

fix ut

b26ada9

github-actions bot added the component/expression label Feb 26, 2021

rebelice approved these changes Feb 26, 2021

View reviewed changes

Reminiscent added 2 commits February 26, 2021 15:07

Merge branch 'master' into fastAnalyze

6c21108

Merge branch 'master' into fastAnalyze

6d62124

Reminiscent and others added 10 commits March 1, 2021 14:28

fix ut

fb1c048

Merge branch 'master' of https://github.com/pingcap/tidb into fastAna…

6d4ce46

…lyze � Conflicts: � executor/analyze.go � executor/analyze_test.go � planner/core/planbuilder.go

Merge remote-tracking branch 'origin/fastAnalyze' into fastAnalyze

9517465

revert the change for solve conflicts

93218c1

fix ut

0b720c9

fix ut

70e5bee

fix ut

66d62ef

fix ut

352fa6e

Merge branch 'master' of https://github.com/pingcap/tidb into fastAna…

b7a2501

…lyze

Merge branch 'master' into fastAnalyze

f7de949

Merge branch 'master' into fastAnalyze

5f31eba

qw4990 approved these changes Mar 1, 2021

View reviewed changes

qw4990 merged commit 550ca8e into pingcap:master Mar 1, 2021

Reminiscent deleted the fastAnalyze branch August 5, 2021 01:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: enables global-level stats to be generated in fast analyze mode #22931

statistics: enables global-level stats to be generated in fast analyze mode #22931

Reminiscent commented Feb 25, 2021 •

edited

Loading

Reminiscent commented Feb 25, 2021

qw4990 Feb 26, 2021 •

edited

Loading

rebelice Feb 26, 2021

Reminiscent Feb 26, 2021

ti-chi-bot commented Feb 26, 2021

qw4990 left a comment

qw4990 left a comment

Reminiscent commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

qw4990 commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

qw4990 left a comment

statistics: enables global-level stats to be generated in fast analyze mode #22931

statistics: enables global-level stats to be generated in fast analyze mode #22931

Conversation

Reminiscent commented Feb 25, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

Reminiscent commented Feb 25, 2021

qw4990 Feb 26, 2021 • edited Loading

Choose a reason for hiding this comment

rebelice Feb 26, 2021

Choose a reason for hiding this comment

Reminiscent Feb 26, 2021

Choose a reason for hiding this comment

ti-chi-bot commented Feb 26, 2021

qw4990 left a comment

Choose a reason for hiding this comment

qw4990 left a comment

Choose a reason for hiding this comment

Reminiscent commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

Reminiscent commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

qw4990 commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

ti-chi-bot commented Mar 1, 2021

qw4990 left a comment

Choose a reason for hiding this comment

Reminiscent commented Feb 25, 2021 •

edited

Loading

qw4990 Feb 26, 2021 •

edited

Loading