statistics: support tracking histogram #36170

Yisaer · 2022-07-13T05:35:26Z

What problem does this PR solve?

Issue Number: ref #34052

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

test report

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: yisaer <[email protected]>

Signed-off-by: yisaer <[email protected]> make loaded atomic Signed-off-by: yisaer <[email protected]>

Signed-off-by: yisaer <[email protected]>

Signed-off-by: yisaer <[email protected]> add size metrics Signed-off-by: yisaer <[email protected]> Revert "add size metrics" This reverts commit 05b06a3f4f3ad98ae3f6ca389db2fc8ccc193dc4.

Signed-off-by: yisaer <[email protected]>

Signed-off-by: yisaer <[email protected]> add comment Signed-off-by: yisaer <[email protected]>

This reverts commit e25e725.

…sketch

Signed-off-by: yisaer <[email protected]>

…topn

Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> use topn Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> fix test Signed-off-by: yisaer <[email protected]> Revert "fix test" This reverts commit bdaa13a04a475df129e40ffbc70a16064df37886. Revert "fix test" This reverts commit a206543ea2ac7712d6a1f2748d1acb50ff0f426f. Revert "fix test" This reverts commit 290d516009f6f92711a9b4a73d5c90f7051413c2. Revert "fix test" This reverts commit f29737ace34c3174af4e64943640675f2aaa1178.

Signed-off-by: yisaer <[email protected]>

ti-chi-bot · 2022-07-13T05:35:27Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

chrysan
xuyifangreeneyes

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: yisaer <[email protected]>

…hist

Signed-off-by: yisaer <[email protected]>

statistics/handle/lru_cache.go

statistics/histogram.go

…hist

Signed-off-by: yisaer <[email protected]>

xuyifangreeneyes · 2022-07-27T06:43:56Z

statistics/column.go

+func (c *Column) dropHist() {
+	c.Histogram.Bounds = chunk.NewChunkWithCapacity([]*types.FieldType{types.NewFieldType(mysql.TypeBlob)}, 0)
+	c.Histogram.Buckets = make([]Bucket, 0)
+	c.Histogram.scalars = make([]scalar, 0)


Can we just set Bounds, Buckets and scalars to nil?

This may cause panic in some Histogram functions which are hard to cover, thus I tend to set it as empty slice.

xuyifangreeneyes · 2022-07-27T06:56:07Z

statistics/histogram.go

@@ -139,8 +139,11 @@ func (hg *Histogram) MemoryUsage() (sum int64) {
 	if hg == nil {
 		return
 	}
+	if len(hg.Buckets) == 0 && len(hg.scalars) == 0 && hg.Bounds.Capacity() == 0 {
+		return


hg still takes EmptyHistogramSize if it is not nil?

Right, but we will think it as 0, the difference is acceptable.

statistics/histogram.go

Signed-off-by: yisaer <[email protected]>

…into support_tracking_hist

Signed-off-by: yisaer <[email protected]>

sre-bot · 2022-07-27T08:58:52Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/5223e7c1ebcb0a099ebb3d6f0bb1da84861f5386

statistics/histogram.go

Yisaer · 2022-07-28T02:47:37Z

/merge

ti-chi-bot · 2022-07-28T02:47:40Z

This pull request has been accepted and is ready to merge.

Commit hash: 1c04420

sre-bot · 2022-07-28T03:48:34Z

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-common-test	🟥 failed 1, success 16, total 17	32 min	New failing
idc-jenkins-ci/integration-cdc-test	🟢 all 36 tests passed	29 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 11 tests passed	17 min	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	7 min 46 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	6 min 15 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 6 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	5 min 14 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	4 min 0 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 19 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

Yisaer added 23 commits June 2, 2022 16:45

support tracking column cmsketch

891b61f

Signed-off-by: yisaer <[email protected]>

fix test

12d4941

Signed-off-by: yisaer <[email protected]>

make loaded atomic

be6d2ab

Signed-off-by: yisaer <[email protected]> make loaded atomic Signed-off-by: yisaer <[email protected]>

fix

ccaf3a0

Signed-off-by: yisaer <[email protected]>

allow column init cmsketch

7cefc59

Signed-off-by: yisaer <[email protected]> add size metrics Signed-off-by: yisaer <[email protected]> Revert "add size metrics" This reverts commit 05b06a3f4f3ad98ae3f6ca389db2fc8ccc193dc4.

add testcase

f448110

Signed-off-by: yisaer <[email protected]>

remove useless code

2030048

Signed-off-by: yisaer <[email protected]>

fix test

1651163

Signed-off-by: yisaer <[email protected]>

Merge branch 'master' into support_column_cmsketch

c69e9af

Merge branch 'master' into support_column_cmsketch

baf94f8

revise loaded

e25e725

Signed-off-by: yisaer <[email protected]> add comment Signed-off-by: yisaer <[email protected]>

Revert "revise loaded"

f7b08af

This reverts commit e25e725.

Merge remote-tracking branch 'upstream/master' into support_column_cm…

8d60457

…sketch

use new load status

b5643b3

Signed-off-by: yisaer <[email protected]>

fix

806eaad

Signed-off-by: yisaer <[email protected]>

fix test

6c8c1f8

Signed-off-by: yisaer <[email protected]>

support tracking topn

0995d8c

Signed-off-by: yisaer <[email protected]>

Merge remote-tracking branch 'upstream/master' into support_tracking_…

a4bfa3e

…topn

fix conflict

45d13e9

Signed-off-by: yisaer <[email protected]>

update

2a59451

Signed-off-by: yisaer <[email protected]>

support track hist

ef1375e

Signed-off-by: yisaer <[email protected]>

fix conflict

c9f8912

Signed-off-by: yisaer <[email protected]>

ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 13, 2022

Yisaer added 3 commits July 15, 2022 10:42

fix lint

157149d

Signed-off-by: yisaer <[email protected]>

revise test

aba8ea6

Signed-off-by: yisaer <[email protected]>

revise test

34263dc

Signed-off-by: yisaer <[email protected]>

Yisaer requested review from xuyifangreeneyes and chrysan July 19, 2022 07:59

Merge remote-tracking branch 'upstream/master' into support_tracking_…

ea4d45f

…hist

Yisaer force-pushed the support_tracking_hist branch from 986e237 to ea4d45f Compare July 21, 2022 03:03

Yisaer added 2 commits July 21, 2022 11:13

revise test

c10f3ee

Signed-off-by: yisaer <[email protected]>

revise test

8376405

Signed-off-by: yisaer <[email protected]>

chrysan reviewed Jul 22, 2022

View reviewed changes

statistics/handle/lru_cache.go Show resolved Hide resolved

statistics/handle/lru_cache.go Show resolved Hide resolved

statistics/histogram.go Outdated Show resolved Hide resolved

Yisaer added 2 commits July 26, 2022 12:24

Merge remote-tracking branch 'upstream/master' into support_tracking_…

c335363

…hist

address the comment

eab795b

Signed-off-by: yisaer <[email protected]>

xuyifangreeneyes reviewed Jul 27, 2022

View reviewed changes

statistics/histogram.go Outdated Show resolved Hide resolved

Yisaer added 5 commits July 27, 2022 15:47

Merge branch 'master' into support_tracking_hist

0f3b343

fix test

89ab01f

Signed-off-by: yisaer <[email protected]>

Merge branch 'support_tracking_hist' of https://github.com/Yisaer/tidb …

05a964b

…into support_tracking_hist

address the comment

b527efa

Signed-off-by: yisaer <[email protected]>

fix test

1c04420

Signed-off-by: yisaer <[email protected]>

xuyifangreeneyes approved these changes Jul 27, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 27, 2022

chrysan reviewed Jul 27, 2022

View reviewed changes

statistics/histogram.go Show resolved Hide resolved

chrysan approved these changes Jul 28, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 28, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 28, 2022

Merge branch 'master' into support_tracking_hist

5223e7c

ti-chi-bot merged commit 212d62e into pingcap:master Jul 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: support tracking histogram #36170

statistics: support tracking histogram #36170

Yisaer commented Jul 13, 2022 •

edited

Loading

ti-chi-bot commented Jul 13, 2022 •

edited

Loading

xuyifangreeneyes Jul 27, 2022

Yisaer Jul 27, 2022 •

edited

Loading

xuyifangreeneyes Jul 27, 2022

Yisaer Jul 27, 2022

sre-bot commented Jul 27, 2022 •

edited

Loading

Yisaer commented Jul 28, 2022

ti-chi-bot commented Jul 28, 2022

sre-bot commented Jul 28, 2022

statistics: support tracking histogram #36170

statistics: support tracking histogram #36170

Conversation

Yisaer commented Jul 13, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jul 13, 2022 • edited Loading

xuyifangreeneyes Jul 27, 2022

Choose a reason for hiding this comment

Yisaer Jul 27, 2022 • edited Loading

Choose a reason for hiding this comment

xuyifangreeneyes Jul 27, 2022

Choose a reason for hiding this comment

Yisaer Jul 27, 2022

Choose a reason for hiding this comment

sre-bot commented Jul 27, 2022 • edited Loading

Yisaer commented Jul 28, 2022

ti-chi-bot commented Jul 28, 2022

sre-bot commented Jul 28, 2022

TiDB MergeCI notify

Yisaer commented Jul 13, 2022 •

edited

Loading

ti-chi-bot commented Jul 13, 2022 •

edited

Loading

Yisaer Jul 27, 2022 •

edited

Loading

sre-bot commented Jul 27, 2022 •

edited

Loading