Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

util/paging: choose min paging size default value as 128, and max value as 8192 #36331

Merged
merged 6 commits into from
Jul 19, 2022
Merged

util/paging: choose min paging size default value as 128, and max value as 8192 #36331

merged 6 commits into from
Jul 19, 2022

Conversation

tiancaiamao
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #36328

Problem Summary:

What is changed and how it works?

When min paging size is too small, we need more network round-trip.
But when it's too large, the packet size grow and consume more memory.

There are two variable, min paging size and max paging size to control behaviour.
After expirement, I'd like to choose the default min paging size to 128.

In the following picture, I expirement with:

  • disable paging
  • enable paging, min paging size = 16
  • min paging size = 32
  • min paging size = 64 (the current default)
  • min paging size = 128
  • min paging size = 256
  • min paging size = 512

image

The DistSQL QPS panel reflect the total qps
The Coprocessor Query Per Second panel reflect the actual paging requests, one distsql query may result in several, they are in 1:N relationship

Coprocessor query for two reason:

  1. one distsql query visit several regions, so it's splited per region
  2. paging result too large, so it involves multiple network round-trip

For 2, we want to choose the best defalut value for tidb_min_paging_size to reduce network round-trip to get better performance.
From the Coprocessor Query Per Second panel, it's obvious that size = 16 is too small,
128, 256, 512 are basicaly the same...

min paging size qps
off 16944.88
16 16616.43
32 18496.36
64 17970.63
128 19771.29
256 19649.98
512 19608.02

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 19, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • XuHuaiyu
  • wshwsh12

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-triage-completed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 19, 2022
@tiancaiamao
Copy link
Contributor Author 

paging = false
SQL statistics:
    queries performed:
        read:                            5337920
        write:                           0
        other:                           762560
        total:                           6100480
    transactions:                        381280 (1059.05 per sec.)
    queries:                             6100480 (16944.88 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0185s
    total number of events:              381280

Latency (ms):
         min:                                    7.02
         avg:                                   15.10
         max:                                   52.02
         95th percentile:                       21.50
         sum:                              5758691.76

Threads fairness:
    events (avg/stddev):           23830.0000/24.94
    execution time (avg/stddev):   359.9182/0.00



16
SQL statistics:
    queries performed:
        read:                            5234334
        write:                           0
        other:                           747762
        total:                           5982096
    transactions:                        373881 (1038.53 per sec.)
    queries:                             5982096 (16616.43 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0102s
    total number of events:              373881

Latency (ms):
         min:                                    9.37
         avg:                                   15.40
         max:                                   52.22
         95th percentile:                       21.89
         sum:                              5758569.05

Threads fairness:
    events (avg/stddev):           23367.5625/26.36
    execution time (avg/stddev):   359.9106/0.00


32
SQL statistics:
    queries performed:
        read:                            5826548
        write:                           0
        other:                           832364
        total:                           6658912
    transactions:                        416182 (1156.02 per sec.)
    queries:                             6658912 (18496.36 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0114s
    total number of events:              416182

Latency (ms):
         min:                                    8.34
         avg:                                   13.84
         max:                                   77.02
         95th percentile:                       17.63
         sum:                              5758468.82

Threads fairness:
    events (avg/stddev):           26011.3750/31.81
    execution time (avg/stddev):   359.9043/0.00

64
SQL statistics:
    queries performed:
        read:                            5660956
        write:                           0
        other:                           808708
        total:                           6469664
    transactions:                        404354 (1123.16 per sec.)
    queries:                             6469664 (17970.63 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0125s
    total number of events:              404354

Latency (ms):
         min:                                    8.32
         avg:                                   14.24
         max:                                   68.64
         95th percentile:                       19.65
         sum:                              5758484.25

Threads fairness:
    events (avg/stddev):           25272.1250/19.71
    execution time (avg/stddev):   359.9053/0.00

128
SQL statistics:
    queries performed:
        read:                            6228152
        write:                           0
        other:                           889736
        total:                           7117888
    transactions:                        444868 (1235.71 per sec.)
    queries:                             7117888 (19771.29 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0107s
    total number of events:              444868

Latency (ms):
         min:                                    7.42
         avg:                                   12.94
         max:                                   52.55
         95th percentile:                       16.71
         sum:                              5758347.01

Threads fairness:
    events (avg/stddev):           27804.2500/30.62
    execution time (avg/stddev):   359.8967/0.00


256
SQL statistics:
    queries performed:
        read:                            6190002
        write:                           0
        other:                           884286
        total:                           7074288
    transactions:                        442143 (1228.12 per sec.)
    queries:                             7074288 (19649.98 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0138s
    total number of events:              442143

Latency (ms):
         min:                                    7.58
         avg:                                   13.02
         max:                                   57.51
         95th percentile:                       17.32
         sum:                              5758387.66

Threads fairness:
    events (avg/stddev):           27633.9375/31.00
    execution time (avg/stddev):   359.8992/0.00


512
SQL statistics:
    queries performed:
        read:                            6176744
        write:                           0
        other:                           882392
        total:                           7059136
    transactions:                        441196 (1225.50 per sec.)
    queries:                             7059136 (19608.02 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          360.0120s
    total number of events:              441196

Latency (ms):
         min:                                    7.58
         avg:                                   13.05
         max:                                   66.60
         95th percentile:                       17.32
         sum:                              5758364.57

Threads fairness:
    events (avg/stddev):           27574.7500/26.30
    execution time (avg/stddev):   359.8978/0.00

@tiancaiamao tiancaiamao marked this pull request as ready for review July 19, 2022 10:11
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 19, 2022
@tiancaiamao
Copy link
Contributor Author

B.T.W, copr-cache hit rate may account for that paging performance become better than non-paging, but it's just my speculation.

@tiancaiamao
Copy link
Contributor Author

For max paging size = 8192, we've tested that setting will not bring obvious performance regression comparing to non-paging, in some other AP testing scenarios not listed here.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 19, 2022
@tiancaiamao tiancaiamao mentioned this pull request Jul 19, 2022
Closed
24 tasks
@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed do-not-merge/needs-triage-completed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 19, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Jul 19, 2022 •
edited
Loading

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/8c106a4de20aed92e2175df82246b70ccacb3522

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 19, 2022
@tiancaiamao
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b06a101

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 19, 2022
@hawkingrei
Copy link
Member

/run-check_dev_2

@tiancaiamao
Copy link
Contributor Author

/run-mysql-test

1 similar comment
@hawkingrei
Copy link
Member

/run-mysql-test

@tiancaiamao
Copy link
Contributor Author

The mysql-test case is fixed in https://github.com/pingcap/tidb-test/pull/1892/files
But this integration test seems to used the old one? @hawkingrei

@tiancaiamao
Copy link
Contributor Author

/run-mysql-test

@tiancaiamao
Copy link
Contributor Author

/run-mysql-test tidb-test=pr/1892

@ti-chi-bot ti-chi-bot merged commit 0cecfc8 into pingcap:master Jul 19, 2022
@tiancaiamao tiancaiamao deleted the default-paging-size branch July 19, 2022 14:23
@sre-bot
Copy link
Contributor

sre-bot commented Jul 19, 2022

TiDB MergeCI notify

🔴 Bad News! [2] CI still failing after this pr merged.
These failed integration tests don't seem to be introduced by the current PR.

CI Name Result Duration Compare with Parent commit
idc-jenkins-ci-tidb/integration-common-test 🔴 failed 2, success 9, total 11 20 min Existing failure
idc-jenkins-ci-tidb/common-test 🔴 failed 2, success 10, total 12 11 min Existing failure
idc-jenkins-ci/integration-cdc-test 🟢 all 36 tests passed 24 min Existing passed
idc-jenkins-ci-tidb/integration-ddl-test 🟢 all 6 tests passed 5 min 23 sec Existing passed
idc-jenkins-ci-tidb/tics-test 🟢 all 1 tests passed 4 min 52 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2 🟢 all 28 tests passed 4 min 33 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1 🟢 all 26 tests passed 4 min 23 sec Existing passed
idc-jenkins-ci-tidb/mybatis-test 🟢 all 1 tests passed 3 min 3 sec Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test 🟢 all 1 tests passed 2 min 51 sec Existing passed
idc-jenkins-ci-tidb/plugin-test 🟢 build success, plugin test success 4min Existing passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wshwsh12 wshwsh12 wshwsh12 approved these changes

@XuHuaiyu XuHuaiyu XuHuaiyu approved these changes

@you06 you06 Awaiting requested review from you06

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Expirement to choose the best value for page size (or self-adapting)
6 participants
@tiancaiamao @ti-chi-bot @sre-bot @hawkingrei @XuHuaiyu @wshwsh12