Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: table partition dynamic prune mode with Physical Table ID from unistore #31634

Merged
merged 39 commits into from
Mar 3, 2022
Merged

*: table partition dynamic prune mode with Physical Table ID from unistore #31634

merged 39 commits into from
Mar 3, 2022

Conversation

mjonss
Copy link
Contributor

@mjonss mjonss commented Jan 12, 2022

What problem does this PR solve?

Issue Number: close #29851

Problem Summary:

What is changed and how it works?

The current issue with dynamic prune mode is that it uses a single TableReader which loses context which partition (physical table id) the different rows comes from, which is needed for SelectLock (SELECT FOR UPDATE) and comparison with the Transaction buffer.

By adding a virtual column with ExtraPhysTblID the engine can extract the physical table ID from the KV key that is read and append it to the row/chunk data.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility (Needs support from store, but does not check its capabilities)

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fixes reads with transactions for table partition dynamic pruning mode.

mjonss added 3 commits January 12, 2022 08:06
@mjonss
WIP add Physical Table ID as column in result.
fc62492 
Table Partition Dynamic Prune Mode uses a single TableReader
meaning it is not currently possible to know which partition
a row/record comes from.

By adding ExtraPhysTblID column this can be used when the row
needs the Physical Table ID (partition id) for SelectLock or
checking current transaction buffer (if the row has been changed/deleted).
@mjonss
Added support for more cases, like Delete
4821fae
@mjonss
Cleanup and added test from pr/31025
21dd902
@mjonss mjonss requested a review from a team as a code owner January 12, 2022 23:33
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jan 12, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • qw4990
  • tiancaiamao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 12, 2022
@mjonss
Copy link
Contributor Author

mjonss commented Jan 13, 2022

/run-unit-test

@sre-bot
Copy link
Contributor

sre-bot commented Jan 17, 2022 •
edited
Loading

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/0700f749f3e08dde22e133232360159201a1f1d9

tiancaiamao
tiancaiamao reviewed Jan 19, 2022
View reviewed changes
Copy link
Contributor

@tiancaiamao tiancaiamao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I like the idea behind this PR.
It will not change the kvproto, and the modification of TiKV is not too much.

executor/executor.go Outdated Show resolved Hide resolved
executor/index_merge_reader_test.go Show resolved Hide resolved
executor/table_reader.go Outdated Show resolved Hide resolved
executor/union_scan.go Outdated Show resolved Hide resolved
store/mockstore/unistore/cophandler/closure_exec.go Show resolved Hide resolved
@mjonss mjonss marked this pull request as draft January 21, 2022 11:31
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2022
mjonss added 3 commits January 22, 2022 20:17
@mjonss
wip ExtraPhysTblIDCol
9d9eb52
@mjonss
wip ExtraPhysTblIDCol
d6e062f 
TODO: Check if UseDynamicPruneMode() and only then add ExtraPhysTblIDCol
in logic plan

In UnionScan/SelectLockExec
if partition and not ExtraPhysTblIDCol then use GetPhysicalId
@mjonss
WIP fix SelectLock and UnionScan for both dynamic and static prune
d5d7996
@tiancaiamao tiancaiamao mentioned this pull request Jan 24, 2022
Closed
12 tasks
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 25, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2022
@mjonss
Copy link
Contributor Author

mjonss commented Feb 3, 2022

This PR also added the tests so we can close #28073 and also close #30382

tiancaiamao
tiancaiamao reviewed Feb 21, 2022
View reviewed changes
executor/executor.go Outdated Show resolved Hide resolved
executor/executor.go Show resolved Hide resolved
executor/union_scan.go Show resolved Hide resolved
executor/union_scan.go Outdated Show resolved Hide resolved
planner/core/logical_plan_builder.go Outdated Show resolved Hide resolved
planner/core/planbuilder.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 23, 2022
@mjonss mjonss mentioned this pull request Feb 23, 2022
Closed
12 tasks
@mjonss
Copy link
Contributor Author

mjonss commented Feb 28, 2022

@qw4990 Please take a look, the TiKV PR has just been merged.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 3, 2022
@qw4990
Copy link
Contributor

qw4990 commented Mar 3, 2022

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 718067c

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 3, 2022
@mjonss
Copy link
Contributor Author

mjonss commented Mar 3, 2022

/run-mysql-test pr=1601

@mjonss
Copy link
Contributor Author

mjonss commented Mar 3, 2022

/run-unit-test

@mjonss
Copy link
Contributor Author

mjonss commented Mar 3, 2022

/run-mysql-test tidb-test=pr/1601

@ti-chi-bot ti-chi-bot merged commit 493eb45 into pingcap:master Mar 3, 2022
This was referenced Mar 3, 2022
Closed
Closed
Closed
@windtalker windtalker mentioned this pull request Mar 4, 2022
Closed
@mjonss mjonss mentioned this pull request Mar 7, 2022
Closed
@chrysan chrysan mentioned this pull request Jun 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guo-shaoge guo-shaoge guo-shaoge left review comments

@qw4990 qw4990 qw4990 approved these changes

@tiancaiamao tiancaiamao tiancaiamao approved these changes

Assignees
No one assigned
Labels
release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Allow dynamic pruning mode also for transactions
6 participants
@mjonss @ti-chi-bot @sre-bot @qw4990 @tiancaiamao @guo-shaoge