Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DNM] Use Bitmap Filter to do MVCC in Normal Scan #6322

Closed
wants to merge 30 commits into from
Closed

[DNM] Use Bitmap Filter to do MVCC in Normal Scan #6322

wants to merge 30 commits into from

Conversation

JinheLin
Copy link
Contributor

@JinheLin JinheLin commented Nov 16, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #6296

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 16, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Lloyd-Pottiger

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/needs-linked-issue labels Nov 16, 2022
@JinheLin JinheLin changed the title [DNM] [DNM] Use Bitmap Filter to do MVCC in normal scan Nov 17, 2022
@JinheLin JinheLin changed the title [DNM] Use Bitmap Filter to do MVCC in normal scan [DNM] Use Bitmap Filter to do MVCC in Normal Scan Nov 17, 2022
@JinheLin JinheLin force-pushed the bitmap-filter-normal branch from d48b688 to 9d177c1 Compare November 18, 2022 02:26
@JinheLin JinheLin force-pushed the bitmap-filter-normal branch from 1d94f16 to 1df5dc5 Compare November 18, 2022 07:11
@Lloyd-Pottiger
Copy link
Contributor

/rebuild

@JinheLin JinheLin force-pushed the bitmap-filter-normal branch 5 times, most recently from bef1b32 to 2bb52a9 Compare November 24, 2022 12:28
Lloyd-Pottiger
Lloyd-Pottiger approved these changes Nov 25, 2022
View reviewed changes
dbms/src/Storages/DeltaMerge/Segment.cpp
Comment on lines +2237 to +2366
auto bitmap_filter = buildBitmapFilter(
dm_context,
segment_snap,
data_ranges,
filter,
max_version,
expected_block_size);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to build bitmap filter each query? Maybe just one bitmap for one segment snapshot?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

accidentally approved……

Copy link
Contributor Author

@JinheLin JinheLin Nov 25, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. If the snapshots of two queries are identical, they can share merge sort, but the changes may be large, I think it can be a further improvment.
  2. Since the timestamps of each query are different, it is difficult to merge the results of MVCC filtering.
  3. Another simple approach is to cache the bitmap so that the segment that is not written does not need to recompute the bitmap. It will be adopted in this PR soon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although the segment is not written, the filter and rowkey range may change, and it is still troublesome to use the cached bitmap.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 25, 2022
@Lloyd-Pottiger Lloyd-Pottiger self-requested a review November 25, 2022 01:39
@Lloyd-Pottiger Lloyd-Pottiger removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 25, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 25, 2022
@Lloyd-Pottiger
Copy link
Contributor

/status LGT1 cancel

@ti-chi-bot
Copy link
Member

@Lloyd-Pottiger: The label(s) status/cancel cannot be applied, because the repository doesn't have them.

In response to this:

/status LGT1 cancel

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@Lloyd-Pottiger
Copy link
Contributor

/remove-approve

@JinheLin JinheLin force-pushed the bitmap-filter-normal branch 2 times, most recently from 08e51d6 to 821db1d Compare November 28, 2022 02:42
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2022
@JinheLin JinheLin force-pushed the bitmap-filter-normal branch from 821db1d to 900712f Compare November 29, 2022 03:01
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 29, 2022
@JinheLin JinheLin force-pushed the bitmap-filter-normal branch from fd6d136 to 2870428 Compare December 15, 2022 04:44
@JinheLin JinheLin force-pushed the bitmap-filter-normal branch from 2870428 to d9f64e7 Compare December 16, 2022 05:24
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 29, 2022
@ti-chi-bot
Copy link
Member

@JinheLin: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@JinheLin JinheLin closed this Dec 30, 2022
@JinheLin JinheLin deleted the bitmap-filter-normal branch March 28, 2023 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Lloyd-Pottiger Lloyd-Pottiger Awaiting requested review from Lloyd-Pottiger

Assignees
No one assigned
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/LGT1 Indicates that a PR has LGTM 1.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support MVCC Bitmap Filter
3 participants
@JinheLin @ti-chi-bot @Lloyd-Pottiger