Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: enhancement for ListInDisk(support writing after reading) #24379

Merged
merged 16 commits into from
May 19, 2021
Merged

executor: enhancement for ListInDisk(support writing after reading) #24379

merged 16 commits into from
May 19, 2021

Conversation

guo-shaoge
Copy link
Collaborator

@guo-shaoge guo-shaoge commented Apr 29, 2021 •
edited
Loading

Signed-off-by: guo-shaoge [email protected]

What problem does this PR solve?

Issue Number: close #24378

Problem Summary: This is a enhancement for ListInDisk. In some scenarios, like CTE, we need write after read.

Why we can not call write function of ListInDisk if we call read function before:

  1. ListInDisk consists of checksum layer and encrypt/aes layer, to make files in disk more serure.
  2. checksum layer and encrypt/aes layer has its own cache. Data writen to them will first write to cache, when size of data reaches 1024, data will be encoded(to compute its checksum or encrypted data) and write to disk. So the disk file consists of multiple fixed-size blocks. The exception is that the last block size may be less than 1024.
  3. In the origin implementation, if we call read function of ListInDisk, we have to flush all data in cache. Then we can read data from disk. Because in this way, the implementation of read function is clean and simple.
  4. So if the read function has been called before, the write function cannot be called anymore. Because except for the last block, the guarantee of the fixed size of other blocks will be broken.

How to fix this problem:

The root cause of the problem is that the read function of the checksum/encrypt layer does not have the logic to read the cache, which causes we have to flush data every time we want to read it.

So I add a new layer ReaderWithCache, which can help read the cached data. So no need to flush data every time.

What is changed and how it works?

What's Changed:

  1. Add a new layer ReaderWithCache, it will first read data in disk, then read data in mem cache.
  2. Writer for checksum and aes_layer should tell ReaderWithCache the mem cache and the offset of data that resides in mem cache. So GetCache() and GetCacheDataOffset() are added.

The following figure shows the different levels that need to be passed through in the reading process:

RowContainer
  |
ListInDisk
  |
**ReaderWithCache**
  |
checksum
  |
**ReaderWithCache**
  |
aes
  |
disk

Related changes

None

Check List

Tests

  • Integration test:
  1. TestChecksumWriter
  2. TestChecksumWriterAutoFlush
  3. testReaderWithCache
  4. testReaderWithCacheNoFlush

Side effects: Because we add a new layer when read, so there may be performance lost.

Release note

  • executor: enhancement for ListInDisk(support writing after reading)

@guo-shaoge guo-shaoge requested a review from a team as a code owner April 29, 2021 09:01
@guo-shaoge guo-shaoge requested review from qw4990 and removed request for a team April 29, 2021 09:01
@ti-chi-bot ti-chi-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Apr 29, 2021
@guo-shaoge
Copy link
Collaborator Author

/run-all-tests

@guo-shaoge
lint code
96aac28 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
Copy link
Collaborator Author

@wshwsh12 PTAL

@github-actions github-actions bot added the sig/execution SIG execution label Apr 29, 2021
@guo-shaoge
Copy link
Collaborator Author

/cc @ichn-hu

@ti-chi-bot ti-chi-bot requested a review from ichn-hu April 29, 2021 12:45
@guo-shaoge
Copy link
Collaborator Author

/run-all-tests

guo-shaoge added 3 commits May 6, 2021 05:13
@guo-shaoge
fix cases
0928bba 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix cases
2972591 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix case
e25e8a6 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
Copy link
Collaborator Author

/run-all-tests

@guo-shaoge
Copy link
Collaborator Author

/run-unit-test

@guo-shaoge
Copy link
Collaborator Author

/cc @wshwsh12

@ti-chi-bot ti-chi-bot requested a review from wshwsh12 May 6, 2021 09:52
wjhuang2016
wjhuang2016 reviewed May 8, 2021
View reviewed changes
util/chunk/disk.go Outdated Show resolved Hide resolved
wjhuang2016
wjhuang2016 reviewed May 8, 2021
View reviewed changes
util/checksum/checksum.go Show resolved Hide resolved
util/chunk/disk.go Show resolved Hide resolved
util/checksum/checksum_test.go Show resolved Hide resolved
wjhuang2016
wjhuang2016 reviewed May 8, 2021
View reviewed changes
Copy link
Member

@wjhuang2016 wjhuang2016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please describe how these changes solve the problem.

guo-shaoge added 3 commits May 9, 2021 20:20
@guo-shaoge
lint code
0f388a6 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix lint
02e3245 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix comments
a8e9931 
Signed-off-by: guo-shaoge <[email protected]>
@wjhuang2016
Copy link
Member

/lgtm

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 13, 2021
guo-shaoge added 3 commits May 18, 2021 04:37
@guo-shaoge
fix comments. make code more concise.
405eded 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
add isReaderStale
c45b41d 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix lint
923734b 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge guo-shaoge requested a review from wjhuang2016 May 18, 2021 04:50
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • ichn-hu
  • wjhuang2016

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 18, 2021
@wjhuang2016
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b03692057e94196e9ab9fa653e288e436518121f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 18, 2021
@guo-shaoge
Revert "add isReaderStale"
0759556 
This reverts commit c45b41d.
@guo-shaoge
Copy link
Collaborator Author

/run-check_dev_2
/run-integration-common-test
/run-sqllogic-test-2
/run-unit-test

@guo-shaoge
Copy link
Collaborator Author

/run-unit-test

@ti-chi-bot
Copy link
Member

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

@guo-shaoge
Copy link
Collaborator Author

/run-unit-test

@wjhuang2016
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label May 18, 2021
@guo-shaoge
Copy link
Collaborator Author

/run-all-tests

@wjhuang2016
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 33f7e91

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 19, 2021
@wjhuang2016
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

@guo-shaoge: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

@ti-chi-bot ti-chi-bot merged commit 424a5a8 into pingcap:master May 19, 2021
Howie59 pushed a commit to Howie59/tidb that referenced this pull request May 21, 2021
@tiancaiamao @Howie59
*: fix revoke statement for CURRENT_USER() and refine error message (p…
f679a36 
…ingcap#24052)

* *: fix revoke statement for CURRENT_USER() and refine error message

planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (pingcap#24516)

store/tikv: remove use of SchemaAmender option in store/tikv (pingcap#24408)

*: the value of tikv-client.store-liveness-timeout should not less than 0 (pingcap#24244)

store/tikv: remove use of EnableAsyncCommit option in store/tikv (pingcap#24462)

txn: Add txn state's view (pingcap#22908)

planner: ignore lock for temporary table of PointGet and BatchPointGet (pingcap#24540)

store/tikv: remove use of ReplicaRead transaction option in store/tikv (pingcap#24409)

store/driver: move error to single package (pingcap#24549)

ddl: add check table compatibility for temporary table (pingcap#24501)

store/tikv: remove use of IsStatenessReadOnly option in store/tikv (pingcap#24464)

store/tikv: change backoff type for missed tiflash peer. (pingcap#24577)

store/tikv: remove use of MatchStoreLabels transaction option in store/tikv (pingcap#24465)

executor, meta: Allocate auto id for global temporary tables (pingcap#24506)

store/tikv: remove use of SampleStep option in store/tikv (pingcap#24461)

executor: add partition pruning tests for adding and dropping partition operations (pingcap#24573)

ddl: forbid partition on temporary mode before put into queue (pingcap#24565)

ddl: speedup test case TestIndexOnMultipleGeneratedColumn (pingcap#24487)

execution: Fix issue 24439 Inconsistent error with MySQL for GRANT CREATE USER ON <specific db>.* (pingcap#24485)

*: fix errcheck (pingcap#24463)

test: make TestExtractStartTs stable (pingcap#24585)

ddl: forbid recover/flashback temporary tables (pingcap#24518)

executor: fix point_get result on clustered index when new-row-format disabled but new-collation enabled (pingcap#24544)

executor: Improve the performance of appending not fixed columns (pingcap#20969)

*: typo fix (pingcap#24564)

planner/core: refresh stale regions in cache for batch cop response (pingcap#24457)

binlog: DML on temporary tables do not write binlog (pingcap#24570)

store/tikv: remove use of GuaranteeLinearizability option in store/tikv (pingcap#24605)

store/tikv: remove use of CollectRuntimeStats option in store/tikv (pingcap#24604)

store/tikv: move Backoffer into a single package (pingcap#24525)

variables: init cte max recursive deeps in a new session (pingcap#24609)

store/tikv: move transaction options out to /kv (pingcap#24619)

store/driver: move backoff driver into single package so we can use i… (pingcap#24624)

server: close the temporary session in HTTP API to avoid memory leak (pingcap#24339)

store/tikv: use latest PD TS plus one as min commit ts (pingcap#24579)

planner: fix incorrect TableDual plan built from nulleq (pingcap#24596)

ranger: fix the case which could have duplicate ranges (pingcap#24590)

 executor, store: Pass the SQL digest down to pessimistic lock request (pingcap#24380)

kv: remove UnionStore interface (pingcap#24625)

*: enable gosimple linter (pingcap#24617)

txn: avoid the gc resolving pessimistic locks of ongoing transactions (pingcap#24601)

util: fix wrong enum building for index range  (pingcap#24632)

sessionctx: change innodb large prefix default (pingcap#24555)

store: fix data race about KVStore.tikvClient (pingcap#24655)

executor, privileges: Add dynamic privileges to SHOW PRIVILEGES (pingcap#24646)

ddl: refactor rule [4/6] (pingcap#24007)

cmd: ddl_test modify retryCnt from 5 to 20 (pingcap#24662)

executor: add correctness tests about direct reading with ORDER BY and LIMIT (pingcap#24455)

store/tikv: remove options from unionstore (pingcap#24629)

planner: fix wrongly check for update statement (pingcap#24614)

store/tikv: remove CompareTS (pingcap#24657)

planner, privilege: Add security enhanced mode part 4 (pingcap#24416)

executor: add some test cases about partition table dynamic-mode with split-region (pingcap#24665)

planner: fix wrong column offsets when processing dynamic pruning for IndexJoin (pingcap#24659)

*: Add security enhanced mode part 3 (pingcap#24412)

store/tikv: resolve ReplicaReadType dependencies (pingcap#24653)

executor: add test cases about partition table with `expression` (pingcap#24628)

tablecodec: fix write wrong prefix index value when collation is ascii_bin/latin1_bin (pingcap#24578)

*: compatibility with staleread (pingcap#24285)

session: test that temporary tables will also be retried (pingcap#24505)

domain, session: Add new sysvarcache to replace global values cache (pingcap#24359)

ddl, transaction: DDL on temporary tables won't affect transactions (pingcap#24534)

*: implement tidb_bounded_staleness built-in function (pingcap#24328)

executor: add correctness tests for partition table with different joins (pingcap#24673)

expression: fix the spelling of word arithmetical (pingcap#24713)

store/copr: balance region for batch cop task (pingcap#24521)

store, metrics: Add metrics for safetTS updating (pingcap#24687)

sem: add tidbredact log to restricted variables (pingcap#24701)

session: fix dml_batch_size doesn't load the global variable (pingcap#24710)

store/tikv: retry TSO RPC (pingcap#24682)

expression, planner: push cast down to control function with enum type. (pingcap#24542)

executor: add correctness tests about IndexMerge (pingcap#24674)

variable: change default for DefDMLBatchSize tidbOptInt64 call (pingcap#24697)

planner: add partitioning pruning tests for range partitioning (pingcap#24554)

*: add option for enum push down (pingcap#24685)

txn: break dependency from store/tikv to tidb/kv cause by TransactionOption (pingcap#24656)

executor: enhancement for ListInDisk(support writing after reading) (pingcap#24379)

kv: move TxnScope into kv (pingcap#24715)

execution: fix the incorrect use of cached plan for point get (pingcap#24749)

executor: add correctness tests about direct reading with indexJoin (pingcap#24497)

variable:  fix sysvar datarace with deep copy (pingcap#24732)

*: Implementing RENAME USER (pingcap#24413)

infoschema, executor: Add the deadlock table (pingcap#24524)

docs: Some proposal for renaming and configurations for Lock View (pingcap#24718)

planner: add range partition boundaries tests with BETWEEN expression (pingcap#24598)

oracle: simplify timestamp utilities (pingcap#24688)

executor: fix wrong enum key in point get (pingcap#24618)

ranger: fix incorrect enum range for xxx_ci collation (pingcap#24661)

executor: add some test cases about dynamic-mode and apply operator (pingcap#24683)

store/tikv: remove Variables.Hook (pingcap#24758)

ddl: speed up the execution time of `TestBackwardCompatibility`. (pingcap#24704)

*: prepare errors for CTE (pingcap#24763)

expression: support cast real/int as real (pingcap#24670)

executor: add table name in log (pingcap#24666)

expression: add builtin function ``json_pretty`` (pingcap#24675)

ddl: make `TestDropLastVisibleColumns` stable (pingcap#24790)

* ddl: make `TestDropLastVisibleColumns` stable

*: support AS OF TIMESTAMP read-only begin statement (pingcap#24740)

executor: Fix unstable TestTiDBLastTxnInfoCommitMode (pingcap#24779)

planner: add tests for partition range boundaries for LT/GT (pingcap#24574)

test: record random seed in TestIssue20658 (pingcap#24782)

store/tikv/retry: define Config instead of BackoffType (pingcap#24692)

config: ignore tiflash when show config (pingcap#24770)

privileges: improve dynamic privs registration and tests (pingcap#24773)

README: remove the link of TiDB Monthly Update (pingcap#24791)

region_cache: filter peers on tombstone or dropped stores (pingcap#24726)

util/stmtsummary: remove import package tikv (pingcap#24776)

ddl: grammar check for create unsupported temporary table (pingcap#24723)

*: update go.etcd.io/bbolt (pingcap#24799)

ddl: speed up the execution time of `ddl test` and `Test Chunk pingcap#7 ddl-other` (pingcap#24780)

executor: remove the unnecessary use of fmt.Sprintf (pingcap#24815)

executor: fix index join panic on prefix index on some cases (pingcap#24568)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ichn-hu ichn-hu ichn-hu approved these changes

@qw4990 qw4990 Awaiting requested review from qw4990

@wshwsh12 wshwsh12 Awaiting requested review from wshwsh12

@wjhuang2016 wjhuang2016 Awaiting requested review from wjhuang2016

Assignees
No one assigned
Labels
sig/execution SIG execution size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enhancement for ListInDisk
4 participants
@guo-shaoge @wjhuang2016 @ti-chi-bot @ichn-hu