Do region flush in Region::handleWriteRaftCmd #542
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
flowbehappy
requested review from
JaySon-Huang,
zanmato1984 and
solotzg
and removed request for
JaySon-Huang
JaySon-Huang
added
the
needs-cherry-pick-release-3.1
zanmato1984
reviewed
Mar 23, 2020
| } | ||
| } | ||
|
|
||
| /// Save removed data to outer. |
There was a problem hiding this comment.
Why still move data_list_read to outer if data is already removed above.
There was a problem hiding this comment.
It is used by RegionTable::tryFlushRegion and finally used at here: https://github.com/pingcap/tics/blob/master/dbms/src/Storages/StorageDeltaMerge.cpp#L495
BTW we will remove that use case later. But let's do it in another PR.
|
@zanmato1984 @solotzg PTAL |
flowbehappy
added
the
status/can-merge
|
Your auto merge job has been accepted, waiting for 544, 547 |
|
/run-all-tests |
|
@flowbehappy merge failed. |
|
/merge |
|
Your auto merge job has been accepted, waiting for 540, 543 |
|
/run-all-tests |
|
cherry pick to release-3.1 in PR #551 |
JaySon-Huang
pushed a commit
that referenced
this pull request
Mar 24, 2020
* Flush Region committed data in Region::handleWriteRaftCmd * optimize code Co-authored-by: flow <[email protected]>
windtalker
added a commit
that referenced
this pull request
Mar 26, 2020
* fix daily test fail (#520) * fix daily test fail * fix * Add fullstack test for engine DeltaTree (#524) ## Add fullstack test for engine DeltaTree. * Refine `tests/docker/run.sh` and split `tests/docker/docker-compose.yaml` into `tests/docker/{gtest/mock-test/cluster/tiflash-dt/tiflash-tmt}.yaml` `fullstack/ddl`,`fullstack-test/fault-inject` will be enabled in #526 ## Others * Add column `tidb_table_id` in `system.tables` * Add some debugging info Signed-off-by: JaySon-Huang <[email protected]> * Bugfix: schrodinger bank2 fail (#521) * Flush committed data in Region after resolve locks * Stop append into last packs after split. * Remove last_cache in Delta to reduce code complexity. * Add system table: dt_tables and dt_segments, for debug. * [FLASH-1008] Support br restore & ingest sst (#529) * [flash-1018]fix bug of datetime default value (#534) * fix bug of datetime default value * address comment * Using SegmentSnapshotPtr instead of SegmentSnapshot (#532) * [Flash-664] Enable DDL for engine DeltaTree (#526) - [x] Enable unittest in gtest_dbms - [x] Enable mock test in tests/delta-merge-test - [x] Enable fullstack-test/ddl - [x] Enable fullstack-test/inject (Imported in #443) - [x] Refine exception while read / write to DeltaTree (FLASH-994) * Use `Exception::addMessage` to add more diagnostics for locate which table is wrong (commit: 716ae4a) - [x] shutdown should cancel all background tasks (FLASH-995) (commit: 7470c2f) - [x] Run schrodinger/sddl test ## Others * Add atomic-rename table test in `tests/fullstack-test/fault-inject/rename-table.test`, but did not enable this. We will fix it later. * "dt" engine ONLY support disable_bg_flush = true. If background flush is enabled, read will not triggle schema sync. Which means that we may not get the right result with out-dated schema. * Found that 'zero' value of type year is not match with tikv (FLASH-1023) Signed-off-by: JaySon-Huang <[email protected]> * [FLASH-1027] Fix: proxy override system signal listening (#541) * remove signal listening from proxy. * while terminating, stop all learner read. * [FLASH-1026] Synchronization between drop table and remove region (#538) * Set default storage engine to DT (#547) * Do region flush in Region::handleWriteRaftCmd (#542) * [FLASH-1028] Region merge should not remove data (#544) * Remove region after region merge should not remove data Signed-off-by: JaySon-Huang <[email protected]> * Fix region init index Signed-off-by: JaySon-Huang <[email protected]> * Add region_merge.test for DT Signed-off-by: JaySon-Huang <[email protected]> * Fix different behavior between DT and TMT Signed-off-by: JaySon-Huang <[email protected]> * Fix mock remove region Signed-off-by: JaySon-Huang <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * clean useless code Co-authored-by: JaySon <[email protected]> Co-authored-by: Flowyi <[email protected]> Co-authored-by: Tong Zhigao <[email protected]> Co-authored-by: Han Fei <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]>
hanfei1991
added a commit
that referenced
this pull request
Jun 22, 2020
* implement join * fix * update tipb * update tipb * save work: refine dag interpreter, introduce dag interpreter query block * fix bug * comment useless code * refine code * fix bug * refine code * fix bug * save work * tiny fix * save work * support remote read * update tipb * refine code * fix bug * update client-c * refine cop read * update client-c to support cop reading from tiflash * refine code * support batch cop * fix build error * fix daily test fail * some bug fix * log dag execution time without encode to chunk * fix bug * fix bug * log dag execution time without encode * make encode multi processors * fix * parallel encode * refine code of batch coprocessor * refine code * delete useless code * update kvproto and client-c * [flash 1002]refine coprocessor read (#530) * refind coprocessor * fix * try fix ci * support key ranges in batch coprocessor (#533) * save work * save work * save work * save work * dt support key ranges in dag request * fix bug * fix bug * fix bug * add some comments * fix bug * address comments * merge master branch (#556) * fix daily test fail (#520) * fix daily test fail * fix * Add fullstack test for engine DeltaTree (#524) ## Add fullstack test for engine DeltaTree. * Refine `tests/docker/run.sh` and split `tests/docker/docker-compose.yaml` into `tests/docker/{gtest/mock-test/cluster/tiflash-dt/tiflash-tmt}.yaml` `fullstack/ddl`,`fullstack-test/fault-inject` will be enabled in #526 ## Others * Add column `tidb_table_id` in `system.tables` * Add some debugging info Signed-off-by: JaySon-Huang <[email protected]> * Bugfix: schrodinger bank2 fail (#521) * Flush committed data in Region after resolve locks * Stop append into last packs after split. * Remove last_cache in Delta to reduce code complexity. * Add system table: dt_tables and dt_segments, for debug. * [FLASH-1008] Support br restore & ingest sst (#529) * [flash-1018]fix bug of datetime default value (#534) * fix bug of datetime default value * address comment * Using SegmentSnapshotPtr instead of SegmentSnapshot (#532) * [Flash-664] Enable DDL for engine DeltaTree (#526) - [x] Enable unittest in gtest_dbms - [x] Enable mock test in tests/delta-merge-test - [x] Enable fullstack-test/ddl - [x] Enable fullstack-test/inject (Imported in #443) - [x] Refine exception while read / write to DeltaTree (FLASH-994) * Use `Exception::addMessage` to add more diagnostics for locate which table is wrong (commit: 716ae4a) - [x] shutdown should cancel all background tasks (FLASH-995) (commit: 7470c2f) - [x] Run schrodinger/sddl test ## Others * Add atomic-rename table test in `tests/fullstack-test/fault-inject/rename-table.test`, but did not enable this. We will fix it later. * "dt" engine ONLY support disable_bg_flush = true. If background flush is enabled, read will not triggle schema sync. Which means that we may not get the right result with out-dated schema. * Found that 'zero' value of type year is not match with tikv (FLASH-1023) Signed-off-by: JaySon-Huang <[email protected]> * [FLASH-1027] Fix: proxy override system signal listening (#541) * remove signal listening from proxy. * while terminating, stop all learner read. * [FLASH-1026] Synchronization between drop table and remove region (#538) * Set default storage engine to DT (#547) * Do region flush in Region::handleWriteRaftCmd (#542) * [FLASH-1028] Region merge should not remove data (#544) * Remove region after region merge should not remove data Signed-off-by: JaySon-Huang <[email protected]> * Fix region init index Signed-off-by: JaySon-Huang <[email protected]> * Add region_merge.test for DT Signed-off-by: JaySon-Huang <[email protected]> * Fix different behavior between DT and TMT Signed-off-by: JaySon-Huang <[email protected]> * Fix mock remove region Signed-off-by: JaySon-Huang <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * clean useless code Co-authored-by: JaySon <[email protected]> Co-authored-by: Flowyi <[email protected]> Co-authored-by: Tong Zhigao <[email protected]> Co-authored-by: Han Fei <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * fix type mismatch bug in broadcast join * broadcast join support join keys with different data type (#580) * fix type mismatch bug in broadcast join * refine code * refine code * some improvement for broadcast join (#600) * some improvement for broadcast join * format code * refine code * address comment * fix bug * address comments * fix bug * fix bug * make TiFlash backward compatible to old tipb (#653) * 1. re-enable exec info in dag response, 2. support old style dag request * basic support for execute summary * refine support of executor time for join plan * format code * address comments * fix bug * refine code * update header * Fix execute details regression after merge master (#678) * refine code * fix bug * fix bug * format code * update kvproto * fmt code * update client-c * update tipb Co-authored-by: xufei <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: JaySon <[email protected]> Co-authored-by: Flowyi <[email protected]> Co-authored-by: Tong Zhigao <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]>
hanfei1991
added a commit
that referenced
this pull request
Jul 2, 2020
* implement join * fix * update tipb * update tipb * save work: refine dag interpreter, introduce dag interpreter query block * fix bug * comment useless code * refine code * fix bug * refine code * fix bug * save work * tiny fix * save work * support remote read * update tipb * refine code * fix bug * update client-c * refine cop read * update client-c to support cop reading from tiflash * refine code * support batch cop * fix build error * fix daily test fail * some bug fix * log dag execution time without encode to chunk * fix bug * fix bug * log dag execution time without encode * make encode multi processors * fix * parallel encode * refine code of batch coprocessor * refine code * delete useless code * update kvproto and client-c * [flash 1002]refine coprocessor read (#530) * refind coprocessor * fix * try fix ci * support key ranges in batch coprocessor (#533) * save work * save work * save work * save work * dt support key ranges in dag request * fix bug * fix bug * fix bug * add some comments * fix bug * address comments * merge master branch (#556) * fix daily test fail (#520) * fix daily test fail * fix * Add fullstack test for engine DeltaTree (#524) ## Add fullstack test for engine DeltaTree. * Refine `tests/docker/run.sh` and split `tests/docker/docker-compose.yaml` into `tests/docker/{gtest/mock-test/cluster/tiflash-dt/tiflash-tmt}.yaml` `fullstack/ddl`,`fullstack-test/fault-inject` will be enabled in #526 ## Others * Add column `tidb_table_id` in `system.tables` * Add some debugging info Signed-off-by: JaySon-Huang <[email protected]> * Bugfix: schrodinger bank2 fail (#521) * Flush committed data in Region after resolve locks * Stop append into last packs after split. * Remove last_cache in Delta to reduce code complexity. * Add system table: dt_tables and dt_segments, for debug. * [FLASH-1008] Support br restore & ingest sst (#529) * [flash-1018]fix bug of datetime default value (#534) * fix bug of datetime default value * address comment * Using SegmentSnapshotPtr instead of SegmentSnapshot (#532) * [Flash-664] Enable DDL for engine DeltaTree (#526) - [x] Enable unittest in gtest_dbms - [x] Enable mock test in tests/delta-merge-test - [x] Enable fullstack-test/ddl - [x] Enable fullstack-test/inject (Imported in #443) - [x] Refine exception while read / write to DeltaTree (FLASH-994) * Use `Exception::addMessage` to add more diagnostics for locate which table is wrong (commit: 716ae4a) - [x] shutdown should cancel all background tasks (FLASH-995) (commit: 7470c2f) - [x] Run schrodinger/sddl test ## Others * Add atomic-rename table test in `tests/fullstack-test/fault-inject/rename-table.test`, but did not enable this. We will fix it later. * "dt" engine ONLY support disable_bg_flush = true. If background flush is enabled, read will not triggle schema sync. Which means that we may not get the right result with out-dated schema. * Found that 'zero' value of type year is not match with tikv (FLASH-1023) Signed-off-by: JaySon-Huang <[email protected]> * [FLASH-1027] Fix: proxy override system signal listening (#541) * remove signal listening from proxy. * while terminating, stop all learner read. * [FLASH-1026] Synchronization between drop table and remove region (#538) * Set default storage engine to DT (#547) * Do region flush in Region::handleWriteRaftCmd (#542) * [FLASH-1028] Region merge should not remove data (#544) * Remove region after region merge should not remove data Signed-off-by: JaySon-Huang <[email protected]> * Fix region init index Signed-off-by: JaySon-Huang <[email protected]> * Add region_merge.test for DT Signed-off-by: JaySon-Huang <[email protected]> * Fix different behavior between DT and TMT Signed-off-by: JaySon-Huang <[email protected]> * Fix mock remove region Signed-off-by: JaySon-Huang <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * clean useless code Co-authored-by: JaySon <[email protected]> Co-authored-by: Flowyi <[email protected]> Co-authored-by: Tong Zhigao <[email protected]> Co-authored-by: Han Fei <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * fix type mismatch bug in broadcast join * broadcast join support join keys with different data type (#580) * fix type mismatch bug in broadcast join * refine code * refine code * some improvement for broadcast join (#600) * some improvement for broadcast join * format code * refine code * address comment * fix bug * address comments * fix bug * fix bug * make TiFlash backward compatible to old tipb (#653) * 1. re-enable exec info in dag response, 2. support old style dag request * basic support for execute summary * refine support of executor time for join plan * format code * address comments * fix bug * refine code * update header * Fix execute details regression after merge master (#678) * refine code * fix bug * fix bug * format code * update kvproto * fmt code * update client-c * update tipb Co-authored-by: xufei <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: JaySon <[email protected]> Co-authored-by: Flowyi <[email protected]> Co-authored-by: Tong Zhigao <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: xufei <[email protected]> Co-authored-by: JaySon <[email protected]> Co-authored-by: Flowyi <[email protected]> Co-authored-by: Tong Zhigao <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Region::handleWriteRaftCmd, so that it be protected byRegion::mutexandRegion::predecode_mutexRegionTable::writeBlockByRegionfromRegionTable:: flushRegion, so thatwriteBlockByRegioncan be called directly fromRegion::handleWriteRaftCmd. Note that we don't callflushRegiondirectly because we try to avoid getting instance ofRegionTable, some unnecessary check and print log. AsRegion::handleWriteRaftCmdis very performance critical.