Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make TiFlash backward compatible to old tipb #653

Merged
merged 5 commits into from
Apr 23, 2020
Merged

make TiFlash backward compatible to old tipb #653

merged 5 commits into from
Apr 23, 2020

Conversation

windtalker
Copy link
Contributor

  1. make TiFlash backward compatible to old tipb
  2. support executor summary in broadcast join
  3. fix potential bug for broadcast join when current timezone is not UTC

hanfei1991
hanfei1991 reviewed Apr 22, 2020
View reviewed changes
dbms/src/Flash/Coprocessor/DAGQueryBlock.cpp
@@ -60,30 +60,34 @@ void collectOutPutFieldTypesFromAgg(std::vector<tipb::FieldType> & field_type, c
}
}

DAGQueryBlock::DAGQueryBlock(UInt32 id_, const tipb::Executor * root_)
: id(id_), root(root_), qb_column_prefix("__QB_" + std::to_string(id_) + "_")
DAGQueryBlock::DAGQueryBlock(UInt32 id_, const tipb::Executor & root_)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some comments that point out this ctor is for new tipb, the other is for old format

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp Outdated
{
const tipb::Executor * current = root;
auto * exec = dag_req.add_executors();
auto * exec = dag_req.mutable_root_executor();
int i = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exec_id may be a better name ?

dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp
if (org_req.has_time_zone_name() && org_req.time_zone_name().length() > 0)
dag_req.set_time_zone_name(org_req.time_zone_name());
else if (org_req.has_time_zone_offset())
dag_req.set_time_zone_offset(org_req.time_zone_offset());
}

void DAGQueryBlockInterpreter::executeRemoteQuery(Pipeline & pipeline)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does remote (global) subquery record their execution time, should it get execution summary by remote server's response ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For simplicity, remote subquery only record the CopBlockInputStream time, for example, if the remote subquery is TS, then it's execute time is the execute time of CopBlockInputStream, if the remote subquery is TS SEL, then both TS and SEL's execute time is the same as the CopBlockInputStream

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, we can add this to comments

hanfei1991
hanfei1991 approved these changes Apr 23, 2020
View reviewed changes
Copy link
Member

@hanfei1991 hanfei1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hanfei1991 hanfei1991 merged commit 1d13d59 into pingcap:broadcast_join Apr 23, 2020
@hanfei1991 hanfei1991 deleted the broadcast_join branch April 23, 2020 06:23
hanfei1991 added a commit that referenced this pull request Jun 22, 2020
@hanfei1991 @windtalker
@JaySon-Huang @flowbehappy @solotzg @sre-bot
support broadcast join (#675)
6cdffb3 
* implement join

* fix

* update tipb

* update tipb

* save work: refine dag interpreter, introduce dag interpreter query block

* fix bug

* comment useless code

* refine code

* fix bug

* refine code

* fix bug

* save work

* tiny fix

* save work

* support remote read

* update tipb

* refine code

* fix bug

* update client-c

* refine cop read

* update client-c to support cop reading from tiflash

* refine code

* support batch cop

* fix build error

* fix daily test fail

* some bug fix

* log dag execution time without encode to chunk

* fix bug

* fix bug

* log dag execution time without encode

* make encode multi processors

* fix

* parallel encode

* refine code of batch coprocessor

* refine code

* delete useless code

* update kvproto and client-c

* [flash 1002]refine coprocessor read (#530)

* refind coprocessor

* fix

* try fix ci

* support key ranges in batch coprocessor (#533)

* save work

* save work

* save work

* save work

* dt support key ranges in dag request

* fix bug

* fix bug

* fix bug

* add some comments

* fix bug

* address comments

* merge master branch (#556)

* fix daily test fail (#520)

* fix daily test fail

* fix

* Add fullstack test for engine DeltaTree (#524)

## Add fullstack test for engine DeltaTree.
* Refine `tests/docker/run.sh` and split `tests/docker/docker-compose.yaml` into `tests/docker/{gtest/mock-test/cluster/tiflash-dt/tiflash-tmt}.yaml`

`fullstack/ddl`,`fullstack-test/fault-inject` will be enabled in #526 

## Others
* Add column `tidb_table_id` in `system.tables`
* Add some debugging info

Signed-off-by: JaySon-Huang <[email protected]>

* Bugfix: schrodinger bank2 fail (#521)

* Flush committed data in Region after resolve locks
* Stop append into last packs after split.
* Remove last_cache in Delta to reduce code complexity.
* Add system table: dt_tables and dt_segments, for debug.

* [FLASH-1008] Support br restore & ingest sst (#529)

* [flash-1018]fix bug of datetime default value (#534)

* fix bug of datetime default value

* address comment

* Using SegmentSnapshotPtr instead of SegmentSnapshot (#532)

* [Flash-664] Enable DDL for engine DeltaTree (#526)

- [x] Enable unittest in gtest_dbms
- [x] Enable mock test in tests/delta-merge-test
- [x] Enable fullstack-test/ddl
- [x] Enable fullstack-test/inject (Imported in #443)
- [x] Refine exception while read / write to DeltaTree (FLASH-994)
  * Use `Exception::addMessage` to add more diagnostics for locate which table is wrong (commit: 716ae4a)
- [x] shutdown should cancel all background tasks (FLASH-995) (commit: 7470c2f)
- [x] Run schrodinger/sddl test

## Others 
* Add atomic-rename table test in `tests/fullstack-test/fault-inject/rename-table.test`, but did not enable this. We will fix it later.
* "dt" engine ONLY support disable_bg_flush = true.
If background flush is enabled, read will not triggle schema sync. Which means that we may not get the right result with out-dated schema.
* Found that 'zero' value of type year is not match with tikv (FLASH-1023)

Signed-off-by: JaySon-Huang <[email protected]>

* [FLASH-1027] Fix: proxy override system signal listening (#541)

* remove signal listening from proxy.

* while terminating, stop all learner read.

* [FLASH-1026] Synchronization between drop table and remove region (#538)

* Set default storage engine to DT (#547)

* Do region flush in Region::handleWriteRaftCmd (#542)

* [FLASH-1028] Region merge should not remove data (#544)

* Remove region after region merge should not remove data

Signed-off-by: JaySon-Huang <[email protected]>

* Fix region init index

Signed-off-by: JaySon-Huang <[email protected]>

* Add region_merge.test for DT

Signed-off-by: JaySon-Huang <[email protected]>

* Fix different behavior between DT and TMT

Signed-off-by: JaySon-Huang <[email protected]>

* Fix mock remove region

Signed-off-by: JaySon-Huang <[email protected]>

Co-authored-by: pingcap-github-bot <[email protected]>

* clean useless code

Co-authored-by: JaySon <[email protected]>
Co-authored-by: Flowyi <[email protected]>
Co-authored-by: Tong Zhigao <[email protected]>
Co-authored-by: Han Fei <[email protected]>
Co-authored-by: pingcap-github-bot <[email protected]>

* fix type mismatch bug in broadcast join

* broadcast join support join keys with different data type (#580)

* fix type mismatch bug in broadcast join

* refine code

* refine code

* some improvement for broadcast join (#600)

* some improvement for broadcast join

* format code

* refine code

* address comment

* fix bug

* address comments

* fix bug

* fix bug

* make TiFlash backward compatible to old tipb (#653)

* 1. re-enable exec info in dag response, 2. support old style dag request

* basic support for execute summary

* refine support of executor time for join plan

* format code

* address comments

* fix bug

* refine code

* update header

* Fix execute details regression after merge master (#678)

* refine code

* fix bug

* fix bug

* format code

* update kvproto

* fmt code

* update client-c

* update tipb

Co-authored-by: xufei <[email protected]>
Co-authored-by: xufei <[email protected]>
Co-authored-by: JaySon <[email protected]>
Co-authored-by: Flowyi <[email protected]>
Co-authored-by: Tong Zhigao <[email protected]>
Co-authored-by: pingcap-github-bot <[email protected]>
hanfei1991 added a commit that referenced this pull request Jul 2, 2020
@hanfei1991 @windtalker
@JaySon-Huang @flowbehappy @solotzg @sre-bot
support broadcast join (#675) (#867)
846eb75 
* implement join

* fix

* update tipb

* update tipb

* save work: refine dag interpreter, introduce dag interpreter query block

* fix bug

* comment useless code

* refine code

* fix bug

* refine code

* fix bug

* save work

* tiny fix

* save work

* support remote read

* update tipb

* refine code

* fix bug

* update client-c

* refine cop read

* update client-c to support cop reading from tiflash

* refine code

* support batch cop

* fix build error

* fix daily test fail

* some bug fix

* log dag execution time without encode to chunk

* fix bug

* fix bug

* log dag execution time without encode

* make encode multi processors

* fix

* parallel encode

* refine code of batch coprocessor

* refine code

* delete useless code

* update kvproto and client-c

* [flash 1002]refine coprocessor read (#530)

* refind coprocessor

* fix

* try fix ci

* support key ranges in batch coprocessor (#533)

* save work

* save work

* save work

* save work

* dt support key ranges in dag request

* fix bug

* fix bug

* fix bug

* add some comments

* fix bug

* address comments

* merge master branch (#556)

* fix daily test fail (#520)

* fix daily test fail

* fix

* Add fullstack test for engine DeltaTree (#524)

## Add fullstack test for engine DeltaTree.
* Refine `tests/docker/run.sh` and split `tests/docker/docker-compose.yaml` into `tests/docker/{gtest/mock-test/cluster/tiflash-dt/tiflash-tmt}.yaml`

`fullstack/ddl`,`fullstack-test/fault-inject` will be enabled in #526 

## Others
* Add column `tidb_table_id` in `system.tables`
* Add some debugging info

Signed-off-by: JaySon-Huang <[email protected]>

* Bugfix: schrodinger bank2 fail (#521)

* Flush committed data in Region after resolve locks
* Stop append into last packs after split.
* Remove last_cache in Delta to reduce code complexity.
* Add system table: dt_tables and dt_segments, for debug.

* [FLASH-1008] Support br restore & ingest sst (#529)

* [flash-1018]fix bug of datetime default value (#534)

* fix bug of datetime default value

* address comment

* Using SegmentSnapshotPtr instead of SegmentSnapshot (#532)

* [Flash-664] Enable DDL for engine DeltaTree (#526)

- [x] Enable unittest in gtest_dbms
- [x] Enable mock test in tests/delta-merge-test
- [x] Enable fullstack-test/ddl
- [x] Enable fullstack-test/inject (Imported in #443)
- [x] Refine exception while read / write to DeltaTree (FLASH-994)
  * Use `Exception::addMessage` to add more diagnostics for locate which table is wrong (commit: 716ae4a)
- [x] shutdown should cancel all background tasks (FLASH-995) (commit: 7470c2f)
- [x] Run schrodinger/sddl test

## Others 
* Add atomic-rename table test in `tests/fullstack-test/fault-inject/rename-table.test`, but did not enable this. We will fix it later.
* "dt" engine ONLY support disable_bg_flush = true.
If background flush is enabled, read will not triggle schema sync. Which means that we may not get the right result with out-dated schema.
* Found that 'zero' value of type year is not match with tikv (FLASH-1023)

Signed-off-by: JaySon-Huang <[email protected]>

* [FLASH-1027] Fix: proxy override system signal listening (#541)

* remove signal listening from proxy.

* while terminating, stop all learner read.

* [FLASH-1026] Synchronization between drop table and remove region (#538)

* Set default storage engine to DT (#547)

* Do region flush in Region::handleWriteRaftCmd (#542)

* [FLASH-1028] Region merge should not remove data (#544)

* Remove region after region merge should not remove data

Signed-off-by: JaySon-Huang <[email protected]>

* Fix region init index

Signed-off-by: JaySon-Huang <[email protected]>

* Add region_merge.test for DT

Signed-off-by: JaySon-Huang <[email protected]>

* Fix different behavior between DT and TMT

Signed-off-by: JaySon-Huang <[email protected]>

* Fix mock remove region

Signed-off-by: JaySon-Huang <[email protected]>

Co-authored-by: pingcap-github-bot <[email protected]>

* clean useless code

Co-authored-by: JaySon <[email protected]>
Co-authored-by: Flowyi <[email protected]>
Co-authored-by: Tong Zhigao <[email protected]>
Co-authored-by: Han Fei <[email protected]>
Co-authored-by: pingcap-github-bot <[email protected]>

* fix type mismatch bug in broadcast join

* broadcast join support join keys with different data type (#580)

* fix type mismatch bug in broadcast join

* refine code

* refine code

* some improvement for broadcast join (#600)

* some improvement for broadcast join

* format code

* refine code

* address comment

* fix bug

* address comments

* fix bug

* fix bug

* make TiFlash backward compatible to old tipb (#653)

* 1. re-enable exec info in dag response, 2. support old style dag request

* basic support for execute summary

* refine support of executor time for join plan

* format code

* address comments

* fix bug

* refine code

* update header

* Fix execute details regression after merge master (#678)

* refine code

* fix bug

* fix bug

* format code

* update kvproto

* fmt code

* update client-c

* update tipb

Co-authored-by: xufei <[email protected]>
Co-authored-by: xufei <[email protected]>
Co-authored-by: JaySon <[email protected]>
Co-authored-by: Flowyi <[email protected]>
Co-authored-by: Tong Zhigao <[email protected]>
Co-authored-by: pingcap-github-bot <[email protected]>

Co-authored-by: xufei <[email protected]>
Co-authored-by: xufei <[email protected]>
Co-authored-by: JaySon <[email protected]>
Co-authored-by: Flowyi <[email protected]>
Co-authored-by: Tong Zhigao <[email protected]>
Co-authored-by: pingcap-github-bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hanfei1991 hanfei1991 hanfei1991 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@windtalker @hanfei1991