Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KVStore: introduce flexible region format #8614

Merged
merged 53 commits into from
Jan 3, 2024

Conversation

CalvinNeo
Copy link
Member

@CalvinNeo CalvinNeo commented Dec 27, 2023

What problem does this PR solve?

Issue Number: close #8590 close #8624 close #8625

Problem Summary:

The idea is mainly explained in #8590.

Also fix some problems.

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    upgrade test
    image
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

Signed-off-by: CalvinNeo <[email protected]>
@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 27, 2023
Signed-off-by: CalvinNeo <[email protected]>
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. do-not-merge/needs-linked-issue labels Dec 27, 2023
@CalvinNeo
Copy link
Member Author

/run-all-tests

Comment on lines 142 to 151
// Try serialize extra flags
if (binary_version >= 2)
{
static_assert(sizeof(eager_truncated_index) == sizeof(UInt64));
WriteBufferFromOwnString sub_buf;
auto sub_size = writeBinary2(eager_truncated_index, sub_buf);
RUNTIME_CHECK(sub_size, sizeof(UInt64));
auto s = sub_buf.releaseStr();
total_size += writePersistExtension(buf, RegionPersistFormat::EagerTruncate, s.data(), s.size());
}

This comment was marked as resolved.

Signed-off-by: CalvinNeo <[email protected]>
@CalvinNeo CalvinNeo changed the title KVStore: introduce flexible region format wip KVStore: introduce flexible region format Dec 28, 2023
@ti-chi-bot ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 28, 2023
Signed-off-by: CalvinNeo <[email protected]>
Signed-off-by: CalvinNeo <[email protected]>
Signed-off-by: CalvinNeo <[email protected]>
f
Signed-off-by: CalvinNeo <[email protected]>
@CalvinNeo CalvinNeo changed the title wip KVStore: introduce flexible region format KVStore: introduce flexible region format Dec 28, 2023
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 28, 2023
dbms/src/Storages/KVStore/MultiRaft/RegionSerde.cpp Outdated Show resolved Hide resolved
dbms/src/Storages/KVStore/MultiRaft/RegionSerde.cpp Outdated Show resolved Hide resolved
dbms/src/Storages/KVStore/MultiRaft/RegionSerde.cpp Outdated Show resolved Hide resolved
dbms/src/Storages/KVStore/MultiRaft/RegionSerde.cpp Outdated Show resolved Hide resolved
Comment on lines 118 to 119
UInt32 flags = RegionPersistFormat::HAS_EAGER_TRUNCATE_INDEX;
flags |= (expected_extension_count << 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brilliant!

Signed-off-by: CalvinNeo <[email protected]>
@CalvinNeo
Copy link
Member Author

/run-all-tests

@CalvinNeo
Copy link
Member Author

CalvinNeo commented Jan 2, 2024

[2024-01-02T11:08:12.268Z]   File: fullstack-test2/ddl/alter_drop_table.test
[2024-01-02T11:08:12.269Z]   Error line: 64
[2024-01-02T11:08:12.269Z]   Error: select tidb_database,tidb_name from system.tables where tidb_database = 'test' and tidb_name = 't3' and is_tombstone = 0
[2024-01-02T11:08:12.269Z]   Result:
[2024-01-02T11:08:12.269Z]     Received exception from server (version 7.6.0):
[2024-01-02T11:08:12.269Z]     Code: 1000. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Exception: Receive TsoResponse failed.
[2024-01-02T11:08:12.269Z]   Expected:
[2024-01-02T11:08:12.269Z]     <nothing>
[2024-01-02T11:08:12.279Z] [Pipeline] sh
[2024-01-02T11:08:12.955Z] fullstack-test/expr/column_filter.test: OK [5.845 s]
[2024-01-02T11:08:12.955Z] fullstack-test/expr/compare_year_with_date.test: Running
[2024-01-02T11:08:14.697Z] tidb-ci/force_enable_lm/generated_index.test: OK [4.180 s]
[2024-01-02T11:08:14.697Z] tidb-ci/force_enable_lm/get_format.test: Running
[2024-01-02T11:08:15.469Z] + docker ps -a
[2024-01-02T11:08:15.469Z] CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS               NAMES
[2024-01-02T11:08:15.469Z] 94372599fd92        hub.pingcap.net/qa/tidb:master            "/tidb-server --stor…"   8 minutes ago       Up 8 minutes        4000/tcp            fullstack-test2_tidb0_1
[2024-01-02T11:08:15.469Z] 0503accc183f        hub.pingcap.net/tiflash/tiflash-ci-base   "/tiflash/tiflash se…"   8 minutes ago       Up 8 minutes                            fullstack-test2_tiflash0_1
[2024-01-02T11:08:15.469Z] 3ddcfd2c63ac        hub.pingcap.net/qa/tikv:master            "/tikv-server --addr…"   8 minutes ago       Up 8 minutes        20160/tcp           fullstack-test2_tikv0_1
[2024-01-02T11:08:15.469Z] 76455d9eb948        hub.pingcap.net/qa/pd:master              "/pd-server --name=p…"   8 minutes ago       Up 8 minutes        2379-2380/tcp   

@CalvinNeo
Copy link
Member Author

/run-all-tests

@CalvinNeo
Copy link
Member Author

/run-all-tests

@CalvinNeo
Copy link
Member Author

/unhold

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2024
@CalvinNeo
Copy link
Member Author

/unhold

@CalvinNeo
Copy link
Member Author

/hold

@ti-chi-bot ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2024
z
Signed-off-by: CalvinNeo <[email protected]>
Signed-off-by: CalvinNeo <[email protected]>
@CalvinNeo
Copy link
Member Author

/unhold

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2024
@CalvinNeo
Copy link
Member Author

/run-all-tests

@CalvinNeo
Copy link
Member Author

/run-integration-test

@CalvinNeo
Copy link
Member Author

[2024/01/03 01:45:31.814 +08:00] [ERROR] [SegmentReader.cpp:118] ["ErrMsg: Argument at index 1 for function fromUnixTime must be constant StackTrace \n       0x42ad38e\tStackTrace::StackTrace() [tiflash+69915534]\n       0x429bb62\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+69843810]\n       0x459a481\tDB::Exception::Exception<unsigned long&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) [tiflash+72983681]\n       0xdf315f8\tDB::IExecutableFunction::defaultImplementationForConstantArguments(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [tiflash+234034680]\n       0xdf31ec1\tDB::IExecutableFunction::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [tiflash+234036929]\n       0x478c45e\tDB::IFunctionBase::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [tiflash+75023454]\n       0xcb3cb4a\tDB::ExpressionAction::execute(DB::Block&) const [tiflash+213109578]\n       0xcb3e6aa\tDB::ExpressionActions::execute(DB::Block&) const [tiflash+213116586]\n       0xc72ffee\tDB::FilterTransformAction::FilterTransformAction(DB::Block const&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208863214]\n       0xc72d4c6\tDB::FilterBlockInputStream::FilterBlockInputStream(std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208852166]\n       0xc44dbe7\tDB::FilterBlockInputStream* std::__1::construct_at<DB::FilterBlockInputStream, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::FilterBlockInputStream*>(DB::FilterBlockInputStream*, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+205839335]\n       0xc44d990\tvoid std::__1::allocator_traits<std::__1::allocator<DB::FilterBlockInputStream> >::construct<DB::FilterBlockInputStream, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void, void>(std::__1::allocator<DB::FilterBlockInputStream>&, DB::FilterBlockInputStream*, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+205838736]\n       0xc44d683\tstd::__1::__shared_ptr_emplace<DB::FilterBlockInputStream, std::__1::allocator<DB::FilterBlockInputStream> >::__shared_ptr_emplace<std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::allocator<DB::FilterBlockInputStream>, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+205837955]\n       0xc44d495\tstd::__1::shared_ptr<DB::FilterBlockInputStream> std::__1::allocate_shared<DB::FilterBlockInputStream, std::__1::allocator<DB::FilterBlockInputStream>, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void>(std::__1::allocator<DB::FilterBlockInputStream> const&, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+205837461]\n       0xc3cc15c\tstd::__1::shared_ptr<DB::FilterBlockInputStream> std::__1::make_shared<DB::FilterBlockInputStream, std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void>(std::__1::shared_ptr<DB::IBlockInputStream>&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+205308252]\n       0xc3ba9fc\tDB::DM::Segment::getLateMaterializationStream(std::__1::shared_ptr<DB::DM::BitmapFilter>&&, DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, unsigned long, unsigned long) [tiflash+205236732]\n       0xc3a4818\tDB::DM::Segment::getBitmapFilterInputStream(DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, unsigned long, unsigned long, unsigned long) [tiflash+205146136]\n       0xc3a28e2\tDB::DM::Segment::getInputStream(DB::DM::ReadMode const&, DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, unsigned long, unsigned long) [tiflash+205138146]\n       0xc476727\tDB::DM::SegmentReadTask::doInitInputStream(std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, DB::DM::ReadMode, unsigned long) [tiflash+206006055]\n       0xc4761ef\tDB::DM::SegmentReadTask::doInitInputStreamWithErrorFallback(std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, DB::DM::ReadMode, unsigned long, bool) [tiflash+206004719]\n       0xc475fba\tDB::DM::SegmentReadTask::initInputStream(std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, unsigned long, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, DB::DM::ReadMode, unsigned long, bool) [tiflash+206004154]\n       0xc4af9fe\tDB::DM::SegmentReadTaskPool::buildInputStream(std::__1::shared_ptr<DB::DM::SegmentReadTask>&) [tiflash+206240254]\n       0xc65c4c1\tDB::DM::MergedTask::initOnce() [tiflash+207996097]\n       0xc65c3d9\tDB::DM::MergedTask::readBlock() [tiflash+207995865]\n       0xc66bacb\tDB::DM::SegmentReader::readSegments() [tiflash+208059083]\n       0xc66adde\tDB::DM::SegmentReader::run() [tiflash+208055774]\n       0xc66de59\tdecltype(*(static_cast<DB::DM::SegmentReader*>(fp0)).*fp()) std::__1::__invoke<void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, void>(void (DB::DM::SegmentReader::*&&)(), DB::DM::SegmentReader*&&) [tiflash+208068185]\n       0xc66ddde\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*>&, std::__1::__tuple_indices<2ul>) [tiflash+208068062]\n       0xc66daf2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*> >(void*) [tiflash+208067314]\n  0x7f4e388e1ea5\tstart_thread [libpthread.so.0+32421]\n  0x7f4e381f096d\t__clone [libc.so.6+1042797]"] [source="MPP<gather_id:1, query_ts:1704217531646343366, local_query_id:302, server_id:173, start_ts:446750400603750405,task_id:1> table_id=525"] [thread_id=36]

@CalvinNeo
Copy link
Member Author

/run-all-tests

Copy link
Contributor

ti-chi-bot bot commented Jan 3, 2024

@CalvinNeo: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Copy link
Contributor

ti-chi-bot bot commented Jan 3, 2024

@CalvinNeo: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@JaySon-Huang
Copy link
Contributor

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/15729/pipeline/265

[2024-01-03T01:12:33.468Z]   File: fullstack-test2/ddl/flashback/flashback_database.test
[2024-01-03T01:12:33.468Z]   Error line: 167
[2024-01-03T01:12:33.468Z]   Error: set session tidb_isolation_read_engines='tiflash'; select * from d1_new.t2 order by id;
[2024-01-03T01:12:33.468Z]   Result:
[2024-01-03T01:12:33.468Z]     ERROR 1105 (HY000) at line 1: DB::TiFlashException: Failed to register MPP Task MPP<gather_id:1, query_ts:1704244343012329513, local_query_id:248, server_id:132, start_ts:446757429042216965,task_id:1>, reason: task is already registered
[2024-01-03T01:12:33.468Z]   Expected:
[2024-01-03T01:12:33.468Z]     +----+------+------+
[2024-01-03T01:12:33.468Z]     | id | name | b    |
[2024-01-03T01:12:33.468Z]     +----+------+------+
[2024-01-03T01:12:33.468Z]     |  1 | abc  | NULL |
[2024-01-03T01:12:33.468Z]     |  2 | cde  | NULL |
[2024-01-03T01:12:33.468Z]     | 53 | efg  | NULL |
[2024-01-03T01:12:33.468Z]     +----+------+------+

@JaySon-Huang
Copy link
Contributor

/run-integration-test

@ti-chi-bot ti-chi-bot bot merged commit 7273bbc into pingcap:master Jan 3, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
4 participants