Add backward compatibility check in stress test #27928

Avogar · 2021-08-20T12:18:50Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add backward compatibility check in stress test.
Closes #25088

Detailed description / Documentation draft:
...

Avogar · 2021-08-23T11:51:16Z

@mergify update

mergify · 2021-08-23T11:51:37Z

Command update: success

Hey, I reacted but my real name is @Mergifyio

Avogar · 2021-08-23T16:17:10Z

@Mergifyio update

mergify · 2021-08-23T16:17:24Z

Command update: success

Avogar · 2021-08-27T11:38:05Z

@Mergifyio update

mergify · 2021-08-27T11:38:38Z

Command update: success

Avogar · 2021-08-30T10:35:18Z

@Mergifyio update

mergify · 2021-08-30T10:35:31Z

Command update: success

Avogar · 2021-09-03T08:55:51Z

@Mergifyio update

mergify · 2021-09-03T08:56:11Z

Command update: success

Avogar · 2021-09-06T11:48:19Z

@Mergifyio update

mergify · 2021-09-06T11:49:00Z

Command update: success

Avogar · 2021-09-09T14:31:31Z

@Mergifyio update

mergify · 2021-09-09T14:31:36Z

Command update: success

Avogar · 2021-09-17T12:34:19Z

@Mergifyio update

mergify · 2021-09-17T12:34:34Z

Command update: success

CLAassistant · 2021-09-28T08:47:21Z

All committers have signed the CLA.

Avogar · 2021-09-29T10:49:53Z

@Mergifyio update

mergify · 2021-09-29T10:49:58Z

Command update: failure

Avogar · 2021-09-29T11:09:16Z

@Mergifyio update

mergify · 2021-09-29T11:09:21Z

Command update: success

Avogar · 2022-03-03T13:04:23Z

Found new problem:

<Error> DDLWorker(test_5): Error on initialization of test_5: Code: 49. DB::Exception: It's new replica, but database is not empty. (LOGICAL_ERROR)

Investigating.

Avogar · 2022-03-05T08:33:45Z

@Mergifyio update

mergify · 2022-03-05T08:34:08Z

✅ Branch has been successfully updated

Avogar · 2022-03-05T16:06:42Z

New problem was found:

2022.03.05 17:09:23.584385 [ 165479 ] {} <Fatal> BaseDaemon: ########################################
2022.03.05 17:09:23.586348 [ 165479 ] {} <Fatal> BaseDaemon: (version 22.3.1.1, build id: 7262517FF0840CD4) (from thread 165477) (no query) Received signal Aborted (6)
2022.03.05 17:09:23.587168 [ 165479 ] {} <Fatal> BaseDaemon:
2022.03.05 17:09:23.587644 [ 165479 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fca925a703b 0x7fca92586859 0x273206d4 0x2b2aa568 0x2b28f3ce 0x2b28f33d 0x2b28f2e0 0x2b29793d 0x2b297905 0x2b297662 0x7fca9275e609 0x7fca92683163
2022.03.05 17:09:23.588232 [ 165479 ] {} <Fatal> BaseDaemon: 4. raise @ 0x7fca925a703b in ?
2022.03.05 17:09:23.588418 [ 165479 ] {} <Fatal> BaseDaemon: 5. abort @ 0x7fca92586859 in ?
2022.03.05 17:09:23.765745 [ 165479 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/Coordination/KeeperStateManager.cpp:124: ? @ 0x273206d4 in /usr/bin/clickhouse
2022.03.05 17:09:23.948419 [ 165479 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../contrib/NuRaft/src/handle_commit.cxx:157: nuraft::raft_server::commit_in_bg() @ 0x2b2aa568 in /usr/bin/clickhouse
2022.03.05 17:09:24.208912 [ 165479 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3617: decltype(*(std::__1::forward<nuraft::raft_server*&>(fp0)).*fp()) std::__1::__invoke<void (nuraft::raft_server::*&)(), nuraft::raft_server*&, voi
d>(void (nuraft::raft_server::*&)(), nuraft::raft_server*&) @ 0x2b28f3ce in /usr/bin/clickhouse
2022.03.05 17:09:24.466859 [ 165479 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2857: std::__1::__bind_return<void (nuraft::raft_server::*)(), std::__1::tuple<nuraft::raft_server*>, std::__1::tuple<>, __is_valid_bind_return<void (n
uraft::raft_server::*)(), std::__1::tuple<nuraft::raft_server*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (nuraft::raft_server::*)(), std::__1::tuple<nuraft::raft_server*>, 0ul, std::__1::tuple<> >(void (nuraft::raft_server::*&)(), std::__1::tuple<nur
aft::raft_server*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) @ 0x2b28f33d in /usr/bin/clickhouse
2022.03.05 17:09:24.740276 [ 165479 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2890: std::__1::__bind_return<void (nuraft::raft_server::*)(), std::__1::tuple<nuraft::raft_server*>, std::__1::tuple<>, __is_valid_bind_return<void (
nuraft::raft_server::*)(), std::__1::tuple<nuraft::raft_server*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (nuraft::raft_server::*)(), nuraft::raft_server*>::operator()<>() @ 0x2b28f2e0 in /usr/bin/clickhouse
2022.03.05 17:09:25.020073 [ 165479 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676: decltype(std::__1::forward<std::__1::__bind<void (nuraft::raft_server::*)(), nuraft::raft_server*> >(fp)()) std::__1::__invoke<std::__1::__bind
<void (nuraft::raft_server::*)(), nuraft::raft_server*> >(std::__1::__bind<void (nuraft::raft_server::*)(), nuraft::raft_server*>&&) @ 0x2b29793d in /usr/bin/clickhouse
2022.03.05 17:09:25.296354 [ 165479 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:281: void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bi
nd<void (nuraft::raft_server::*)(), nuraft::raft_server*> >(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (nuraft::raft_server::*)(), nuraft::raft_server*> >&, std::__1::__tuple_in
dices<>) @ 0x2b297905 in /usr/bin/clickhouse
2022.03.05 17:09:25.553137 [ 165479 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:291: void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >,
 std::__1::__bind<void (nuraft::raft_server::*)(), nuraft::raft_server*> > >(void*) @ 0x2b297662 in /usr/bin/clickhouse
2022.03.05 17:09:25.553730 [ 165479 ] {} <Fatal> BaseDaemon: 14. ? @ 0x7fca9275e609 in ?
2022.03.05 17:09:25.554114 [ 165479 ] {} <Fatal> BaseDaemon: 15. clone @ 0x7fca92683163 in ?
2022.03.05 17:09:26.676828 [ 165479 ] {} <Fatal> BaseDaemon: Calculated checksum of the binary: 36D7E04286D7C2622A93B9E8AFAB18E2. There is no information about the reference checksum.
2022.03.05 17:09:26.680519 [ 165479 ] {} <Information> SentryWriter: Not sending crash report
2022.03.05 17:09:43.766270 [ 165252 ] {} <Fatal> Application: Child process was terminated by signal 6.

Will be fixed in #35021

Avogar · 2022-03-09T16:20:00Z

@Mergifyio update

mergify · 2022-03-09T16:22:46Z

✅ Branch has been successfully updated

Should be merged with ClickHouse#27928

Avogar · 2022-03-17T11:10:32Z

@Mergifyio update

mergify · 2022-03-17T11:10:37Z

✅ Branch has been successfully updated

tavplubix · 2022-07-26T12:27:42Z

src/Client/ClientBase.cpp

+    if (fake_drop)
+    {
+        if (parsed_query->as<ASTDropQuery>())
+            return;


Does it make sense to replace DROP with TRUNCATE for table engines like Memory or Join to reduce memory usage?

But how we will know which table engine is used by drop query?

By executing one more query to get table engine or by moving this logic to server (but it's questionable)

Makes sense, I will try it (created issue to not forget)

robot-clickhouse added the pr-build Pull request with build/testing/packaging improvement label Aug 20, 2021

Avogar marked this pull request as draft August 20, 2021 12:18

Avogar force-pushed the stress-test branch from 6403605 to 791e3e7 Compare August 31, 2021 07:40

Avogar force-pushed the stress-test branch from 84f40b5 to baba027 Compare September 15, 2021 13:48

Avogar added the force tests label Sep 19, 2021

Avogar force-pushed the stress-test branch 2 times, most recently from 9987c45 to 9a60109 Compare September 27, 2021 12:41

Avogar mentioned this pull request Sep 27, 2021

Server failed to restart. #29456

Closed

Avogar mentioned this pull request Sep 30, 2021

Cannot restart server: <Error> Application: DB::Exception: Dictionary (...) not found #29559

Closed

Avogar added 2 commits February 28, 2022 13:33

Update run.sh

6252588

Update run.sh

8f6ba99

Avogar added 2 commits March 4, 2022 14:24

Update run.sh

0f90996

Merge branch 'master' into stress-test

dbba752

Merge branch 'master' into stress-test

977bd30

Fix style

607ec6c

Merge branch 'master' into stress-test

78c8029

Avogar added 2 commits March 10, 2022 14:38

Increase the number of attempts to start server

91b9c11

Update run.sh

3d36ece

vdimir added a commit to vdimir/ClickHouse that referenced this pull request Mar 11, 2022

Use download_previous_release for bugfix validation

4947d6d

Should be merged with ClickHouse#27928

Avogar and others added 7 commits March 14, 2022 16:40

Fix tests

b9496af

Update 00942_dataparts_500.sh

b249a1a

Update 00732_quorum_insert_lost_part_and_alive_part_zookeeper_long.sql

f955920

Update 02222_create_table_without_columns_metadata.sh

9058e77

Update 02025_storage_filelog_virtual_col.sh

e68a66d

Update run.sh

6b3a25e

Update run.sh

fa6d963

Merge branch 'master' into stress-test

5295dd0

Avogar merged commit fbcc27a into ClickHouse:master Mar 21, 2022

tavplubix reviewed Jul 26, 2022

View reviewed changes

Avogar mentioned this pull request Jul 27, 2022

Try to truncate Memory/Join tables instead of fake drop in backward-compatibility check to reduce memory usage #39636

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add backward compatibility check in stress test #27928

Add backward compatibility check in stress test #27928

Avogar commented Aug 20, 2021 •

edited

Loading

Avogar commented Aug 23, 2021

mergify bot commented Aug 23, 2021

Avogar commented Aug 23, 2021

mergify bot commented Aug 23, 2021

Avogar commented Aug 27, 2021

mergify bot commented Aug 27, 2021

Avogar commented Aug 30, 2021

mergify bot commented Aug 30, 2021

Avogar commented Sep 3, 2021

mergify bot commented Sep 3, 2021

Avogar commented Sep 6, 2021

mergify bot commented Sep 6, 2021

Avogar commented Sep 9, 2021

mergify bot commented Sep 9, 2021

Avogar commented Sep 17, 2021

mergify bot commented Sep 17, 2021

CLAassistant commented Sep 28, 2021 •

edited

Loading

Avogar commented Sep 29, 2021

mergify bot commented Sep 29, 2021

Avogar commented Sep 29, 2021

mergify bot commented Sep 29, 2021

Avogar commented Mar 3, 2022 •

edited

Loading

Avogar commented Mar 5, 2022

mergify bot commented Mar 5, 2022

Avogar commented Mar 5, 2022

Avogar commented Mar 9, 2022

mergify bot commented Mar 9, 2022

Avogar commented Mar 17, 2022

mergify bot commented Mar 17, 2022

tavplubix Jul 26, 2022

Avogar Jul 27, 2022

tavplubix Jul 27, 2022

Avogar Jul 27, 2022

Add backward compatibility check in stress test #27928

Add backward compatibility check in stress test #27928

Conversation

Avogar commented Aug 20, 2021 • edited Loading

Avogar commented Aug 23, 2021

mergify bot commented Aug 23, 2021

Avogar commented Aug 23, 2021

mergify bot commented Aug 23, 2021

Avogar commented Aug 27, 2021

mergify bot commented Aug 27, 2021

Avogar commented Aug 30, 2021

mergify bot commented Aug 30, 2021

Avogar commented Sep 3, 2021

mergify bot commented Sep 3, 2021

Avogar commented Sep 6, 2021

mergify bot commented Sep 6, 2021

Avogar commented Sep 9, 2021

mergify bot commented Sep 9, 2021

Avogar commented Sep 17, 2021

mergify bot commented Sep 17, 2021

CLAassistant commented Sep 28, 2021 • edited Loading

Avogar commented Sep 29, 2021

mergify bot commented Sep 29, 2021

Avogar commented Sep 29, 2021

mergify bot commented Sep 29, 2021

Avogar commented Mar 3, 2022 • edited Loading

Avogar commented Mar 5, 2022

mergify bot commented Mar 5, 2022

✅ Branch has been successfully updated

Avogar commented Mar 5, 2022

Avogar commented Mar 9, 2022

mergify bot commented Mar 9, 2022

✅ Branch has been successfully updated

Avogar commented Mar 17, 2022

mergify bot commented Mar 17, 2022

✅ Branch has been successfully updated

tavplubix Jul 26, 2022

Choose a reason for hiding this comment

Avogar Jul 27, 2022

Choose a reason for hiding this comment

tavplubix Jul 27, 2022

Choose a reason for hiding this comment

Avogar Jul 27, 2022

Choose a reason for hiding this comment

Avogar commented Aug 20, 2021 •

edited

Loading

CLAassistant commented Sep 28, 2021 •

edited

Loading

Avogar commented Mar 3, 2022 •

edited

Loading