[DocDB] Tablet splitting test TabletSplitITest.PostSplitCompactionDoesntBlockTabletCleanup failure

[rthallamko3]

Jira Link: [DB-441](https://yugabyte.atlassian.net/browse/DB-441)

Description

The tablet split itest fails consistenty - https://detective.dev.yugabyte.com/stability/test?branch=master&class=TabletSplitITest&name=PostSplitCompactionDoesntBlockTabletCleanup

Logs from one the runs (https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang11-debug/333/artifact/build/debug-clang11-dynamic-ninja/yb-test-logs/tests-integration-tests__tablet-split-itest/TabletSplitITest_PostSplitCompactionDoesntBlockTabletCleanup.log) indicates that the test timed out while waiting for split to complete.

I0411 12:52:33.202724 29119 tablet-split-itest-base.cc:458] Tablet splitting did not complete. Crashing test with core dump. Received error: Timed out (yb/util/test_util.cc:287): Operation 'Wait for tablet split to be completed' didn't complete within 40000ms
*** Aborted at 1649681553 (unix time) try "date -d @1649681553" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x71000071bf) received by PID 29119 (TID 0x7ff23abf3200) from PID 29119; stack trace: ***
@ 0x7ff23cddd630 (unknown)
@ 0x7ff23cddd4fb raise
@ 0x7ff248315a7e yb::TabletSplitITest::WaitForTabletSplitCompletion()
@ 0x45163b yb::TabletSplitITest_PostSplitCompactionDoesntBlockTabletCleanup_Test::TestBody()
@ 0x7ff245d58e28 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x7ff245d40b11 testing::Test::Run()
@ 0x7ff245d41930 testing::TestInfo::Run()
@ 0x7ff245d42137 testing::TestCase::Run()
@ 0x7ff245d49c67 testing::internal::UnitTestImpl::RunAllTests()
@ 0x7ff245d59828 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x7ff245d49745 testing::UnitTest::Run()
@ 0x7ff2482b1c71 RUN_ALL_TESTS()
@ 0x7ff2482b16ec main
@ 0x7ff23ca22555 __libc_start_main
@ 0x44c4eb (unknown)
I0411 12:52:33.333200 29118 run-with-timeout.cc:198] subprocess.Wait finished, waitpid() returned 139
I0411 12:52:33.333266 29118 run-with-timeout.cc:205] Waiting for reaper thread to join
I0411 12:52:33.333298 29120 run-with-timeout.cc:182] Reaper thread: finished=1, subprocess_is_running=0, timeout_sec=601
I0411 12:52:33.333350 29118 run-with-timeout.cc:209] Child process returned exit code 0
E0411 12:52:33.333362 29118 run-with-timeout.cc:221] Child process terminated due to signal 11
I0411 12:52:33.333366 29118 run-with-timeout.cc:224] Returning exit code 1
I0411 12:52:33.333370 29118 run-with-timeout.cc:226] Total time taken: 48.831 sec
Found a core file at '/tmp/yb_test.tests-integration-tests__tablet-split-itest___TabletSplitITest.PostSplitCompactionDoesntBlockTabletCleanup.20220411125144422004.5810197560/core.29119', backtrace:

warning: core file may not match specified executable file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/nfusr/centos-gcp-cloud/jenkins-worker-6smu6e/jenkins/jenkins-github-yugabyte-d'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007ff23cddd4fb in raise () from /lib64/libpthread.so.0
#0 0x00007ff23cddd4fb in raise () from /lib64/libpthread.so.0
#1 0x00007ff248315a7e in yb::TabletSplitITest::WaitForTabletSplitCompletion(unsigned long, unsigned long, unsigned long, yb::client::YBTableName const&, bool) (this=0x2ace240, expected_non_split_tablets=3, expected_split_tablets=1, num_replicas_online=3, table=..., core_dump_on_failure=true) at ../../src/yb/integration-tests/tablet-split-itest-base.cc:460
#2 0x000000000045163b in yb::TabletSplitITest_PostSplitCompactionDoesntBlockTabletCleanup_Test::TestBody() (this=0x2ace240) at ../../src/yb/integration-tests/tablet-split-itest.cc:310
#3 0x00007ff245d58e28 in HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=, method=, location=0x7ff245d28f1b "the test body") at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2402
#4 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) (object=, method=, location=0x7ff245d28f1b "the test body") at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2438
#5 0x00007ff245d40b11 in testing::Test::Run() (this=0x2ace240) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2474
#6 0x00007ff245d41930 in testing::TestInfo::Run() (this=0x2af43c0) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2656
#7 0x00007ff245d42137 in testing::TestCase::Run() (this=0x2b06690) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2774
#8 0x00007ff245d49c67 in testing::internal::UnitTestImpl::RunAllTests() (this=this@entry=0x2a64200) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:4649
#9 0x00007ff245d59828 in HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=, method=, location=0x7ff245d26ec9 "auxiliary test code (environments or event listeners)") at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2402
#10 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) (object=, method=, location=0x7ff245d26ec9 "auxiliary test code (environments or event listeners)") at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:2438
#11 0x00007ff245d49745 in testing::UnitTest::Run() (this=0x7ff245d71c38 testing::UnitTest::GetInstance()::instance) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/src/gmock-1.8.0/googletest/src/gtest.cc:4257
#12 0x00007ff2482b1c71 in RUN_ALL_TESTS() () at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220223071629-16e9b7854d-centos7-x86_64-clang11/installed/uninstrumented/include/gtest/gtest.h:2233

[bmatican]

Seems fixed on master