Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] ASAN heap-use-after-free in SealTxnTest.Update #17983

Open
1 task done
bmatican opened this issue Jun 28, 2023 · 2 comments
Open
1 task done

[DocDB] ASAN heap-use-after-free in SealTxnTest.Update #17983

bmatican opened this issue Jun 28, 2023 · 2 comments
Assignees
@yusong-yan
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug kind/failing-test Tests and testing infra priority/medium Medium priority issue

Comments

@bmatican
Copy link
Contributor

bmatican commented Jun 28, 2023 •
edited by jira bot
Loading

Jira Link: DB-7046

Description

https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=SealTxnTest&fail_tag=asan_heap_use_after_free&name=Update&platform=linux

example log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-alma8-master-clang16-asan/138/artifact/build/asan-clang16-dynamic-ninja/yb-test-logs/tests-client__seal-txn-test/SealTxnTest_Update.log

==22080==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000510110 at pc 0x7fa9b1e8d1d5 bp 0x7fa8d3fcb190 sp 0x7fa8d3fcb188
READ of size 8 at 0x614000510110 thread T234 (rpc_tp_TabletSe)
#0 0x7fa9b1e8d1d4 in std::shared_ptr::operator->[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/shared_ptr.h:857:16
#1 0x7fa9b1e8d1d4 in yb::tablet::TransactionCoordinator::Impl::GetStatus(google::protobuf::RepeatedPtrField const&, std::chrono::time_point>>, yb::tserver::GetTransactionStatusResponsePB*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1204:37
...

freed by thread T262 (consensus [work) here:
#0 0x5555a9b28edd in operator delete(void*) /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-1-1687337167-5c765d34-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:152:3
#1 0x7fa9b1e6ad22 in void std::__libcpp_operator_delete[abi:v160006](void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:276:3
#2 0x7fa9b1e6ad22 in void std::__do_deallocate_handle_size[abi:v160006]<>(void*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:300:10
#3 0x7fa9b1e6ad22 in std::__libcpp_deallocate[abi:v160006](void*, unsigned long, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:316:14
#4 0x7fa9b1e6ad22 in std::allocator>>>>>>::deallocate[abi:v160006](boost::multi_index::detail::hashed_index_node>>>>>*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/allocator.h:131:13
#5 0x7fa9b1e6ad22 in std::allocator_traits>>>>>>>::deallocate[abi:v160006](std::allocator>>>>>>&, boost::multi_index::detail::hashed_index_node>>>>>*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/allocator_traits.h:288:13
#6 0x7fa9b1e6ad22 in boost::multi_index::multi_index_container const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator>::deallocate_node(boost::multi_index::detail::hashed_index_node>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:645:5
#7 0x7fa9b1e6ad22 in boost::multi_index::multi_index_container const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator>::delete_node_(boost::multi_index::detail::hashed_index_node>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:956:5
#8 0x7fa9b1e6ad22 in boost::multi_index::multi_index_container const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator>::erase_(boost::multi_index::detail::hashed_index_node>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:950:5
#9 0x7fa9b1e6ad22 in boost::multi_index::detail::index_base const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator>::final_erase_(boost::multi_index::detail::hashed_index_node>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/detail/index_base.hpp:288:49
#10 0x7fa9b1e6ad22 in boost::multi_index::detail::hashed_index const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, boost::hash const>, std::equal_to const>, boost::multi_index::detail::nth_layer<1, yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, boost::multi_index::ordered_non_unique, boost::multi_index::const_mem_fun, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator>, boost::mpl::vector0, boost::multi_index::detail::hashed_unique_tag>::erase(boost::multi_index::detail::hashed_index_iterator>>>>>, boost::multi_index::detail::bucket_array>, boost::multi_index::detail::hashed_unique_tag, boost::multi_index::detail::hashed_index_global_iterator_tag>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230621185529-6777477baa-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/hashed_index.hpp:373:11
#11 0x7fa9b1e6ad22 in yb::tablet::TransactionCoordinator::Impl::CheckCompleted(boost::multi_index::detail::hashed_index_iterator>>>>>, boost::multi_index::detail::bucket_array>, boost::multi_index::detail::hashed_unique_tag, boost::multi_index::detail::hashed_index_global_iterator_tag>) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1868:29
#12 0x7fa9b1e801df in yb::tablet::TransactionCoordinator::Impl::ProcessReplicated(yb::tablet::TransactionCoordinator::ReplicatedData const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1379:7
#13 0x7fa9b1e57c65 in yb::tablet::TransactionCoordinator::ProcessReplicated(yb::tablet::TransactionCoordinator::ReplicatedData const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1934:17
#14 0x7fa9b1ff84fa in yb::tablet::UpdateTxnOperation::DoReplicated(long, yb::Status*) ${BUILD_ROOT}/../../src/yb/tablet/operations/update_txn_operation.cc:77:45
#15 0x7fa9b1f63dd0 in yb::tablet::Operation::Replicated(long, yb::StronglyTypedBool) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation.cc:82:3
#16 0x7fa9b1fa6fe0 in yb::tablet::OperationDriver::ApplyTask(long, std::vector>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:404:31
#17 0x7fa9b1fa643b in yb::tablet::OperationDriver::ReplicationFinished(yb::Status const&, long, std::vector>*)
...

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@bmatican bmatican added kind/failing-test Tests and testing infra area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Jun 28, 2023
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 28, 2023
@bmatican bmatican mentioned this issue Jun 28, 2023
Closed
1 task
@bmatican bmatican assigned robertsami and unassigned spolitov Jun 29, 2023
@bmatican
Copy link
Contributor Author

Hey Rob, I asked Sergei to do some quick triage and seems this points to https://phorge.dev.yugabyte.com/D12824, however, can't tell if it's specific to txn sealing (which we don't use in production as it was never finished), or generic to the txn subsystem.

Could you do one level further of triage to reason through that, so we see if it's a core product issue or not?

@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Jul 3, 2023
@rthallamko3
Copy link
Contributor

More recent asan failure points to different problem


Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-alma8-master-clang16-asan/753/artifact/build/asan-clang16-dynamic-ninja/yb-test-logs/tests-client__seal-txn-test/SealTxnTest_Update.log



==147991==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000382308 at pc 0x7ffa847f3668 bp 0x7ff9acc84f30 sp 0x7ff9acc84f28
READ of size 8 at 0x614000382308 thread T297 (rpc_tp_TabletSe)
[ts-2] I1017 08:50:25.099920 148281 retryable_requests_flusher.cc:151] Tablet {transactions_tablet_id3} is submitting flush retryable requests task...
[ts-2] I1017 08:50:25.099989 148281 log.cc:759] T {transactions_tablet_id3} P {ts2_peer_id}: Last appended OpId in segment ${TEST_TMPDIR}/seal-txn-test.SealTxnTest.Update.1697532620099406-147991/minicluster-data/ts-2-root/yb-data/tserver/wals/table-c4fe11baf0f74e54a5e6f0651b327511/tablet-{transactions_tablet_id3}/wal-000000004: 1.4
[ts-2] I1017 08:50:25.100373 148281 log.cc:768] T {transactions_tablet_id3} P {ts2_peer_id}: Rolled over to a new segment: ${TEST_TMPDIR}/seal-txn-test.SealTxnTest.Update.1697532620099406-147991/minicluster-data/ts-2-root/yb-data/tserver/wals/table-c4fe11baf0f74e54a5e6f0651b327511/tablet-{transactions_tablet_id3}/wal-000000005
[ts-3] I1017 08:50:25.101920 148276 log.cc:882] T {transactions_tablet_id3} P {ts3_peer_id}: Max segment size 128 reached. Starting new segment allocation.
[ts-3] I1017 08:50:25.118372 148276 retryable_requests_flusher.cc:151] Tablet {transactions_tablet_id3} is submitting flush retryable requests task...
[ts-3] I1017 08:50:25.118772 148276 log.cc:759] T {transactions_tablet_id3} P {ts3_peer_id}: Last appended OpId in segment ${TEST_TMPDIR}/seal-txn-test.SealTxnTest.Update.1697532620099406-147991/minicluster-data/ts-3-root/yb-data/tserver/wals/table-c4fe11baf0f74e54a5e6f0651b327511/tablet-{transactions_tablet_id3}/wal-000000004: 1.4
[ts-3] I1017 08:50:25.119408 148276 log.cc:768] T {transactions_tablet_id3} P {ts3_peer_id}: Rolled over to a new segment: ${TEST_TMPDIR}/seal-txn-test.SealTxnTest.Update.1697532620099406-147991/minicluster-data/ts-3-root/yb-data/tserver/wals/table-c4fe11baf0f74e54a5e6f0651b327511/tablet-{transactions_tablet_id3}/wal-000000005
[ts-3] I1017 08:50:25.210230 148222 transaction_status_cache.cc:152] TXN: 61ff93fd-70ce-45f2-bc8c-2a0c4602389d: Timed out waiting txn status, waited: 0.150s, future status: 1, left to deadline: 59.848s
[ts-1] I1017 08:50:25.213253 147991 tablet_server.cc:673] TabletServer shutting down...
[ts-1] I1017 08:50:25.213238 148060 tunnel.cc:172] Shutdown tunnel: 127.0.0.3:29576 => 127.0.0.2:29576
[ts-3] I1017 08:50:25.236014 148168 tablet_rpc.cc:158] Unable to pick leader for {transactions_tablet_id3}, replicas: [0x0000611000391c58 -> { uuid: {ts2_peer_id} private: [host: "127.0.0.4" port: 17689] public: [host: "127.0.0.5.ip.yugabyte" port: 17689] cloud_info: placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone", 0x000061100028a058 -> { uuid: {ts3_peer_id} private: [host: "127.0.0.6" port: 22635] public: [host: "127.0.0.7.ip.yugabyte" port: 22635] cloud_info: placement_cloud: "cloud2" placement_region: "rack3" placement_zone: "zone"], followers: [{0x000061100028a058 -> { uuid: {ts3_peer_id} private: [host: "127.0.0.6" port: 22635] public: [host: "127.0.0.7.ip.yugabyte" port: 22635] cloud_info: placement_cloud: "cloud2" placement_region: "rack3" placement_zone: "zone", { status: Illegal state (yb/consensus/consensus.cc:161): Not the leader (tablet server error 15) time: 0.007s }}, {0x0000611000391c58 -> { uuid: {ts2_peer_id} private: [host: "127.0.0.4" port: 17689] public: [host: "127.0.0.5.ip.yugabyte" port: 17689] cloud_info: placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone", { status: Illegal state (yb/consensus/consensus.cc:161): Not the leader (tablet server error 15) time: 0.014s }}]
[ts-3] I1017 08:50:25.246148 148168 tcp_stream.cc:324] { local: 127.0.0.7:37105 remote: 127.0.0.3:29576 }: Recv failed: Network error (yb/util/net/socket.cc:540): recvmsg error: Connection refused (system error 111)
#0 0x7ffa847f3667 in std::shared_ptr<yb::SubtxnSetAndPB const>::operator->[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/shared_ptr.h:857:16
#1 0x7ffa847f3667 in yb::tablet::TransactionCoordinator::Impl::GetStatus(google::protobuf::RepeatedPtrField<string> const&, std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>, yb::tserver::GetTransactionStatusResponsePB*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1210:37
#2 0x7ffa847be6c5 in yb::tablet::TransactionCoordinator::GetStatus(google::protobuf::RepeatedPtrField<string> const&, std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>, yb::tserver::GetTransactionStatusResponsePB*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1939:17
#3 0x7ffa85e3406a in yb::tserver::TabletServiceImpl::GetTransactionStatus(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext)::$_0::operator()(yb::tserver::LeaderTabletPeer const&) const ${BUILD_ROOT}/../../src/yb/tserver/tablet_service.cc:1157:37
#4 0x7ffa85e3406a in void yb::tserver::TabletServiceImpl::PerformAtLeader<yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB, yb::tserver::TabletServiceImpl::GetTransactionStatus(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext)::$_0>(yb::tserver::GetTransactionStatusRequestPB const* const&, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext*, yb::tserver::TabletServiceImpl::GetTransactionStatus(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext)::$_0 const&) ${BUILD_ROOT}/../../src/yb/tserver/tablet_service.cc:1132:17
#5 0x7ffa85e3406a in yb::tserver::TabletServiceImpl::GetTransactionStatus(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext) ${BUILD_ROOT}/../../src/yb/tserver/tablet_service.cc:1149:3
#6 0x7ffa7cbfc424 in yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext)::operator()(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext) const ${BUILD_ROOT}/src/yb/tserver/tserver_service.service.cc:818:9
#7 0x7ffa7cbfc424 in auto yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::tserver::GetTransactionStatusRequestPB, yb::tserver::GetTransactionStatusResponsePB>, yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext)>(std::shared_ptr<yb::rpc::InboundCall>, yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::GetTransactionStatusRequestPB const*, yb::tserver::GetTransactionStatusResponsePB*, yb::rpc::RpcContext)) ${BUILD_ROOT}/../../src/yb/rpc/local_call.h:122:7
#8 0x7ffa7cbfc424 in yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10::operator()(std::shared_ptr<yb::rpc::InboundCall>) const ${BUILD_ROOT}/src/yb/tserver/tserver_service.service.cc:816:7
#9 0x7ffa7cbfc424 in decltype(std::declval<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10&>()(std::declval<std::shared_ptr<yb::rpc::InboundCall>>())) std::__invoke[abi:v160006]<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10&, std::shared_ptr<yb::rpc::InboundCall>>(yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/invoke.h:394:23
#10 0x7ffa7cbfc424 in void std::__invoke_void_return_wrapper<void, true>::__call<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10&, std::shared_ptr<yb::rpc::InboundCall>>(yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/invoke.h:487:9
#11 0x7ffa7cbfc424 in std::__function::__alloc_func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10, std::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:185:16
#12 0x7ffa7cbfc424 in std::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10, std::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_10>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:356:12
#13 0x7ffa7cbea874 in std::__function::__value_func<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#14 0x7ffa7cbea874 in std::function<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#15 0x7ffa7cbea874 in yb::tserver::TabletServerServiceIf::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/src/yb/tserver/tserver_service.service.cc:651:3
#16 0x7ffa792cd280 in yb::rpc::ServicePoolImpl::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:261:19
#17 0x7ffa7916e675 in yb::rpc::InboundCall::InboundCallTask::Run() ${BUILD_ROOT}/../../src/yb/rpc/inbound_call.cc:244:13
#18 0x7ffa792fdb90 in yb::rpc::(anonymous namespace)::Worker::Execute() ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:104:15
#19 0x7ffa7735784f in std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#20 0x7ffa7735784f in std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#21 0x7ffa7735784f in yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3
#22 0x7ffa722f01c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: c46c0e44b55ff27501f607770ed2ae993fe0b823)
#23 0x7ffa71d42e72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: fd0dc204c2454dadefbca307a1c7e2c2ef2071ac)

0x614000382308 is located 200 bytes inside of 416-byte region [0x614000382240,0x6140003823e0)
freed by thread T33 (consensus [work) here:
#0 0x5630c54ab0dd in operator delete(void*) /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:152:3
#1 0x7ffa847d0b52 in void std::__libcpp_operator_delete[abi:v160006]<void*>(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:276:3
#2 0x7ffa847d0b52 in void std::__do_deallocate_handle_size[abi:v160006]<>(void*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:300:10
#3 0x7ffa847d0b52 in std::__libcpp_deallocate[abi:v160006](void*, unsigned long, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:316:14
#4 0x7ffa847d0b52 in std::allocator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>>::deallocate[abi:v160006](boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/allocator.h:131:13
#5 0x7ffa847d0b52 in std::allocator_traits<std::allocator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>>>::deallocate[abi:v160006](std::allocator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>>&, boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/allocator_traits.h:288:13
#6 0x7ffa847d0b52 in boost::multi_index::multi_index_container<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::deallocate_node(boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:645:5
#7 0x7ffa847d0b52 in boost::multi_index::multi_index_container<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::delete_node_(boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:956:5
#8 0x7ffa847d0b52 in boost::multi_index::multi_index_container<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::erase_(boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:950:5
#9 0x7ffa847d0b52 in boost::multi_index::detail::index_base<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::final_erase_(boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/detail/index_base.hpp:288:49
#10 0x7ffa847d0b52 in boost::multi_index::detail::hashed_index<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, boost::hash<yb::StronglyTypedUuid<yb::TransactionId_Tag> const>, std::equal_to<yb::StronglyTypedUuid<yb::TransactionId_Tag> const>, boost::multi_index::detail::nth_layer<1, yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::mpl::vector0<mpl_::na>, boost::multi_index::detail::hashed_unique_tag>::erase(boost::multi_index::detail::hashed_index_iterator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>, boost::multi_index::detail::bucket_array<std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::multi_index::detail::hashed_unique_tag, boost::multi_index::detail::hashed_index_global_iterator_tag>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/hashed_index.hpp:373:11
#11 0x7ffa847d0b52 in yb::tablet::TransactionCoordinator::Impl::CheckCompleted(boost::multi_index::detail::hashed_index_iterator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>, boost::multi_index::detail::bucket_array<std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::multi_index::detail::hashed_unique_tag, boost::multi_index::detail::hashed_index_global_iterator_tag>) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1838:29
#12 0x7ffa847e657f in yb::tablet::TransactionCoordinator::Impl::ProcessReplicated(yb::tablet::TransactionCoordinator::ReplicatedData const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1403:7
#13 0x7ffa847be195 in yb::tablet::TransactionCoordinator::ProcessReplicated(yb::tablet::TransactionCoordinator::ReplicatedData const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1903:17
#14 0x7ffa84969b9e in yb::tablet::UpdateTxnOperation::DoReplicated(long, yb::Status*) ${BUILD_ROOT}/../../src/yb/tablet/operations/update_txn_operation.cc:75:45
#15 0x7ffa848d4ab0 in yb::tablet::Operation::Replicated(long, yb::StronglyTypedBool<yb::tablet::WasPending_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation.cc:82:3
#16 0x7ffa84918030 in yb::tablet::OperationDriver::ApplyTask(long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:406:31
#17 0x7ffa84917347 in yb::tablet::OperationDriver::ReplicationFinished(yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:364:5
#18 0x7ffa83c030b7 in yb::consensus::ConsensusRound::NotifyReplicationFinished(yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/consensus/consensus_round.cc:47:14
#19 0x7ffa83cf363d in yb::consensus::ReplicaState::NotifyReplicationFinishedUnlocked(scoped_refptr<yb::consensus::ConsensusRound> const&, yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:1464:10
#20 0x7ffa83cfc73a in yb::consensus::ReplicaState::ApplyPendingOperationsUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:961:5
#21 0x7ffa83cfa159 in yb::consensus::ReplicaState::AdvanceCommittedOpIdUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:873:17
#22 0x7ffa83cf9040 in yb::consensus::ReplicaState::UpdateMajorityReplicatedUnlocked(yb::OpId const&, yb::OpId*, bool*, yb::OpId*) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:771:32
#23 0x7ffa83c7832e in yb::consensus::RaftConsensus::UpdateMajorityReplicated(yb::consensus::MajorityReplicatedData const&, yb::OpId*, yb::OpId*) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:1398:15
#24 0x7ffa83bd2930 in yb::consensus::PeerMessageQueue::NotifyObserversOfMajorityReplOpChangeTask(yb::consensus::MajorityReplicatedData const&) ${BUILD_ROOT}/../../src/yb/consensus/consensus_queue.cc:1593:15
#25 0x7ffa773853af in yb::ThreadPool::DispatchThread(bool) ${BUILD_ROOT}/../../src/yb/util/threadpool.cc:612:22
#26 0x7ffa7735784f in std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#27 0x7ffa7735784f in std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#28 0x7ffa7735784f in yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3
#29 0x7ffa722f01c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: c46c0e44b55ff27501f607770ed2ae993fe0b823)

previously allocated by thread T86 (rpc_tp_TabletSe) here:
#0 0x5630c54aa87d in operator new(unsigned long) /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:95:3
#1 0x7ffa847eaae6 in void* std::__libcpp_operator_new[abi:v160006]<unsigned long>(unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:266:10
#2 0x7ffa847eaae6 in std::__libcpp_allocate[abi:v160006](unsigned long, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/new:292:10
#3 0x7ffa847eaae6 in std::allocator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>>::allocate[abi:v160006](unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/allocator.h:115:38
#4 0x7ffa847eaae6 in std::allocator_traits<std::allocator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>>>::allocate[abi:v160006](std::allocator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>>&, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/allocator_traits.h:268:20
#5 0x7ffa847eaae6 in boost::multi_index::multi_index_container<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::allocate_node() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:640:14
#6 0x7ffa847eaae6 in std::pair<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*, bool> boost::multi_index::multi_index_container<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::emplace_<yb::tablet::TransactionCoordinator::Impl*, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long, yb::HybridTime, string const&>(yb::tablet::TransactionCoordinator::Impl*&&, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long&&, yb::HybridTime&&, string const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index_container.hpp:783:24
#7 0x7ffa847eaae6 in std::pair<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>*, bool> boost::multi_index::detail::index_base<yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>::final_emplace_<yb::tablet::TransactionCoordinator::Impl*, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long, yb::HybridTime, string const&>(yb::tablet::TransactionCoordinator::Impl*&&, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long&&, yb::HybridTime&&, string const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/detail/index_base.hpp:248:20
#8 0x7ffa847eaae6 in std::pair<boost::multi_index::detail::hashed_index_iterator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>, boost::multi_index::detail::bucket_array<std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::multi_index::detail::hashed_unique_tag, boost::multi_index::detail::hashed_index_global_iterator_tag>, bool> boost::multi_index::detail::hashed_index<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, boost::hash<yb::StronglyTypedUuid<yb::TransactionId_Tag> const>, std::equal_to<yb::StronglyTypedUuid<yb::TransactionId_Tag> const>, boost::multi_index::detail::nth_layer<1, yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::mpl::vector0<mpl_::na>, boost::multi_index::detail::hashed_unique_tag>::emplace_impl<yb::tablet::TransactionCoordinator::Impl*, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long, yb::HybridTime, string const&>(yb::tablet::TransactionCoordinator::Impl*&&, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long&&, yb::HybridTime&&, string const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/hashed_index.hpp:1668:13
#9 0x7ffa847eaae6 in std::pair<boost::multi_index::detail::hashed_index_iterator<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<yb::tablet::(anonymous namespace)::TransactionState, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>>>>>, boost::multi_index::detail::bucket_array<std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::multi_index::detail::hashed_unique_tag, boost::multi_index::detail::hashed_index_global_iterator_tag>, bool> boost::multi_index::detail::hashed_index<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, boost::hash<yb::StronglyTypedUuid<yb::TransactionId_Tag> const>, std::equal_to<yb::StronglyTypedUuid<yb::TransactionId_Tag> const>, boost::multi_index::detail::nth_layer<1, yb::tablet::(anonymous namespace)::TransactionState, boost::multi_index::indexed_by<boost::multi_index::hashed_unique<boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, &yb::tablet::(anonymous namespace)::TransactionState::id() const>, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::LastTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, yb::HybridTime, &yb::tablet::(anonymous namespace)::TransactionState::last_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstTouchTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, unsigned long, &yb::tablet::(anonymous namespace)::TransactionState::first_touch() const>, mpl_::na>, boost::multi_index::ordered_non_unique<boost::multi_index::tag<yb::tablet::TransactionCoordinator::Impl::FirstEntryIndexTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::const_mem_fun<yb::tablet::(anonymous namespace)::TransactionState, long, &yb::tablet::(anonymous namespace)::TransactionState::first_entry_raft_index() const>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<yb::tablet::(anonymous namespace)::TransactionState>>, boost::mpl::vector0<mpl_::na>, boost::multi_index::detail::hashed_unique_tag>::emplace<yb::tablet::TransactionCoordinator::Impl*, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long, yb::HybridTime, string const&>(yb::tablet::TransactionCoordinator::Impl*&&, yb::StronglyTypedUuid<yb::TransactionId_Tag>&, unsigned long&&, yb::HybridTime&&, string const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/boost/multi_index/hashed_index.hpp:277:3
#10 0x7ffa847eaae6 in yb::tablet::TransactionCoordinator::Impl::Handle(std::unique_ptr<yb::tablet::UpdateTxnOperation, std::default_delete<yb::tablet::UpdateTxnOperation>>, long) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1477:38
#11 0x7ffa847be3d2 in yb::tablet::TransactionCoordinator::Handle(std::unique_ptr<yb::tablet::UpdateTxnOperation, std::default_delete<yb::tablet::UpdateTxnOperation>>, long) ${BUILD_ROOT}/../../src/yb/tablet/transaction_coordinator.cc:1920:10
#12 0x7ffa85e325d4 in yb::tserver::TabletServiceImpl::UpdateTransaction(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext) ${BUILD_ROOT}/../../src/yb/tserver/tablet_service.cc:1111:20
#13 0x7ffa7cbfa924 in yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext)::operator()(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext) const ${BUILD_ROOT}/src/yb/tserver/tserver_service.service.cc:803:9
#14 0x7ffa7cbfa924 in auto yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::tserver::UpdateTransactionRequestPB, yb::tserver::UpdateTransactionResponsePB>, yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext)>(std::shared_ptr<yb::rpc::InboundCall>, yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::tserver::UpdateTransactionRequestPB const*, yb::tserver::UpdateTransactionResponsePB*, yb::rpc::RpcContext)) ${BUILD_ROOT}/../../src/yb/rpc/local_call.h:122:7
#15 0x7ffa7cbfa924 in yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9::operator()(std::shared_ptr<yb::rpc::InboundCall>) const ${BUILD_ROOT}/src/yb/tserver/tserver_service.service.cc:801:7
#16 0x7ffa7cbfa924 in decltype(std::declval<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&>()(std::declval<std::shared_ptr<yb::rpc::InboundCall>>())) std::__invoke[abi:v160006]<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::shared_ptr<yb::rpc::InboundCall>>(yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/invoke.h:394:23
#17 0x7ffa7cbfa924 in void std::__invoke_void_return_wrapper<void, true>::__call<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::shared_ptr<yb::rpc::InboundCall>>(yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/invoke.h:487:9
#18 0x7ffa7cbfa924 in std::__function::__alloc_func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:185:16
#19 0x7ffa7cbfa924 in std::__function::__func<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9, std::allocator<yb::tserver::TabletServerServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_9>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:356:12
#20 0x7ffa7cbea874 in std::__function::__value_func<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#21 0x7ffa7cbea874 in std::function<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#22 0x7ffa7cbea874 in yb::tserver::TabletServerServiceIf::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/src/yb/tserver/tserver_service.service.cc:651:3
#23 0x7ffa792cd280 in yb::rpc::ServicePoolImpl::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:261:19
#24 0x7ffa7916e675 in yb::rpc::InboundCall::InboundCallTask::Run() ${BUILD_ROOT}/../../src/yb/rpc/inbound_call.cc:244:13
#25 0x7ffa792fdb90 in yb::rpc::(anonymous namespace)::Worker::Execute() ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:104:15
#26 0x7ffa7735784f in std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#27 0x7ffa7735784f in std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#28 0x7ffa7735784f in yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3
#29 0x7ffa722f01c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: c46c0e44b55ff27501f607770ed2ae993fe0b823)

Thread T297 (rpc_tp_TabletSe) created by T66 (TabletServer_re) here:
#0 0x5630c5457f6a in pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x7ffa77353e96 in yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15
#2 0x7ffa79300189 in yb::Status yb::Thread::Create<void (yb::rpc::(anonymous namespace)::Worker::*)(), yb::rpc::(anonymous namespace)::Worker*>(string const&, string const&, void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:165:12
#3 0x7ffa79300189 in yb::rpc::(anonymous namespace)::Worker::Start(unsigned long) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:61:12
#4 0x7ffa79300189 in yb::rpc::ThreadPool::Impl::Enqueue(yb::rpc::ThreadPoolTask*) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:201:35
#5 0x7ffa792cc847 in yb::rpc::ServicePoolImpl::Enqueue(std::shared_ptr<yb::rpc::InboundCall> const&) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:196:18
#6 0x7ffa7918dd5b in yb::rpc::Messenger::Handle(std::shared_ptr<yb::rpc::InboundCall>, yb::StronglyTypedBool<yb::rpc::Queue_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:516:21
#7 0x7ffa79306a93 in yb::rpc::YBInboundConnectionContext::HandleCall(std::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc:189:38
#8 0x7ffa79307e61 in non-virtual thunk to yb::rpc::YBInboundConnectionContext::HandleCall(std::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc
#9 0x7ffa79107cec in yb::rpc::BinaryCallParser::Parse(std::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, void, void> const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>, std::shared_ptr<yb::MemTracker> const*) ${BUILD_ROOT}/../../src/yb/rpc/binary_call_parser.cc:163:7
#10 0x7ffa793053d9 in yb::rpc::YBInboundConnectionContext::ProcessCalls(std::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, void, void> const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc:154:19
#11 0x7ffa79138305 in yb::rpc::Connection::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/connection.cc:423:27
#12 0x7ffa7922aeb3 in yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc
#13 0x7ffa7922d191 in non-virtual thunk to yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc
#14 0x7ffa792f0d7e in yb::rpc::TcpStream::TryProcessReceived() ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:408:17
#15 0x7ffa792ed8b3 in yb::rpc::TcpStream::ReadHandler() ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:334:31
#16 0x7ffa792ec344 in yb::rpc::TcpStream::Handler(ev::io&, int) ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:276:14
#17 0x7ffa760a96ca in ev_invoke_pending (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/common/lib/libev.so.4+0x86ca)
#18 0x7ffa760aa3c6 in ev_run (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/common/lib/libev.so.4+0x93c6)
#19 0x7ffa791eb6bc in ev::loop_ref::run(int) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/common/include/ev++.h:211:7
#20 0x7ffa791eb6bc in yb::rpc::Reactor::RunThread() ${BUILD_ROOT}/../../src/yb/rpc/reactor.cc:726:9
#21 0x7ffa7735784f in std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#22 0x7ffa7735784f in std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#23 0x7ffa7735784f in yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3
#24 0x7ffa722f01c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: c46c0e44b55ff27501f607770ed2ae993fe0b823)

Thread T66 (TabletServer_re) created by T0 here:
#0 0x5630c5457f6a in pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x7ffa77353e96 in yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15
#2 0x7ffa791ead6e in yb::Status yb::Thread::Create<void (yb::rpc::Reactor::*)(), yb::rpc::Reactor*>(string const&, string const&, void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:165:12
#3 0x7ffa791ead6e in yb::rpc::Reactor::Init() ${BUILD_ROOT}/../../src/yb/rpc/reactor.cc:305:10
#4 0x7ffa791858b1 in yb::rpc::Messenger::Init(yb::rpc::MessengerBuilder const&) ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:616:5
#5 0x7ffa79185105 in yb::rpc::MessengerBuilder::Build() ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:154:3
#6 0x7ffa7c69b18a in yb::server::RpcServerBase::Init() ${BUILD_ROOT}/../../src/yb/server/server_base.cc:304:16
#7 0x7ffa7c6a2e9d in yb::server::RpcAndWebServerBase::Init() ${BUILD_ROOT}/../../src/yb/server/server_base.cc:516:3
#8 0x7ffa85b2f20b in yb::tserver::DbServerBase::Init() ${BUILD_ROOT}/../../src/yb/tserver/db_server_base.cc:47:3
#9 0x7ffa85dce6a6 in yb::tserver::TabletServer::Init() ${BUILD_ROOT}/../../src/yb/tserver/tablet_server.cc:423:3
#10 0x7ffa8a454142 in yb::tserver::MiniTabletServer::Start(yb::StronglyTypedBool<yb::tserver::WaitTabletsBootstrapped_Tag>) ${BUILD_ROOT}/../../src/yb/tserver/mini_tablet_server.cc:136:3
#11 0x7ffa8ab396a3 in yb::MiniCluster::AddTabletServer(yb::tserver::TabletServerOptions const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:376:3
#12 0x7ffa8ab3a3cf in yb::MiniCluster::AddTabletServer() ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:384:10
#13 0x7ffa8ab34b03 in yb::MiniCluster::StartAsync(std::vector<yb::tserver::TabletServerOptions, std::allocator<yb::tserver::TabletServerOptions>> const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:252:9
#14 0x7ffa8ab3acc2 in yb::MiniCluster::Start(std::vector<yb::tserver::TabletServerOptions, std::allocator<yb::tserver::TabletServerOptions>> const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:307:3
#15 0x7ffa8af66686 in yb::client::QLDmlTestBase<yb::MiniCluster>::StartCluster() ${BUILD_ROOT}/../../src/yb/client/ql-dml-test-base.cc:101:3
#16 0x7ffa8af7623a in yb::client::QLDmlTestBase<yb::MiniCluster>::SetUp() ${BUILD_ROOT}/../../src/yb/client/ql-dml-test-base.cc:118:3
#17 0x7ffa8af9265a in yb::client::TransactionTestBase<yb::MiniCluster>::SetUp() ${BUILD_ROOT}/../../src/yb/client/txn-test-base.cc:120:3
#18 0x7ffa8645d69a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10
#19 0x7ffa8645d69a in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14
#20 0x7ffa863f48f2 in testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2669:3
#21 0x7ffa863f7349 in testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2853:11
#22 0x7ffa863f9e97 in testing::TestSuite::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:3012:30
#23 0x7ffa86437b08 in testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:5870:44
#24 0x7ffa86461e09 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10
#25 0x7ffa86461e09 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14
#26 0x7ffa864357b4 in testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:5444:10
#27 0x7ffa88c6b2b5 in RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/gtest/gtest.h:2293:73
#28 0x7ffa88c6b2b5 in main ${BUILD_ROOT}/../../src/yb/util/test_main.cc:110:13
#29 0x7ffa71d43d84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: fd0dc204c2454dadefbca307a1c7e2c2ef2071ac)

Thread T33 (consensus [work) created by T0 here:
#0 0x5630c5457f6a in pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x7ffa77353e96 in yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15
#2 0x7ffa7738b561 in yb::Status yb::Thread::Create<void (yb::ThreadPool::*)(bool), yb::ThreadPool*, bool>(string const&, string const&, void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:171:12
#3 0x7ffa773826b2 in yb::ThreadPool::CreateThreadUnlocked() ${BUILD_ROOT}/../../src/yb/util/threadpool.cc:673:14
#4 0x7ffa7737a4ab in yb::ThreadPool::Init() ${BUILD_ROOT}/../../src/yb/util/threadpool.cc:324:21
#5 0x7ffa7737a0b2 in yb::ThreadPoolBuilder::Build(std::unique_ptr<yb::ThreadPool, std::default_delete<yb::ThreadPool>>*) const ${BUILD_ROOT}/../../src/yb/util/threadpool.cc:110:3
#6 0x7ffa85efaa91 in yb::tserver::TSTabletManager::TSTabletManager(yb::FsManager*, yb::tserver::TabletServer*, yb::MetricRegistry*) ${BUILD_ROOT}/../../src/yb/tserver/ts_tablet_manager.cc:446:3
#7 0x7ffa85dc7fd7 in yb::tserver::TabletServer::TabletServer(yb::tserver::TabletServerOptions const&) ${BUILD_ROOT}/../../src/yb/tserver/tablet_server.cc:309:27
#8 0x7ffa8a4540f9 in yb::tserver::MiniTabletServer::Start(yb::StronglyTypedBool<yb::tserver::WaitTabletsBootstrapped_Tag>) ${BUILD_ROOT}/../../src/yb/tserver/mini_tablet_server.cc:135:44
#9 0x7ffa8ab396a3 in yb::MiniCluster::AddTabletServer(yb::tserver::TabletServerOptions const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:376:3
#10 0x7ffa8ab3a3cf in yb::MiniCluster::AddTabletServer() ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:384:10
#11 0x7ffa8ab34b03 in yb::MiniCluster::StartAsync(std::vector<yb::tserver::TabletServerOptions, std::allocator<yb::tserver::TabletServerOptions>> const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:252:9
#12 0x7ffa8ab3acc2 in yb::MiniCluster::Start(std::vector<yb::tserver::TabletServerOptions, std::allocator<yb::tserver::TabletServerOptions>> const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:307:3
#13 0x7ffa8af66686 in yb::client::QLDmlTestBase<yb::MiniCluster>::StartCluster() ${BUILD_ROOT}/../../src/yb/client/ql-dml-test-base.cc:101:3
#14 0x7ffa8af7623a in yb::client::QLDmlTestBase<yb::MiniCluster>::SetUp() ${BUILD_ROOT}/../../src/yb/client/ql-dml-test-base.cc:118:3
#15 0x7ffa8af9265a in yb::client::TransactionTestBase<yb::MiniCluster>::SetUp() ${BUILD_ROOT}/../../src/yb/client/txn-test-base.cc:120:3
#16 0x7ffa8645d69a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10
#17 0x7ffa8645d69a in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14
#18 0x7ffa863f48f2 in testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2669:3
#19 0x7ffa863f7349 in testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2853:11
#20 0x7ffa863f9e97 in testing::TestSuite::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:3012:30
#21 0x7ffa86437b08 in testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:5870:44
#22 0x7ffa86461e09 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10
#23 0x7ffa86461e09 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14
#24 0x7ffa864357b4 in testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:5444:10
#25 0x7ffa88c6b2b5 in RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/gtest/gtest.h:2293:73
#26 0x7ffa88c6b2b5 in main ${BUILD_ROOT}/../../src/yb/util/test_main.cc:110:13
#27 0x7ffa71d43d84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: fd0dc204c2454dadefbca307a1c7e2c2ef2071ac)

Thread T86 (rpc_tp_TabletSe) created by T67 (TabletServer_re) here:
#0 0x5630c5457f6a in pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x7ffa77353e96 in yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15
#2 0x7ffa79300189 in yb::Status yb::Thread::Create<void (yb::rpc::(anonymous namespace)::Worker::*)(), yb::rpc::(anonymous namespace)::Worker*>(string const&, string const&, void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:165:12
#3 0x7ffa79300189 in yb::rpc::(anonymous namespace)::Worker::Start(unsigned long) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:61:12
#4 0x7ffa79300189 in yb::rpc::ThreadPool::Impl::Enqueue(yb::rpc::ThreadPoolTask*) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:201:35
#5 0x7ffa791bb5fb in yb::rpc::OutboundCall::InvokeCallback(std::optional<std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>>) ${BUILD_ROOT}/../../src/yb/rpc/outbound_call.cc:428:32
#6 0x7ffa791bd5c7 in yb::rpc::OutboundCall::SetResponse(yb::rpc::CallResponse&&) ${BUILD_ROOT}/../../src/yb/rpc/outbound_call.cc:550:7
#7 0x7ffa79139d22 in yb::rpc::Connection::HandleCallResponse(yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/connection.cc:464:9
#8 0x7ffa79310c68 in yb::rpc::YBOutboundConnectionContext::HandleCall(std::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc:452:22
#9 0x7ffa79310c68 in non-virtual thunk to yb::rpc::YBOutboundConnectionContext::HandleCall(std::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc
#10 0x7ffa79107cec in yb::rpc::BinaryCallParser::Parse(std::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, void, void> const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>, std::shared_ptr<yb::MemTracker> const*) ${BUILD_ROOT}/../../src/yb/rpc/binary_call_parser.cc:163:7
#11 0x7ffa79311537 in yb::rpc::YBOutboundConnectionContext::ProcessCalls(std::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, void, void> const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc:473:19
#12 0x7ffa79138305 in yb::rpc::Connection::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/connection.cc:423:27
#13 0x7ffa7922aeb3 in yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc
#14 0x7ffa7922d191 in non-virtual thunk to yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc
#15 0x7ffa792f0d7e in yb::rpc::TcpStream::TryProcessReceived() ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:408:17
#16 0x7ffa792ed8b3 in yb::rpc::TcpStream::ReadHandler() ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:334:31
#17 0x7ffa792ec344 in yb::rpc::TcpStream::Handler(ev::io&, int) ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:276:14
#18 0x7ffa760a96ca in ev_invoke_pending (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/common/lib/libev.so.4+0x86ca)
#19 0x7ffa760aa3c6 in ev_run (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/common/lib/libev.so.4+0x93c6)
#20 0x7ffa791eb6bc in ev::loop_ref::run(int) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/common/include/ev++.h:211:7
#21 0x7ffa791eb6bc in yb::rpc::Reactor::RunThread() ${BUILD_ROOT}/../../src/yb/rpc/reactor.cc:726:9
#22 0x7ffa7735784f in std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:510:16
#23 0x7ffa7735784f in std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__functional/function.h:1156:12
#24 0x7ffa7735784f in yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3
#25 0x7ffa722f01c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: c46c0e44b55ff27501f607770ed2ae993fe0b823)

Thread T67 (TabletServer_re) created by T0 here:
#0 0x5630c5457f6a in pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-3-1695085546-1e6329f4-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x7ffa77353e96 in yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15
#2 0x7ffa791ead6e in yb::Status yb::Thread::Create<void (yb::rpc::Reactor::*)(), yb::rpc::Reactor*>(string const&, string const&, void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:165:12
#3 0x7ffa791ead6e in yb::rpc::Reactor::Init() ${BUILD_ROOT}/../../src/yb/rpc/reactor.cc:305:10
#4 0x7ffa791858b1 in yb::rpc::Messenger::Init(yb::rpc::MessengerBuilder const&) ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:616:5
#5 0x7ffa79185105 in yb::rpc::MessengerBuilder::Build() ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:154:3
#6 0x7ffa7c69b18a in yb::server::RpcServerBase::Init() ${BUILD_ROOT}/../../src/yb/server/server_base.cc:304:16
#7 0x7ffa7c6a2e9d in yb::server::RpcAndWebServerBase::Init() ${BUILD_ROOT}/../../src/yb/server/server_base.cc:516:3
#8 0x7ffa85b2f20b in yb::tserver::DbServerBase::Init() ${BUILD_ROOT}/../../src/yb/tserver/db_server_base.cc:47:3
#9 0x7ffa85dce6a6 in yb::tserver::TabletServer::Init() ${BUILD_ROOT}/../../src/yb/tserver/tablet_server.cc:423:3
#10 0x7ffa8a454142 in yb::tserver::MiniTabletServer::Start(yb::StronglyTypedBool<yb::tserver::WaitTabletsBootstrapped_Tag>) ${BUILD_ROOT}/../../src/yb/tserver/mini_tablet_server.cc:136:3
#11 0x7ffa8ab396a3 in yb::MiniCluster::AddTabletServer(yb::tserver::TabletServerOptions const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:376:3
#12 0x7ffa8ab3a3cf in yb::MiniCluster::AddTabletServer() ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:384:10
#13 0x7ffa8ab34b03 in yb::MiniCluster::StartAsync(std::vector<yb::tserver::TabletServerOptions, std::allocator<yb::tserver::TabletServerOptions>> const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:252:9
#14 0x7ffa8ab3acc2 in yb::MiniCluster::Start(std::vector<yb::tserver::TabletServerOptions, std::allocator<yb::tserver::TabletServerOptions>> const&) ${BUILD_ROOT}/../../src/yb/integration-tests/mini_cluster.cc:307:3
#15 0x7ffa8af66686 in yb::client::QLDmlTestBase<yb::MiniCluster>::StartCluster() ${BUILD_ROOT}/../../src/yb/client/ql-dml-test-base.cc:101:3
#16 0x7ffa8af7623a in yb::client::QLDmlTestBase<yb::MiniCluster>::SetUp() ${BUILD_ROOT}/../../src/yb/client/ql-dml-test-base.cc:118:3
#17 0x7ffa8af9265a in yb::client::TransactionTestBase<yb::MiniCluster>::SetUp() ${BUILD_ROOT}/../../src/yb/client/txn-test-base.cc:120:3
#18 0x7ffa8645d69a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10
#19 0x7ffa8645d69a in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14
#20 0x7ffa863f48f2 in testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2669:3
#21 0x7ffa863f7349 in testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2853:11
#22 0x7ffa863f9e97 in testing::TestSuite::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:3012:30
#23 0x7ffa86437b08 in testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:5870:44
#24 0x7ffa86461e09 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10
#25 0x7ffa86461e09 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14
#26 0x7ffa864357b4 in testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/src/googletest-1.12.1/googletest/src/gtest.cc:5444:10
#27 0x7ffa88c6b2b5 in RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/include/gtest/gtest.h:2293:73
#28 0x7ffa88c6b2b5 in main ${BUILD_ROOT}/../../src/yb/util/test_main.cc:110:13
#29 0x7ffa71d43d84 in __libc_start_main (/lib64/libc.so.6+0x3ad84) (BuildId: fd0dc204c2454dadefbca307a1c7e2c2ef2071ac)

SUMMARY: AddressSanitizer: heap-use-after-free /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231011232419-9646ccedb1-almalinux8-x86_64-clang16/installed/asan/libcxx/include/c++/v1/__memory/shared_ptr.h:857:16 in std::shared_ptr<yb::SubtxnSetAndPB const>::operator->[abi:v160006]() const
Shadow bytes around the buggy address:
0x614000382080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x614000382100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x614000382180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa
0x614000382200: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x614000382280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x614000382300: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x614000382380: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
0x614000382400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000382480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000382500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000382580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==147991==ABORTING

--------------------------------------------------------------------------------

Rewrites performed by the rewrite_test_log.py script:

test_host_name = spark-workers-b-alma8-2wzh
m1_peer_id = 85538e41ba75456da87049ecaeea8c56
ts1_peer_id = fcf85440f47d4f7492cbb057903a3cfc
ts2_peer_id = 7e4be1b721384eb29d4d04a021fdc1e3
ts3_peer_id = 01770aa8726b49718163f1c77bd88efa
ql_client_test_table_tablet_id1 = 2ab48c4a3add47cc8974270fe0b59a2f
ql_client_test_table_tablet_id2 = e5794eda27084e5daa4c9d282bf0b77e
ql_client_test_table_tablet_id3 = e8ff2cf826fe44229aac9fea5a424c53
transactions_tablet_id1 = 33a54f767cd44c7592b8f57635db2245
transactions_tablet_id2 = 50e89f44718a448d9b2ff86ac76e683b
transactions_tablet_id3 = 617aac87fac04f5797867643dcea6346
sys_catalog_tablet_id = 00000000000000000000000000000000
BUILD_ROOT = /share/jenkins/jenkins-github-yugabyte-db-alma8-master-clang16-asan-753/yugabyte-db/build/asan-clang16-dynamic-ninja
YB_SRC_ROOT = /share/jenkins/jenkins-github-yugabyte-db-alma8-master-clang16-asan-753/yugabyte-db
TEST_TMPDIR = /tmp/yb_test.tests-client__seal-txn-test___SealTxnTest.Update.20231017085019681229.9002192110

--------------------------------------------------------------------------------

@rthallamko3 rthallamko3 assigned yusong-yan and unassigned robertsami Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yusong-yan yusong-yan

Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug kind/failing-test Tests and testing infra priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@bmatican @spolitov @robertsami @rthallamko3 @yugabyte-ci @yusong-yan