Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][CI] ubuntu-cpp-threadsanitizer job fails consistently with a data race on asof_join_node_test #44846

Closed
raulcd opened this issue Nov 25, 2024 · 1 comment
Closed

[C++][CI] ubuntu-cpp-threadsanitizer job fails consistently with a data race on asof_join_node_test #44846

raulcd opened this issue Nov 25, 2024 · 1 comment
Assignees
@raulcd
Labels
Component: C++ Component: Continuous Integration Type: bug
Milestone
19.0.0

Comments

@raulcd
Copy link
Member

raulcd commented Nov 25, 2024

Describe the bug, including details regarding any error messages, version, and platform.

The failure started since these commits were merged: 27bbd59...0cdbdac
They seem harmless to me.
The failure:

See parts of the failure log here:

 ==================
WARNING: ThreadSanitizer: data race (pid=29580)
  Write of size 8 at 0x725400001938 by thread T10 (mutexes: write M0):
    #0 std::_Deque_iterator<std::shared_ptr<arrow::RecordBatch>, std::shared_ptr<arrow::RecordBatch>&, std::shared_ptr<arrow::RecordBatch>*>::_M_set_node(std::shared_ptr<arrow::RecordBatch>**) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_deque.h:265:10 (libarrow_acero.so.1900+0x362b91) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #1 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>::_M_reallocate_map(unsigned long, bool) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/deque.tcc:971:30 (libarrow_acero.so.1900+0x369fa6) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #2 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>::_M_reserve_map_at_back(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_deque.h:2172:4 (libarrow_acero.so.1900+0x369b04) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #3 void std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>::_M_push_back_aux<std::shared_ptr<arrow::RecordBatch> const&>(std::shared_ptr<arrow::RecordBatch> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/deque.tcc:496:2 (libarrow_acero.so.1900+0x3698e4) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #4 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>::push_back(std::shared_ptr<arrow::RecordBatch> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_deque.h:1548:4 (libarrow_acero.so.1900+0x36983f) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #5 std::queue<std::shared_ptr<arrow::RecordBatch>, std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>>::push(std::shared_ptr<arrow::RecordBatch> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_queue.h:286:11 (libarrow_acero.so.1900+0x369725) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #6 arrow::acero::ConcurrentQueue<std::shared_ptr<arrow::RecordBatch>>::PushUnlocked(std::shared_ptr<arrow::RecordBatch> const&) /arrow/cpp/src/arrow/acero/concurrent_queue_internal.h:86:12 (libarrow_acero.so.1900+0x3693d9) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #7 arrow::acero::BackpressureConcurrentQueue<std::shared_ptr<arrow::RecordBatch>>::Push(std::shared_ptr<arrow::RecordBatch> const&) /arrow/cpp/src/arrow/acero/concurrent_queue_internal.h:140:25 (libarrow_acero.so.1900+0x369125) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #8 arrow::acero::InputState::Push(std::shared_ptr<arrow::RecordBatch> const&) /arrow/cpp/src/arrow/acero/asof_join_node.cc:735:14 (libarrow_acero.so.1900+0x367eac) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #9 arrow::acero::InputState::Process(arrow::compute::ExecBatch) /arrow/cpp/src/arrow/acero/asof_join_node.cc:712:12 (libarrow_acero.so.1900+0x362178) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #10 arrow::acero::util::(anonymous namespace)::SerialSequencingQueueImpl::DoProcess(std::unique_lock<std::mutex>&&) /arrow/cpp/src/arrow/acero/accumulation_queue.cc:148:7 (libarrow_acero.so.1900+0x2cd8ef) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #11 arrow::acero::util::(anonymous namespace)::SerialSequencingQueueImpl::InsertBatch(arrow::compute::ExecBatch) /arrow/cpp/src/arrow/acero/accumulation_queue.cc:133:14 (libarrow_acero.so.1900+0x2cd5b4) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 arrow::acero::InputState::InsertBatch(arrow::compute::ExecBatch) /arrow/cpp/src/arrow/acero/asof_join_node.cc:705:24 (libarrow_acero.so.1900+0x35afc6) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #13 arrow::acero::AsofJoinNode::InputReceived(arrow::acero::ExecNode*, arrow::compute::ExecBatch) /arrow/cpp/src/arrow/acero/asof_join_node.cc:1431:5 (libarrow_acero.so.1900+0x33f9dc) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #14 arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()::operator()() const /arrow/cpp/src/arrow/acero/source_node.cc:158:13 (libarrow_acero.so.1900+0x516eb9) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #15 arrow::Status std::__invoke_impl<arrow::Status, arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()&>(std::__invoke_other, arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow_acero.so.1900+0x516ae5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #16 std::enable_if<is_invocable_r_v<arrow::Status, arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()&>, arrow::Status>::type std::__invoke_r<arrow::Status, arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()&>(arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114:9 (libarrow_acero.so.1900+0x516a35) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #17 std::_Function_handler<arrow::Status (), arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&)::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9 (libarrow_acero.so.1900+0x516711) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #18 std::function<arrow::Status ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9 (libarrow_acero.so.1900+0x2d35eb) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #19 std::enable_if<!std::is_void<arrow::Status>::value && !is_future<arrow::Status>::value && (!arrow::Future<arrow::internal::Empty>::is_empty || std::is_same<arrow::Status, arrow::Status>::value), void>::type arrow::detail::ContinueFuture::operator()<std::function<arrow::Status ()>&, arrow::Status, arrow::Future<arrow::internal::Empty>>(arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>&) const /arrow/cpp/src/arrow/util/future.h:150:23 (libarrow_acero.so.1900+0x4d9f71) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #20 void std::__invoke_impl<void, arrow::detail::ContinueFuture&, arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status ()>&>(std::__invoke_other, arrow::detail::ContinueFuture&, arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status ()>&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow_acero.so.1900+0x4d9ede) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #21 std::__invoke_result<arrow::detail::ContinueFuture&, arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status ()>&>::type std::__invoke<arrow::detail::ContinueFuture&, arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status ()>&>(arrow::detail::ContinueFuture&, arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status ()>&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 (libarrow_acero.so.1900+0x4d9d95) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #22 void std::_Bind<arrow::detail::ContinueFuture (arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11 (libarrow_acero.so.1900+0x4d9d1a) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #23 void std::_Bind<arrow::detail::ContinueFuture (arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>)>::operator()<void>() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17 (libarrow_acero.so.1900+0x4d9c76) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #24 arrow::internal::FnOnce<void ()>::FnImpl<std::_Bind<arrow::detail::ContinueFuture (arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>)>>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow_acero.so.1900+0x4d9c19) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #25 arrow::internal::FnOnce<void ()>::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x435d4df) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #26 arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce<void ()>, arrow::StopToken, arrow::internal::FnOnce<void (arrow::Status const&)>&&)::$_0::operator()() /arrow/cpp/src/arrow/util/thread_pool.cc:634:9 (libarrow.so.1900+0x435d188) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #27 arrow::internal::FnOnce<void ()>::FnImpl<arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce<void ()>, arrow::StopToken, arrow::internal::FnOnce<void (arrow::Status const&)>&&)::$_0>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow.so.1900+0x435d119) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #28 arrow::internal::FnOnce<void ()>::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x435d4df) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #29 arrow::internal::WorkerLoop(std::shared_ptr<arrow::internal::ThreadPool::State>, std::_List_iterator<std::thread>) /arrow/cpp/src/arrow/util/thread_pool.cc:457:11 (libarrow.so.1900+0x435c935) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #30 arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0::operator()() const /arrow/cpp/src/arrow/util/thread_pool.cc:618:7 (libarrow.so.1900+0x435c583) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #31 void std::__invoke_impl<void, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>(std::__invoke_other, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow.so.1900+0x435c4c5) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #32 std::__invoke_result<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>::type std::__invoke<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>(arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 (libarrow.so.1900+0x435c435) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #33 void std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 (libarrow.so.1900+0x435c3ed) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #34 std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 (libarrow.so.1900+0x435c395) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #35 std::thread::_State_impl<std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>>::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 (libarrow.so.1900+0x435c0b9) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #36 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Previous read of size 8 at 0x725400001938 by thread T7 (mutexes: write M1):
    #0 std::_Deque_iterator<std::shared_ptr<arrow::RecordBatch>, std::shared_ptr<arrow::RecordBatch> const&, std::shared_ptr<arrow::RecordBatch> const*>::_Deque_iterator<std::_Deque_iterator<std::shared_ptr<arrow::RecordBatch>, std::shared_ptr<arrow::RecordBatch>&, std::shared_ptr<arrow::RecordBatch>*>, void>(std::_Deque_iterator<std::shared_ptr<arrow::RecordBatch>, std::shared_ptr<arrow::RecordBatch>&, std::shared_ptr<arrow::RecordBatch>*> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_deque.h:166:37 (libarrow_acero.so.1900+0x3804cc) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #1 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>::begin() const /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_deque.h:1161:16 (libarrow_acero.so.1900+0x380379) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #2 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>::front() const /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_deque.h:1459:10 (libarrow_acero.so.1900+0x380319) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #3 std::queue<std::shared_ptr<arrow::RecordBatch>, std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch>>>>::front() const /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/stl_queue.h:248:11 (libarrow_acero.so.1900+0x3802d5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #4 arrow::acero::ConcurrentQueue<std::shared_ptr<arrow::RecordBatch>>::UnsyncFront() const /arrow/cpp/src/arrow/acero/concurrent_queue_internal.h:72:48 (libarrow_acero.so.1900+0x380295) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #5 arrow::acero::InputState::GetLatestBatch() const /arrow/cpp/src/arrow/acero/asof_join_node.cc:570:19 (libarrow_acero.so.1900+0x37af39) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #6 arrow::acero::InputState::GetLatestTime() const /arrow/cpp/src/arrow/acero/asof_join_node.cc:616:20 (libarrow_acero.so.1900+0x373cf9) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #7 arrow::acero::AsofJoinNode::ProcessInner() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1014:43 (libarrow_acero.so.1900+0x371706) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #8 arrow::acero::AsofJoinNode::Process() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1072:53 (libarrow_acero.so.1900+0x36da15) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #9 arrow::acero::AsofJoinNode::ProcessThread() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1111:12 (libarrow_acero.so.1900+0x36d73c) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #10 arrow::acero::AsofJoinNode::ProcessThreadWrapper(arrow::acero::AsofJoinNode*) /arrow/cpp/src/arrow/acero/asof_join_node.cc:1117:64 (libarrow_acero.so.1900+0x36d0a5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #11 void std::__invoke_impl<void, void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>(std::__invoke_other, void (*&&)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow_acero.so.1900+0x3ad96a) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 std::__invoke_result<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>::type std::__invoke<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>(void (*&&)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 (libarrow_acero.so.1900+0x3ad865) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #13 void std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 (libarrow_acero.so.1900+0x3ad802) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #14 std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 (libarrow_acero.so.1900+0x3ad795) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #15 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>>::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 (libarrow_acero.so.1900+0x3ad539) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #16 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Location is heap block of size 616 at 0x725400001900 allocated by main thread:
    #0 operator new(unsigned long) <null> (arrow-acero-asof-join-node-test+0x1d3dfb) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #1 std::__detail::_MakeUniq<arrow::acero::InputState>::__single_object std::make_unique<arrow::acero::InputState, unsigned long&, arrow::acero::TolType&, bool&, bool&, arrow::acero::KeyHasher*&, arrow::acero::AsofJoinNode*&, arrow::acero::BackpressureHandler, std::shared_ptr<arrow::Schema> const&, int const&, std::vector<int, std::allocator<int>> const&>(unsigned long&, arrow::acero::TolType&, bool&, bool&, arrow::acero::KeyHasher*&, arrow::acero::AsofJoinNode*&, arrow::acero::BackpressureHandler&&, std::shared_ptr<arrow::Schema> const&, int const&, std::vector<int, std::allocator<int>> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:1070:30 (libarrow_acero.so.1900+0x35ef0b) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #2 arrow::acero::InputState::Make(unsigned long, arrow::acero::TolType, bool, bool, arrow::acero::KeyHasher*, arrow::acero::ExecNode*, arrow::acero::AsofJoinNode*, std::atomic<int>&, std::shared_ptr<arrow::Schema> const&, int, std::vector<int, std::allocator<int>> const&) /arrow/cpp/src/arrow/acero/asof_join_node.cc:518:12 (libarrow_acero.so.1900+0x35de5d) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #3 arrow::acero::AsofJoinNode::Init() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1133:7 (libarrow_acero.so.1900+0x340186) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #4 arrow::acero::(anonymous namespace)::ExecPlanImpl::StartProducing()::'lambda'(arrow::util::AsyncTaskScheduler*)::operator()(arrow::util::AsyncTaskScheduler*) const /arrow/cpp/src/arrow/acero/exec_plan.cc:145:13 (libarrow_acero.so.1900+0x3c8804) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #5 arrow::internal::FnOnce<arrow::Status (arrow::util::AsyncTaskScheduler*)>::FnImpl<arrow::acero::(anonymous namespace)::ExecPlanImpl::StartProducing()::'lambda'(arrow::util::AsyncTaskScheduler*)>::invoke(arrow::util::AsyncTaskScheduler*&&) /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow_acero.so.1900+0x3c83e5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #6 arrow::internal::FnOnce<arrow::Status (arrow::util::AsyncTaskScheduler*)>::operator()(arrow::util::AsyncTaskScheduler*) && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x42016c0) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #7 arrow::util::AsyncTaskScheduler::Make(arrow::internal::FnOnce<arrow::Status (arrow::util::AsyncTaskScheduler*)>, arrow::internal::FnOnce<void (arrow::Status const&)>, arrow::StopToken) /arrow/cpp/src/arrow/util/async_util.cc:471:28 (libarrow.so.1900+0x41f177c) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #8 arrow::acero::(anonymous namespace)::ExecPlanImpl::StartProducing() /arrow/cpp/src/arrow/acero/exec_plan.cc:128:35 (libarrow_acero.so.1900+0x3bd594) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #9 arrow::acero::ExecPlan::StartProducing() /arrow/cpp/src/arrow/acero/exec_plan.cc:439:59 (libarrow_acero.so.1900+0x3bd0b7) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #10 arrow::acero::(anonymous namespace)::DeclarationToExecBatchesImpl(arrow::acero::Declaration, arrow::acero::QueryOptions, arrow::internal::Executor*) /arrow/cpp/src/arrow/acero/exec_plan.cc:692:14 (libarrow_acero.so.1900+0x3c44fa) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #11 arrow::acero::DeclarationToExecBatches(arrow::acero::Declaration, arrow::acero::QueryOptions)::$_0::operator()(arrow::internal::Executor*) /arrow/cpp/src/arrow/acero/exec_plan.cc:896:16 (libarrow_acero.so.1900+0x3d4b5b) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 arrow::internal::FnOnce<arrow::Future<arrow::acero::BatchesWithCommonSchema> (arrow::internal::Executor*)>::FnImpl<arrow::acero::DeclarationToExecBatches(arrow::acero::Declaration, arrow::acero::QueryOptions)::$_0>::invoke(arrow::internal::Executor*&&) /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow_acero.so.1900+0x3d4a95) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #13 arrow::internal::FnOnce<arrow::Future<arrow::acero::BatchesWithCommonSchema> (arrow::internal::Executor*)>::operator()(arrow::internal::Executor*) && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow_acero.so.1900+0x414840) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #14 arrow::Future<arrow::acero::BatchesWithCommonSchema>::SyncType arrow::internal::RunSynchronously<arrow::Future<arrow::acero::BatchesWithCommonSchema>, arrow::acero::BatchesWithCommonSchema>(arrow::internal::FnOnce<arrow::Future<arrow::acero::BatchesWithCommonSchema> (arrow::internal::Executor*)>, bool) /arrow/cpp/src/arrow/util/thread_pool.h:587:16 (libarrow_acero.so.1900+0x3df195) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #15 arrow::acero::DeclarationToExecBatches(arrow::acero::Declaration, arrow::acero::QueryOptions) /arrow/cpp/src/arrow/acero/exec_plan.cc:893:10 (libarrow_acero.so.1900+0x3c51f0) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #16 void arrow::acero::TestSequencing<arrow::Result<arrow::acero::BatchesWithSchema> (*)(std::vector<std::function<long (int)>, std::allocator<std::function<long (int)>>> const&, std::shared_ptr<arrow::Schema> const&, int, int)>(arrow::Result<arrow::acero::BatchesWithSchema> (*)(std::vector<std::function<long (int)>, std::allocator<std::function<long (int)>>> const&, std::shared_ptr<arrow::Schema> const&, int, int), int, int) /arrow/cpp/src/arrow/acero/asof_join_node_test.cc:1579:3 (arrow-acero-asof-join-node-test+0x297083) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #17 arrow::acero::AsofJoinTest_BatchSequencing_Test::TestBody() /arrow/cpp/src/arrow/acero/asof_join_node_test.cc:1586:10 (arrow-acero-asof-join-node-test+0x27d8a2) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #18 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (libarrow_testing.so.1900+0x48ec8e) (BuildId: 113acbd90ed4ee9f7f986424c1901035aee7411d)

  Mutex M0 (0x725400001960) created at:
    #0 pthread_mutex_lock <null> (arrow-acero-asof-join-node-test+0x1563ae) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/x86_64-linux-gnu/c++/13/bits/gthr-default.h:749:12 (arrow-acero-asof-join-node-test+0x1d5ac3) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #2 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_mutex.h:113:17 (arrow-acero-asof-join-node-test+0x1e4a85) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #3 std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_lock.h:141:17 (arrow-acero-asof-join-node-test+0x1fed45) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_lock.h:71:2 (arrow-acero-asof-join-node-test+0x1feb6e) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #5 arrow::acero::ConcurrentQueue<std::shared_ptr<arrow::RecordBatch>>::Empty() const /arrow/cpp/src/arrow/acero/concurrent_queue_internal.h:64:34 (libarrow_acero.so.1900+0x379581) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #6 arrow::acero::InputState::Empty() const /arrow/cpp/src/arrow/acero/asof_join_node.cc:547:19 (libarrow_acero.so.1900+0x372468) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #7 arrow::acero::AsofJoinNode::ProcessInner() /arrow/cpp/src/arrow/acero/asof_join_node.cc:993:33 (libarrow_acero.so.1900+0x371357) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #8 arrow::acero::AsofJoinNode::Process() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1072:53 (libarrow_acero.so.1900+0x36da15) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #9 arrow::acero::AsofJoinNode::ProcessThread() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1111:12 (libarrow_acero.so.1900+0x36d73c) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #10 arrow::acero::AsofJoinNode::ProcessThreadWrapper(arrow::acero::AsofJoinNode*) /arrow/cpp/src/arrow/acero/asof_join_node.cc:1117:64 (libarrow_acero.so.1900+0x36d0a5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #11 void std::__invoke_impl<void, void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>(std::__invoke_other, void (*&&)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow_acero.so.1900+0x3ad96a) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 std::__invoke_result<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>::type std::__invoke<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>(void (*&&)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 (libarrow_acero.so.1900+0x3ad865) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #13 void std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 (libarrow_acero.so.1900+0x3ad802) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #14 std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 (libarrow_acero.so.1900+0x3ad795) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #15 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>>::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 (libarrow_acero.so.1900+0x3ad539) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #16 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Mutex M1 (0x725400001788) created at:
    #0 pthread_mutex_lock <null> (arrow-acero-asof-join-node-test+0x1563ae) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #1 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/x86_64-linux-gnu/c++/13/bits/gthr-default.h:749:12 (arrow-acero-asof-join-node-test+0x1d5ac3) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #2 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_mutex.h:113:17 (arrow-acero-asof-join-node-test+0x1e4a85) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_mutex.h:249:19 (arrow-acero-asof-join-node-test+0x1e48d9) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #4 arrow::acero::AsofJoinNode::Process() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1065:33 (libarrow_acero.so.1900+0x36d9b0) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #5 arrow::acero::AsofJoinNode::ProcessThread() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1111:12 (libarrow_acero.so.1900+0x36d73c) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #6 arrow::acero::AsofJoinNode::ProcessThreadWrapper(arrow::acero::AsofJoinNode*) /arrow/cpp/src/arrow/acero/asof_join_node.cc:1117:64 (libarrow_acero.so.1900+0x36d0a5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #7 void std::__invoke_impl<void, void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>(std::__invoke_other, void (*&&)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow_acero.so.1900+0x3ad96a) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #8 std::__invoke_result<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>::type std::__invoke<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>(void (*&&)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 (libarrow_acero.so.1900+0x3ad865) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #9 void std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 (libarrow_acero.so.1900+0x3ad802) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #10 std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 (libarrow_acero.so.1900+0x3ad795) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(arrow::acero::AsofJoinNode*), arrow::acero::AsofJoinNode*>>>::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 (libarrow_acero.so.1900+0x3ad539) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Thread T10 (tid=29623, running) created by thread T8 at:
    #0 pthread_create <null> (arrow-acero-asof-join-node-test+0x15487f) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) <null> (libstdc++.so.6+0xeceb0) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
    #2 arrow::internal::ThreadPool::LaunchWorkersUnlocked(int) /arrow/cpp/src/arrow/util/thread_pool.cc:616:11 (libarrow.so.1900+0x4355a22) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #3 arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce<void ()>, arrow::StopToken, arrow::internal::FnOnce<void (arrow::Status const&)>&&) /arrow/cpp/src/arrow/util/thread_pool.cc:651:7 (libarrow.so.1900+0x43562ae) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #4 arrow::Result<arrow::Future<arrow::internal::Empty>> arrow::internal::Executor::Submit<std::function<arrow::Status ()>, arrow::Future<arrow::internal::Empty>>(arrow::internal::TaskHints, arrow::StopToken, std::function<arrow::Status ()>&&) /arrow/cpp/src/arrow/util/thread_pool.h:161:5 (libarrow_acero.so.1900+0x4d8e46) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #5 arrow::Result<arrow::Future<arrow::internal::Empty>> arrow::internal::Executor::Submit<std::function<arrow::Status ()>, arrow::Future<arrow::internal::Empty>>(std::function<arrow::Status ()>&&) /arrow/cpp/src/arrow/util/thread_pool.h:187:12 (libarrow_acero.so.1900+0x4d8c27) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #6 arrow::acero::QueryContext::ScheduleTask(std::function<arrow::Status ()>, std::basic_string_view<char, std::char_traits<char>>)::$_0::operator()() /arrow/cpp/src/arrow/acero/query_context.cc:66:59 (libarrow_acero.so.1900+0x4d654d) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #7 arrow::util::AsyncTaskScheduler::SimpleTask<arrow::acero::QueryContext::ScheduleTask(std::function<arrow::Status ()>, std::basic_string_view<char, std::char_traits<char>>)::$_0>::operator()() /arrow/cpp/src/arrow/util/async_util.h:153:53 (libarrow_acero.so.1900+0x4d6479) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #8 arrow::util::(anonymous namespace)::AsyncTaskSchedulerImpl::DoSubmitTask(std::unique_ptr<arrow::util::AsyncTaskScheduler::Task, std::default_delete<arrow::util::AsyncTaskScheduler::Task>>) /arrow/cpp/src/arrow/util/async_util.cc:197:38 (libarrow.so.1900+0x41f4181) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #9 arrow::util::(anonymous namespace)::AsyncTaskSchedulerImpl::SubmitTaskUnlocked(std::unique_ptr<arrow::util::AsyncTaskScheduler::Task, std::default_delete<arrow::util::AsyncTaskScheduler::Task>>, std::unique_lock<std::mutex>&&) /arrow/cpp/src/arrow/util/async_util.cc:267:12 (libarrow.so.1900+0x41f3c04) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #10 arrow::util::(anonymous namespace)::AsyncTaskSchedulerImpl::AddTask(std::unique_ptr<arrow::util::AsyncTaskScheduler::Task, std::default_delete<arrow::util::AsyncTaskScheduler::Task>>) /arrow/cpp/src/arrow/util/async_util.cc:173:5 (libarrow.so.1900+0x41f3a07) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #11 bool arrow::util::AsyncTaskScheduler::AddSimpleTask<arrow::acero::QueryContext::ScheduleTask(std::function<arrow::Status ()>, std::basic_string_view<char, std::char_traits<char>>)::$_0>(arrow::acero::QueryContext::ScheduleTask(std::function<arrow::Status ()>, std::basic_string_view<char, std::char_traits<char>>)::$_0, std::basic_string_view<char, std::char_traits<char>>) /arrow/cpp/src/arrow/util/async_util.h:171:12 (libarrow_acero.so.1900+0x4d4a38) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 arrow::acero::QueryContext::ScheduleTask(std::function<arrow::Status ()>, std::basic_string_view<char, std::char_traits<char>>) /arrow/cpp/src/arrow/acero/query_context.cc:65:21 (libarrow_acero.so.1900+0x4d492d) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #13 arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(arrow::compute::ExecBatch const&) /arrow/cpp/src/arrow/acero/source_node.cc:134:29 (libarrow_acero.so.1900+0x5162e9) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #14 arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&)::operator()(std::optional<arrow::compute::ExecBatch> const&) const /arrow/cpp/src/arrow/acero/source_node.cc:210:13 (libarrow_acero.so.1900+0x515e65) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #15 std::enable_if<is_future<arrow::Future<std::optional<int>>>::value, void>::type arrow::detail::ContinueFuture::operator()<arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&), std::optional<arrow::compute::ExecBatch> const&, arrow::Future<std::optional<int>>, arrow::Future<std::optional<int>>>(arrow::Future<std::optional<int>>, arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&)&&, std::optional<arrow::compute::ExecBatch> const&) const /arrow/cpp/src/arrow/util/future.h:179:9 (libarrow_acero.so.1900+0x515c00) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #16 void arrow::detail::ContinueFuture::IgnoringArgsIf<arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&), arrow::Future<std::optional<int>>, std::optional<arrow::compute::ExecBatch> const&>(std::integral_constant<bool, false>, arrow::Future<std::optional<int>>&&, arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&)&&, std::optional<arrow::compute::ExecBatch> const&) const /arrow/cpp/src/arrow/util/future.h:193:5 (libarrow_acero.so.1900+0x5159e1) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #17 arrow::Future<std::optional<arrow::compute::ExecBatch>>::ThenOnComplete<arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&), arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(arrow::Status const&)>::operator()(arrow::Result<std::optional<arrow::compute::ExecBatch>> const&) && /arrow/cpp/src/arrow/util/future.h:545:25 (libarrow_acero.so.1900+0x515882) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #18 arrow::Future<std::optional<arrow::compute::ExecBatch>>::WrapResultOnComplete::Callback<arrow::Future<std::optional<arrow::compute::ExecBatch>>::ThenOnComplete<arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&), arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(arrow::Status const&)>>::operator()(arrow::FutureImpl const&) && /arrow/cpp/src/arrow/util/future.h:442:9 (libarrow_acero.so.1900+0x5157c5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #19 arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::FnImpl<arrow::Future<std::optional<arrow::compute::ExecBatch>>::WrapResultOnComplete::Callback<arrow::Future<std::optional<arrow::compute::ExecBatch>>::ThenOnComplete<arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(std::optional<arrow::compute::ExecBatch> const&), arrow::acero::(anonymous namespace)::SourceNode::StartProducing()::'lambda'()::operator()() const::'lambda'(arrow::Status const&)>>>::invoke(arrow::FutureImpl const&) /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow_acero.so.1900+0x515759) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #20 arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::operator()(arrow::FutureImpl const&) && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x42c98e7) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #21 arrow::ConcreteFutureImpl::AddCallback(arrow::internal::FnOnce<void (arrow::FutureImpl const&)>, arrow::CallbackOptions)::'lambda'(arrow::FutureImpl const&)::operator()(arrow::FutureImpl const&) /arrow/cpp/src/arrow/util/future.cc:58:7 (libarrow.so.1900+0x42ca668) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #22 arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::FnImpl<arrow::ConcreteFutureImpl::AddCallback(arrow::internal::FnOnce<void (arrow::FutureImpl const&)>, arrow::CallbackOptions)::'lambda'(arrow::FutureImpl const&)>::invoke(arrow::FutureImpl const&) /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow.so.1900+0x42ca5e9) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #23 arrow::internal::FnOnce<void (arrow::FutureImpl const&)>::operator()(arrow::FutureImpl const&) && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x42c98e7) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #24 arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr<arrow::FutureImpl> const&, arrow::FutureImpl::CallbackRecord&&, bool)::'lambda'()::operator()() /arrow/cpp/src/arrow/util/future.cc:106:16 (libarrow.so.1900+0x42c9c9c) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #25 arrow::internal::FnOnce<void ()>::FnImpl<arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr<arrow::FutureImpl> const&, arrow::FutureImpl::CallbackRecord&&, bool)::'lambda'()>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow.so.1900+0x42c9c39) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #26 arrow::internal::FnOnce<void ()>::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x435d4df) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #27 arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce<void ()>, arrow::StopToken, arrow::internal::FnOnce<void (arrow::Status const&)>&&)::$_0::operator()() /arrow/cpp/src/arrow/util/thread_pool.cc:634:9 (libarrow.so.1900+0x435d188) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #28 arrow::internal::FnOnce<void ()>::FnImpl<arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce<void ()>, arrow::StopToken, arrow::internal::FnOnce<void (arrow::Status const&)>&&)::$_0>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow.so.1900+0x435d119) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #29 arrow::internal::FnOnce<void ()>::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x435d4df) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #30 arrow::internal::WorkerLoop(std::shared_ptr<arrow::internal::ThreadPool::State>, std::_List_iterator<std::thread>) /arrow/cpp/src/arrow/util/thread_pool.cc:457:11 (libarrow.so.1900+0x435c935) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #31 arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0::operator()() const /arrow/cpp/src/arrow/util/thread_pool.cc:618:7 (libarrow.so.1900+0x435c583) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #32 void std::__invoke_impl<void, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>(std::__invoke_other, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14 (libarrow.so.1900+0x435c4c5) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #33 std::__invoke_result<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>::type std::__invoke<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>(arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14 (libarrow.so.1900+0x435c435) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #34 void std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13 (libarrow.so.1900+0x435c3ed) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #35 std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11 (libarrow.so.1900+0x435c395) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #36 std::thread::_State_impl<std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_0>>>::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13 (libarrow.so.1900+0x435c0b9) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #37 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Thread T7 (tid=29620, running) created by main thread at:
    #0 pthread_create <null> (arrow-acero-asof-join-node-test+0x15487f) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) <null> (libstdc++.so.6+0xeceb0) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
    #2 arrow::acero::AsofJoinNode::StartProducing() /arrow/cpp/src/arrow/acero/asof_join_node.cc:1518:23 (libarrow_acero.so.1900+0x3405e9) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #3 arrow::acero::(anonymous namespace)::ExecPlanImpl::StartProducing()::'lambda'(arrow::util::AsyncTaskScheduler*)::operator()(arrow::util::AsyncTaskScheduler*) const /arrow/cpp/src/arrow/acero/exec_plan.cc:175:24 (libarrow_acero.so.1900+0x3c8ca3) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #4 arrow::internal::FnOnce<arrow::Status (arrow::util::AsyncTaskScheduler*)>::FnImpl<arrow::acero::(anonymous namespace)::ExecPlanImpl::StartProducing()::'lambda'(arrow::util::AsyncTaskScheduler*)>::invoke(arrow::util::AsyncTaskScheduler*&&) /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow_acero.so.1900+0x3c83e5) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #5 arrow::internal::FnOnce<arrow::Status (arrow::util::AsyncTaskScheduler*)>::operator()(arrow::util::AsyncTaskScheduler*) && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow.so.1900+0x42016c0) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #6 arrow::util::AsyncTaskScheduler::Make(arrow::internal::FnOnce<arrow::Status (arrow::util::AsyncTaskScheduler*)>, arrow::internal::FnOnce<void (arrow::Status const&)>, arrow::StopToken) /arrow/cpp/src/arrow/util/async_util.cc:471:28 (libarrow.so.1900+0x41f177c) (BuildId: 979c6af65bc31a76383ada04625a106c5d7e626b)
    #7 arrow::acero::(anonymous namespace)::ExecPlanImpl::StartProducing() /arrow/cpp/src/arrow/acero/exec_plan.cc:128:35 (libarrow_acero.so.1900+0x3bd594) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #8 arrow::acero::ExecPlan::StartProducing() /arrow/cpp/src/arrow/acero/exec_plan.cc:439:59 (libarrow_acero.so.1900+0x3bd0b7) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #9 arrow::acero::(anonymous namespace)::DeclarationToExecBatchesImpl(arrow::acero::Declaration, arrow::acero::QueryOptions, arrow::internal::Executor*) /arrow/cpp/src/arrow/acero/exec_plan.cc:692:14 (libarrow_acero.so.1900+0x3c44fa) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #10 arrow::acero::DeclarationToExecBatches(arrow::acero::Declaration, arrow::acero::QueryOptions)::$_0::operator()(arrow::internal::Executor*) /arrow/cpp/src/arrow/acero/exec_plan.cc:896:16 (libarrow_acero.so.1900+0x3d4b5b) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #11 arrow::internal::FnOnce<arrow::Future<arrow::acero::BatchesWithCommonSchema> (arrow::internal::Executor*)>::FnImpl<arrow::acero::DeclarationToExecBatches(arrow::acero::Declaration, arrow::acero::QueryOptions)::$_0>::invoke(arrow::internal::Executor*&&) /arrow/cpp/src/arrow/util/functional.h:152:42 (libarrow_acero.so.1900+0x3d4a95) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #12 arrow::internal::FnOnce<arrow::Future<arrow::acero::BatchesWithCommonSchema> (arrow::internal::Executor*)>::operator()(arrow::internal::Executor*) && /arrow/cpp/src/arrow/util/functional.h:140:17 (libarrow_acero.so.1900+0x414840) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #13 arrow::Future<arrow::acero::BatchesWithCommonSchema>::SyncType arrow::internal::RunSynchronously<arrow::Future<arrow::acero::BatchesWithCommonSchema>, arrow::acero::BatchesWithCommonSchema>(arrow::internal::FnOnce<arrow::Future<arrow::acero::BatchesWithCommonSchema> (arrow::internal::Executor*)>, bool) /arrow/cpp/src/arrow/util/thread_pool.h:587:16 (libarrow_acero.so.1900+0x3df195) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #14 arrow::acero::DeclarationToExecBatches(arrow::acero::Declaration, arrow::acero::QueryOptions) /arrow/cpp/src/arrow/acero/exec_plan.cc:893:10 (libarrow_acero.so.1900+0x3c51f0) (BuildId: 64b34dbe92103d75042337f8f1c0f4f9af4adecd)
    #15 void arrow::acero::TestSequencing<arrow::Result<arrow::acero::BatchesWithSchema> (*)(std::vector<std::function<long (int)>, std::allocator<std::function<long (int)>>> const&, std::shared_ptr<arrow::Schema> const&, int, int)>(arrow::Result<arrow::acero::BatchesWithSchema> (*)(std::vector<std::function<long (int)>, std::allocator<std::function<long (int)>>> const&, std::shared_ptr<arrow::Schema> const&, int, int), int, int) /arrow/cpp/src/arrow/acero/asof_join_node_test.cc:1579:3 (arrow-acero-asof-join-node-test+0x297083) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #16 arrow::acero::AsofJoinTest_BatchSequencing_Test::TestBody() /arrow/cpp/src/arrow/acero/asof_join_node_test.cc:1586:10 (arrow-acero-asof-join-node-test+0x27d8a2) (BuildId: 6f16c4def26fb96c1f25afac25e3e4b3068467c5)
    #17 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (libarrow_testing.so.1900+0x48ec8e) (BuildId: 113acbd90ed4ee9f7f986424c1901035aee7411d)

SUMMARY: ThreadSanitizer: data race /arrow/cpp/src/arrow/acero/concurrent_queue_internal.h:86:12 in arrow::acero::ConcurrentQueue<std::shared_ptr<arrow::RecordBatch>>::PushUnlocked(std::shared_ptr<arrow::RecordBatch> const&)

Component(s)

C++, Continuous Integration
raulcd added a commit to raulcd/arrow that referenced this issue Nov 25, 2024
@raulcd
apacheGH-44846: [CI][C++] Investigate apacheGH-44846
c58ba59
@github-actions github-actions bot mentioned this issue Nov 25, 2024
Closed
raulcd added a commit to raulcd/arrow that referenced this issue Nov 25, 2024
@raulcd
apacheGH-44846: [CI][C++] Investigate apacheGH-44846 by reverting pre…
bdb0017 
…vious change
pitrou added a commit to pitrou/arrow that referenced this issue Nov 25, 2024
@pitrou
apacheGH-44846: [C++] Fix thread-unsafe access in ConcurrentQueue::Un…
2f92a5b 
…syncFront
@github-actions github-actions bot mentioned this issue Nov 25, 2024
Merged
pitrou added a commit that referenced this issue Nov 26, 2024
@pitrou
GH-44846: [C++] Fix thread-unsafe access in ConcurrentQueue::UnsyncFr…
c4d17fd 
…ont (#44849)

### Rationale for this change

The `UnsyncFront` method claims that it can access a `std::deque`'s front element without synchronizing with another thread that would call `std::deque::push_back`, but `std::deque::front` can actually be implemented in terms of `std::deque::begin` while `std::deque::push_back` is specified to invalidate iterators.

In the end, `UnsyncFront` is concretely thread-unsafe even though it might ideally be thread-safe. This shows up as occasional Thread Sanitizer failures.

### What changes are included in this PR?

Replace `UnsyncFront` with a thread-safe `Front` method.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.

* GitHub Issue: #44846

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
@pitrou pitrou added this to the 19.0.0 milestone Nov 26, 2024
@pitrou
Copy link
Member

pitrou commented Nov 26, 2024

Issue resolved by pull request 44849
#44849

@pitrou pitrou closed this as completed Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raulcd raulcd

Labels
Component: C++ Component: Continuous Integration Type: bug
Projects
None yet
Milestone
19.0.0
Development

No branches or pull requests
2 participants
@raulcd @pitrou