test_node is flaky #1169
Comments
|
I'm looking into a fix. |
|
Possibly related: I don't think |
jacobperron
added a commit
that referenced
this issue
Nov 19, 2020
gazebo_ros incorrectly assumes rclcpp::MultiThreadedExecutor is thread-safe. This change adds thread-safe wrappers to methods for adding/removing nodes and spinning. I believe this fixes #1169. This may have some performance impact on two fronts: 1) the locking while spinning and 2) switching from calling spin() to spin_once() (I think) removes the multi-threaded nature of the executor. Signed-off-by: Jacob Perron <[email protected]>
jacobperron
added a commit
that referenced
this issue
Nov 20, 2020
The rclcpp::MultiThreadedExecutor is not thread safe, and so it is possible to hit races that cause gazebo_ros to crash. Specifically, there's a race when adding and removing nodes, which can happen when node are created and destroyed during runtime. This is a copy of rclcpp::Executor with some modifications to make it thread safe. A lock was added to the spin implementation and also to the add_node and remove_node methods. In order to avoid deadlock, we must have a timeout in the spin loop while waiting for work. Fixes #1169 Signed-off-by: Jacob Perron <[email protected]>
|
Using ros2/rclcpp#1505, which addresses thread-safety issues, I still get the same crash in |
|
I will try to reproduce, as it doesn't seem obvious what's the problem from the traceback. PS:
|
|
Fixed by #1212 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I first noticed the test failing here: https://build.osrfoundation.org/job/ros2_gazebo_pkgs-ci-default_rolling-focal-amd64/5/
Locally, I can reproduce if I run the test standalone (not with
colcon test). I think the lack of stress is uncovering some sort of race in the test:Backtrace
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f6992d7d859 in __GI_abort () at abort.c:79 #2 0x00007f6993152951 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007f699315e47c in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007f699315e4e7 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007f699315e46f in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007f69935af6ed in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string, std::allocator > const&, rcutils_error_state_t const*, void (*)()) () at /opt/ros/rolling/lib/librclcpp.so #7 0x00007f699363c7a6 in rclcpp::SubscriptionBase::SubscriptionBase(rclcpp::node_interfaces::NodeBaseInterface*, rosidl_message_type_support_t const&, std::__cxx11::basic_string, std::allocator > const&, rcl_subscription_options_t const&, bool) () at /opt/ros/rolling/lib/librclcpp.so #8 0x00007f699365f8ed in rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >::Subscription(rclcpp::node_interfaces::NodeBaseInterface*, rosidl_message_type_support_t const&, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&, rclcpp::AnySubscriptionCallback >, std::allocator >, rclcpp::SubscriptionOptionsWithAllocator > const&, std::shared_ptr >, std::allocator > >, std::shared_ptr > > >) () at /opt/ros/rolling/lib/librclcpp.so #9 0x00007f6993660764 in rclcpp::create_subscription_factory >, std::_Bind))(std::shared_ptr > >)>, std::allocator, rosgraph_msgs::msg::Clock_ >, rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >(std::_Bind))(std::shared_ptr > >)>&&, rclcpp::SubscriptionOptionsWithAllocator > const&, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >::SharedPtr, std::shared_ptr > > >)::{lambda(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&)#1}::operator()(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&) const () at /opt/ros/rolling/lib/librclcpp.so #10 0x00007f6993660948 in std::_Function_handler (rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&), rclcpp::create_subscription_factory >, std::_Bind))(std::shared_ptr > >)>, std::allocator, rosgraph_msgs::msg::Clock_ >, rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >(std::_Bind))(std::shared_ptr > >)>&&, rclcpp::SubscriptionOptionsWithAllocator > const&, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >::SharedPtr, std::shared_ptr > > >)::{lambda(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&)#1}>::_M_invoke(std::_Any_data const&, rclcpp::node_interfaces::NodeBaseInterface*&&, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&) () at /opt/ros/rolling/lib/librclcpp.so #11 0x00007f699360182b in rclcpp::node_interfaces::NodeTopics::create_subscription(std::__cxx11::basic_string, std::allocator > const&, rclcpp::SubscriptionFactory const&, rclcpp::QoS const&) () at /opt/ros/rolling/lib/librclcpp.so #12 0x00007f699365d479 in std::shared_ptr >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > > > rclcpp::create_subscription >, std::_Bind))(std::shared_ptr > >)>, std::allocator, rosgraph_msgs::msg::Clock_ >, rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >, std::shared_ptr&>(std::shared_ptr&, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&, std::_Bind))(std::shared_ptr > >)>&&, rclcpp::SubscriptionOptionsWithAllocator > const&, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >::SharedPtr) () at /opt/ros/rolling/lib/librclcpp.so #13 0x00007f699363ecb6 in rclcpp::TimeSource::create_clock_sub() () at /opt/ros/rolling/lib/librclcpp.so #14 0x00007f6993641638 in rclcpp::TimeSource::on_parameter_event(std::shared_ptr > >) () at /opt/ros/rolling/lib/librclcpp.so #15 0x00007f69936495c8 in std::_Function_handler > >), std::_Bind))(std::shared_ptr > >)> >::_M_invoke(std::_Any_data const&, std::shared_ptr > >&&) () at /opt/ros/rolling/lib/librclcpp.so #16 0x00007f69936594a1 in rclcpp::AnySubscriptionCallback >, std::allocator >::dispatch(std::shared_ptr > >, rclcpp::MessageInfo const&) () at /opt/ros/rolling/lib/librclcpp.so #17 0x00007f699365af6f in rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >::handle_message(std::shared_ptr&, rclcpp::MessageInfo const&) () at /opt/ros/rolling/lib/librclcpp.so #18 0x00007f69935b3d1c in () at /opt/ros/rolling/lib/librclcpp.so #19 0x00007f69935b45db in rclcpp::Executor::execute_subscription(std::shared_ptr) () at /opt/ros/rolling/lib/librclcpp.so #20 0x00007f69935b4d9a in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) () at /opt/ros/rolling/lib/librclcpp.so #21 0x00007f69935bd324 in rclcpp::executors::MultiThreadedExecutor::run(unsigned long) () at /opt/ros/rolling/lib/librclcpp.so #22 0x00007f69935bd915 in rclcpp::executors::MultiThreadedExecutor::spin() () at /opt/ros/rolling/lib/librclcpp.so #23 0x00007f6993753aa5 in gazebo_ros::Executor::run() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #24 0x00007f699375c554 in void std::__invoke_impl(std::__invoke_memfun_deref, void (gazebo_ros::Executor::*&)(), gazebo_ros::Executor*&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #25 0x00007f699375c14b in std::__invoke_result::type std::__invoke(void (gazebo_ros::Executor::*&)(), gazebo_ros::Executor*&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #26 0x00007f699375b81d in void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #27 0x00007f699375ab0f in void std::_Bind::operator()<, void>() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #28 0x00007f699375d33e in void std::__invoke_impl>(std::__invoke_other, std::_Bind&&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #29 0x00007f699375d2e7 in std::__invoke_result>::type std::__invoke>(std::_Bind&&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #30 0x00007f699375d288 in void std::thread::_Invoker > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #31 0x00007f699375d0c6 in std::thread::_Invoker > >::operator()() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #32 0x00007f699375ce00 in std::thread::_State_impl > > >::_M_run() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #33 0x00007f699318acb4 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #34 0x00007f699347f609 in start_thread (arg=) at pthread_create.c:477 #35 0x00007f6992e7a293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f6992d7d859 in __GI_abort () at abort.c:79 #2 0x00007f6993152951 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007f699315e47c in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007f699315e4e7 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007f699315e46f in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007f69935af6ed in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string, std::allocator > const&, rcutils_error_state_t const*, void (*)()) () at /opt/ros/rolling/lib/librclcpp.so #7 0x00007f699363c7a6 in rclcpp::SubscriptionBase::SubscriptionBase(rclcpp::node_interfaces::NodeBaseInterface*, rosidl_message_type_support_t const&, std::__cxx11::basic_string, std::allocator > const&, rcl_subscription_options_t const&, bool) () at /opt/ros/rolling/lib/librclcpp.so #8 0x00007f699365f8ed in rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >::Subscription(rclcpp::node_interfaces::NodeBaseInterface*, rosidl_message_type_support_t const&, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&, rclcpp::AnySubscriptionCallback >, std::allocator >, rclcpp::SubscriptionOptionsWithAllocator > const&, std::shared_ptr >, std::allocator > >, std::shared_ptr > > >) () at /opt/ros/rolling/lib/librclcpp.so #9 0x00007f6993660764 in rclcpp::create_subscription_factory >, std::_Bind))(std::shared_ptr > >)>, std::allocator, rosgraph_msgs::msg::Clock_ >, rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >(std::_Bind))(std::shared_ptr > >)>&&, rclcpp::SubscriptionOptionsWithAllocator > const&, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >::SharedPtr, std::shared_ptr > > >)::{lambda(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&)#1}::operator()(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&) const () at /opt/ros/rolling/lib/librclcpp.so #10 0x00007f6993660948 in std::_Function_handler (rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&), rclcpp::create_subscription_factory >, std::_Bind))(std::shared_ptr > >)>, std::allocator, rosgraph_msgs::msg::Clock_ >, rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >(std::_Bind))(std::shared_ptr > >)>&&, rclcpp::SubscriptionOptionsWithAllocator > const&, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >::SharedPtr, std::shared_ptr > > >)::{lambda(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&)#1}>::_M_invoke(std::_Any_data const&, rclcpp::node_interfaces::NodeBaseInterface*&&, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&) () at /opt/ros/rolling/lib/librclcpp.so #11 0x00007f699360182b in rclcpp::node_interfaces::NodeTopics::create_subscription(std::__cxx11::basic_string, std::allocator > const&, rclcpp::SubscriptionFactory const&, rclcpp::QoS const&) () at /opt/ros/rolling/lib/librclcpp.so #12 0x00007f699365d479 in std::shared_ptr >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > > > rclcpp::create_subscription >, std::_Bind))(std::shared_ptr > >)>, std::allocator, rosgraph_msgs::msg::Clock_ >, rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >, std::shared_ptr&>(std::shared_ptr&, std::__cxx11::basic_string, std::allocator > const&, rclcpp::QoS const&, std::_Bind))(std::shared_ptr > >)>&&, rclcpp::SubscriptionOptionsWithAllocator > const&, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator >::SharedPtr) () at /opt/ros/rolling/lib/librclcpp.so #13 0x00007f699363ecb6 in rclcpp::TimeSource::create_clock_sub() () at /opt/ros/rolling/lib/librclcpp.so #14 0x00007f6993641638 in rclcpp::TimeSource::on_parameter_event(std::shared_ptr > >) () at /opt/ros/rolling/lib/librclcpp.so #15 0x00007f69936495c8 in std::_Function_handler > >), std::_Bind))(std::shared_ptr > >)> >::_M_invoke(std::_Any_data const&, std::shared_ptr > >&&) () at /opt/ros/rolling/lib/librclcpp.so #16 0x00007f69936594a1 in rclcpp::AnySubscriptionCallback >, std::allocator >::dispatch(std::shared_ptr > >, rclcpp::MessageInfo const&) () at /opt/ros/rolling/lib/librclcpp.so #17 0x00007f699365af6f in rclcpp::Subscription >, std::allocator, rclcpp::message_memory_strategy::MessageMemoryStrategy >, std::allocator > >::handle_message(std::shared_ptr&, rclcpp::MessageInfo const&) () at /opt/ros/rolling/lib/librclcpp.so #18 0x00007f69935b3d1c in () at /opt/ros/rolling/lib/librclcpp.so #19 0x00007f69935b45db in rclcpp::Executor::execute_subscription(std::shared_ptr) () at /opt/ros/rolling/lib/librclcpp.so #20 0x00007f69935b4d9a in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) () at /opt/ros/rolling/lib/librclcpp.so #21 0x00007f69935bd324 in rclcpp::executors::MultiThreadedExecutor::run(unsigned long) () at /opt/ros/rolling/lib/librclcpp.so #22 0x00007f69935bd915 in rclcpp::executors::MultiThreadedExecutor::spin() () at /opt/ros/rolling/lib/librclcpp.so #23 0x00007f6993753aa5 in gazebo_ros::Executor::run() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #24 0x00007f699375c554 in void std::__invoke_impl(std::__invoke_memfun_deref, void (gazebo_ros::Executor::*&)(), gazebo_ros::Executor*&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #25 0x00007f699375c14b in std::__invoke_result::type std::__invoke(void (gazebo_ros::Executor::*&)(), gazebo_ros::Executor*&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #26 0x00007f699375b81d in void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #27 0x00007f699375ab0f in void std::_Bind::operator()<, void>() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #28 0x00007f699375d33e in void std::__invoke_impl>(std::__invoke_other, std::_Bind&&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #29 0x00007f699375d2e7 in std::__invoke_result>::type std::__invoke>(std::_Bind&&) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #30 0x00007f699375d288 in void std::thread::_Invoker > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #31 0x00007f699375d0c6 in std::thread::_Invoker > >::operator()() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #32 0x00007f699375ce00 in std::thread::_State_impl > > >::_M_run() () at /home/jacob/ws/ros/gazebo_ros_pkgs/install/gazebo_ros/lib/libgazebo_ros_node.so #33 0x00007f699318acb4 in () at /lib/x86_64-linux-gnu/libstdc++.so.6 #34 0x00007f699347f609 in start_thread (arg=) at pthread_create.c:477 #35 0x00007f6992e7a293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95The text was updated successfully, but these errors were encountered: