Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osrm-routed connection accept loop can exit and not recover #6040

Closed
mjjbell opened this issue May 21, 2021 · 2 comments · Fixed by #6112
Closed

osrm-routed connection accept loop can exit and not recover #6040

mjjbell opened this issue May 21, 2021 · 2 comments · Fixed by #6112

Comments

@mjjbell
Copy link
Member

mjjbell commented May 21, 2021

As discussed in #6033

osrm-routed does not immediately clean up a keep-alive connection when the client closes it. Instead it waits for five seconds of inactivity before removing.

If you have a client that opens and closes a lot of keep-alive connections, it's possible for osrm-routed to run out of file descriptors whilst it waits for the clean-up to trigger.

The key point here is if this does happen, the connection acceptor loop exits. Even after the old connections are cleaned up, new ones will not be created. Any new requests will block until the server is restarted. See: https://github.com/Project-OSRM/osrm-backend/blob/master/include/server/server.hpp#L96

It's possible that there are other errors which will also generate this behaviour.

Ideas for improving the error handling:

  • Close connections as soon as they receive an error (e.g. when the client closes them).
  • Attempt to restart the connection accept loop on error.
  • Don't try to handle these errors, and ensure osrm-routed exits cleanly.
@datendelphin
Copy link

As I complained in the other ticket about rare crashes, this problem description could match at lest some of them. I sometimes could attach a debugger to get a stack trace. One type was definitely all threads exited except one, which would fit this issue, but I didn't save those traces. But others still had threads, though the stack trace was not very enlightening. Not sure if those crashes are related to this problem, but I thought I'll post the stack traces here anyway:

Thread 8 (Thread 0x7f47d5ffb700 (LWP 4139)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562bf163c068) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562bf163c008, cond=0x562bf163c040) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562bf163c040, mutex=0x562bf163c008) at pthread_cond_wait.c:655
#3  0x0000562bf0a8b04c in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#4  0x0000562bf0a8b3c9 in boost::asio::io_context::run() ()
#5  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f47d67fc700 (LWP 4138)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562bf163c068) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562bf163c008, cond=0x562bf163c040) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562bf163c040, mutex=0x562bf163c008) at pthread_cond_wait.c:655
#3  0x0000562bf0a8b04c in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#4  0x0000562bf0a8b3c9 in boost::asio::io_context::run() ()
#5  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f47d6ffd700 (LWP 4137)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562bf163c068) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562bf163c008, cond=0x562bf163c040) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562bf163c040, mutex=0x562bf163c008) at pthread_cond_wait.c:655
#3  0x0000562bf0a8b04c in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#4  0x0000562bf0a8b3c9 in boost::asio::io_context::run() ()
#5  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f47d77fe700 (LWP 4136)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562bf163c06c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562bf163c008, cond=0x562bf163c040) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562bf163c040, mutex=0x562bf163c008) at pthread_cond_wait.c:655
#3  0x0000562bf0a8b04c in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#4  0x0000562bf0a8b3c9 in boost::asio::io_context::run() ()
#5  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f47d7fff700 (LWP 4135)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562bf163c06c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562bf163c008, cond=0x562bf163c040) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562bf163c040, mutex=0x562bf163c008) at pthread_cond_wait.c:655
#3  0x0000562bf0a8b04c in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#4  0x0000562bf0a8b3c9 in boost::asio::io_context::run() ()
#5  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f47dcfdb700 (LWP 4134)):
#0  0x00007f54422697ef in epoll_wait (epfd=5, events=0x7f47dcfda780, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x0000562bf0a85f28 in boost::asio::detail::epoll_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&) ()
#2  0x0000562bf0a8af33 in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
#3  0x0000562bf0a8b3c9 in boost::asio::io_context::run() ()
#4  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f47dd7dc700 (LWP 4133)):
#0  0x00007f54426e6495 in __GI___pthread_timedjoin_ex (threadid=139946627020544, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89
#1  0x00007f5442589d53 in std::thread::join() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000562bf0a8e938 in osrm::server::Server::Run() ()
#3  0x0000562bf0a8eb69 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<int>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<main::{lambda()#1}, std::allocator<int>, int ()>::_M_run()::{lambda()#1}, int> >::_M_invoke(std::_Any_data const&) ()
#4  0x0000562bf0a81c6b in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) ()
#5  0x00007f54426ec997 in __pthread_once_slow (once_control=0x562bf163b8d8, init_routine=0x7f5442588e20 <__once_proxy>) at pthread_once.c:116
#6  0x0000562bf0a84b37 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::packaged_task<int ()> > > >::_M_run() ()
#7  0x00007f5442589b2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f54426e4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f54422694cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f54402a6740 (LWP 2647)):
#0  0x00007f54421a84cc in __GI___sigtimedwait (set=set@entry=0x7ffc1f817c20, info=info@entry=0x7ffc1f817a50, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1  0x00007f54426ef2bc in __sigwait (set=0x7ffc1f817c20, sig=0x7ffc1f817b44) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2  0x0000562bf0a7da64 in main ()
Thread 5 (Thread 0x7f85de9bc700 (LWP 7592)):
#0  0x00007f9244c4c7ef in epoll_wait (epfd=5, events=0x7f85de9bb6b0, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00005576a0551a35 in boost::asio::detail::epoll_reactor::run (this=0x5576a150e500, usec=<optimized out>, ops=...)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:471
#2  0x00005576a05523ec in boost::asio::detail::scheduler::do_run_one (this=0x5576a1505fd0, lock=..., this_thread=..., ec=...)
    at /usr/include/boost/asio/detail/impl/scheduler.ipp:385
#3  0x00005576a0554c53 in boost::asio::detail::scheduler::run (this=0x5576a1505fd0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:154
#4  0x00005576a0554cd1 in boost::asio::io_context::run (this=0x5576a1507fc8) at /usr/include/boost/asio/impl/io_context.ipp:62
#5  0x00005576a0560de8 in boost::_mfi::mf0<unsigned long, boost::asio::io_context>::operator() (this=<optimized out>, p=<optimized out>)
    at /usr/include/boost/bind/mem_fn_template.hpp:47
#6  0x00005576a0560e35 in boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> >::operator()<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list0> (this=0x7f85d8000ea8, f=..., a=...) at /usr/include/boost/bind/bind.hpp:247
#7  0x00005576a0560e5d in boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >::operator() (this=<optimized out>) at /usr/include/boost/bind/bind.hpp:1291
#8  0x00005576a0560e70 in std::__invoke_impl<unsigned long, boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >>(std::__invoke_other, boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >&&) (__f=...) at /usr/include/c++/8/bits/invoke.h:59
#9  0x00005576a0560e83 in std::__invoke<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >>(boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >&&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:89
#10 0x00005576a0563925 in std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > >::_M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/8/thread:241
#11 0x00005576a0563931 in std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > >::operator() (this=<optimized out>) at /usr/include/c++/8/thread:250
#12 0x00005576a0563941 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > > >::_M_run (this=<optimized out>) at /usr/include/c++/8/thread:196
#13 0x00007f9244f6cb2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007f92450c7fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#15 0x00007f9244c4c4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f85df1bd700 (LWP 7591)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1  0x00007f92450ca714 in __GI___pthread_mutex_lock (mutex=0x5576a1506008) at ../nptl/pthread_mutex_lock.c:80
#2  0x00005576a054c44f in boost::asio::detail::posix_mutex::lock (this=<optimized out>) at /usr/include/boost/asio/detail/posix_mutex.hpp:52
#3  0x00005576a054cdb1 in boost::asio::detail::conditionally_enabled_mutex::scoped_lock::lock (this=0x7f85df1bcd60)
    at /usr/include/boost/asio/detail/conditionally_enabled_mutex.hpp:74
#4  0x00005576a0554c3d in boost::asio::detail::scheduler::run (this=0x5576a1505fd0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:156
#5  0x00005576a0554cd1 in boost::asio::io_context::run (this=0x5576a1507fc8) at /usr/include/boost/asio/impl/io_context.ipp:62
#6  0x00005576a0560de8 in boost::_mfi::mf0<unsigned long, boost::asio::io_context>::operator() (this=<optimized out>, p=<optimized out>)
    at /usr/include/boost/bind/mem_fn_template.hpp:47
#7  0x00005576a0560e35 in boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> >::operator()<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list0> (this=0x7f85d8000d08, f=..., a=...) at /usr/include/boost/bind/bind.hpp:247
#8  0x00005576a0560e5d in boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >::operator() (this=<optimized out>) at /usr/include/boost/bind/bind.hpp:1291
#9  0x00005576a0560e70 in std::__invoke_impl<unsigned long, boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >>(std::__invoke_other, boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >&&) (__f=...) at /usr/include/c++/8/bits/invoke.h:59
#10 0x00005576a0560e83 in std::__invoke<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >>(boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >&&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:89
#11 0x00005576a0563925 in std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > >::_M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/8/thread:241
#12 0x00005576a0563931 in std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > >::operator() (this=<optimized out>) at /usr/include/c++/8/thread:250
#13 0x00005576a0563941 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > > >::_M_run (this=<optimized out>) at /usr/include/c++/8/thread:196
#14 0x00007f9244f6cb2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f92450c7fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#16 0x00007f9244c4c4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f85df9be700 (LWP 7590)):
#0  futex_wake (private=<optimized out>, processes_to_wake=1, futex_word=<optimized out>) at ../sysdeps/unix/sysv/linux/futex-internal.h:231
#1  __pthread_cond_signal (cond=0x5576a1506040) at pthread_cond_signal.c:94
#2  0x00005576a054e772 in boost::asio::detail::posix_event::maybe_unlock_and_signal_one<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (this=0x5576a1506040, lock=...) at /usr/include/boost/asio/detail/posix_event.hpp:76
#3  0x00005576a054e793 in boost::asio::detail::conditionally_enabled_event::maybe_unlock_and_signal_one (this=<optimized out>, lock=...) at /usr/include/boost/asio/detail/conditionally_enabled_event.hpp:73
#4  0x00005576a054e7ac in boost::asio::detail::scheduler::wake_one_thread_and_unlock (this=0x5576a1505fd0, lock=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:556
#5  0x00005576a0552324 in boost::asio::detail::scheduler::do_run_one (this=0x5576a1505fd0, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:392
#6  0x00005576a0554c53 in boost::asio::detail::scheduler::run (this=0x5576a1505fd0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:154
#7  0x00005576a0554cd1 in boost::asio::io_context::run (this=0x5576a1507fc8) at /usr/include/boost/asio/impl/io_context.ipp:62
#8  0x00005576a0560de8 in boost::_mfi::mf0<unsigned long, boost::asio::io_context>::operator() (this=<optimized out>, p=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:47
#9  0x00005576a0560e35 in boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> >::operator()<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list0> (this=0x7f85d8000b58, f=..., a=...) at /usr/include/boost/bind/bind.hpp:247
#10 0x00005576a0560e5d in boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >::operator() (this=<optimized out>) at /usr/include/boost/bind/bind.hpp:1291
#11 0x00005576a0560e70 in std::__invoke_impl<unsigned long, boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >>(std::__invoke_other, boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >&&) (__f=...) at /usr/include/c++/8/bits/invoke.h:59
#12 0x00005576a0560e83 in std::__invoke<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >>(boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > >&&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:89
#13 0x00005576a0563925 in std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > >::_M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/8/thread:241
#14 0x00005576a0563931 in std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > >::operator() (this=<optimized out>) at /usr/include/c++/8/thread:250
#15 0x00005576a0563941 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_context>, boost::_bi::list1<boost::_bi::value<boost::asio::io_context*> > > > > >::_M_run (this=<optimized out>) at /usr/include/c++/8/thread:196
#16 0x00007f9244f6cb2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007f92450c7fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#18 0x00007f9244c4c4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f85e01bf700 (LWP 7589)):
#0  0x00007f92450c9495 in __GI___pthread_timedjoin_ex (threadid=140212958914304, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89
#1  0x00007f9244f6cd53 in std::thread::join() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00005576a0560a1d in osrm::server::Server::Run (this=0x5576a1507fc0) at /srv/osrm/osrm-backend/include/server/server.hpp:82
#3  0x00005576a0560a78 in <lambda()>::operator()(void) const (__closure=<optimized out>) at /srv/osrm/osrm-backend/src/tools/routed.cpp:300
#4  0x00005576a0560a90 in std::__invoke_impl<int, main(int, char const**)::<lambda()>&>(std::__invoke_other, <lambda()> &) (__f=...) at /usr/include/c++/8/bits/invoke.h:59
#5  0x00005576a0560aa3 in std::__invoke<main(int, char const**)::<lambda()>&>(<lambda()> &) (__fn=...) at /usr/include/c++/8/bits/invoke.h:89
#6  0x00005576a0560ac8 in std::__future_base::_Task_state<main(int, char const**)::<lambda()>, std::allocator<int>, int()>::<lambda()>::operator()(void) const (this=<optimized out>) at /usr/include/c++/8/future:1420
#7  0x00005576a056430d in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<int>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run(_Args&& ...) [with _Fn = main(int, char const**)::<lambda()>; _Alloc = std::allocator<int>; _Res = int; _Args = {}]::<lambda()>, int>::operator()(void) const (this=0x7f85e01bee20) at /usr/include/c++/8/future:1335
#8  0x00005576a05643b9 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<int>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run(_Args&& ...) [with _Fn = main(int, char const**)::<lambda()>; _Alloc = std::allocator<int>; _Res = int; _Args = {}]::<lambda()>, int> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:280
#9  0x00005576a054fbce in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x7f85e01bee20) at /usr/include/c++/8/bits/std_function.h:682
#10 0x00005576a0558593 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x5576a15058c0, __f=<optimized out>, __did_set=0x7f85e01bedc7) at /usr/include/c++/8/future:561
#11 0x00005576a0555404 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=<optimized out>, __t=<optimized out>, __args#0=@0x7f85e01bedd0: 0x7f85e01bee20, __args#1=<optimized out>) at /usr/include/c++/8/bits/invoke.h:70
#12 0x00005576a055545b in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=@0x7f85e01bede0: (void (std::__future_base::_State_baseV2::*)(class std::__future_base::_State_baseV2 * const, class std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x5576a0558574 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __args#0=<optimized out>, __args#1=<optimized out>, __args#2=<optimized out>) at /usr/include/c++/8/bits/invoke.h:89
#13 0x00005576a05554af in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=0x7f85e01bed70) at /usr/include/c++/8/mutex:671
#14 0x00005576a05554d4 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=<optimized out>) at /usr/include/c++/8/mutex:677
#15 0x00005576a05554e4 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/8/mutex:677
#16 0x00007f92450cf997 in __pthread_once_slow (once_control=0x5576a15058d8, init_routine=0x7f9244f6be20 <__once_proxy>) at pthread_once.c:116
#17 0x00005576a054ae20 in __gthread_once (__once=0x5576a15058d8, __func=0x7f9244f6be20 <__once_proxy>) at /usr/include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:699
#18 0x00005576a05527c0 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=<optimized out>, __args#0=<optimized out>, __args#1=<optimized out>, __args#2=<optimized out>) at /usr/include/c++/8/mutex:667
#19 0x00005576a0552837 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x5576a15058c0, __res=..., __ignore_failure=<optimized out>) at /usr/include/c++/8/future:401
#20 0x00005576a0561d7a in std::__future_base::_Task_state<main(int, char const**)::<lambda()>, std::allocator<int>, int()>::_M_run(void) (this=0x5576a15058c0) at /usr/include/c++/8/future:1418
#21 0x00005576a05583bc in std::packaged_task<int ()>::operator()() (this=0x5576a1509fa8) at /usr/include/c++/8/future:1553
#22 0x00005576a05583d4 in std::__invoke_impl<void, std::packaged_task<int ()>>(std::__invoke_other, std::packaged_task<int ()>&&) (__f=...) at /usr/include/c++/8/bits/invoke.h:59
#23 0x00005576a05583e7 in std::__invoke<std::packaged_task<int ()>>(std::packaged_task<int ()>&&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:89
#24 0x00005576a056399f in std::thread::_Invoker<std::tuple<std::packaged_task<int ()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/8/thread:241
#25 0x00005576a05639ab in std::thread::_Invoker<std::tuple<std::packaged_task<int ()> > >::operator()() (this=<optimized out>) at /usr/include/c++/8/thread:250
#26 0x00005576a05639bb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::packaged_task<int ()> > > >::_M_run() (this=<optimized out>) at /usr/include/c++/8/thread:196
#27 0x00007f9244f6cb2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#28 0x00007f92450c7fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#29 0x00007f9244c4c4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f9242c89740 (LWP 7583)):
#0  0x00007f9244b8b4cc in __GI___sigtimedwait (set=set@entry=0x7ffd58c526e0, info=info@entry=0x7ffd58c52540, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1  0x00007f92450d22bc in __sigwait (set=0x7ffd58c526e0, sig=0x7ffd58c525fc) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2  0x00005576a0568bd9 in main (argc=<optimized out>, argv=0x7ffd58c52b28) at /srv/osrm/osrm-backend/src/tools/routed.cpp:312

@mjjbell
Copy link
Member Author

mjjbell commented Aug 24, 2021

I've observed this on both Mac and Linux builds now.
Will look into what Boost Asio recommends in terms of error recovery as a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants