Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stop_accepting_connections overload action causes crash #13514

Closed
akonradi opened this issue Oct 12, 2020 · 5 comments · Fixed by #13515
Closed

stop_accepting_connections overload action causes crash #13514

akonradi opened this issue Oct 12, 2020 · 5 comments · Fixed by #13515
Labels
area/overload_manager bug

Comments

@akonradi
Copy link
Contributor

akonradi commented Oct 12, 2020 •
edited
Loading

When the stop_accepting_connections changes from active to inactive or vice-versa around the same time as an LDS update, it can result in a segmentation fault due to trying to dereference a null pointer. Thanks to @Christoska for reporting this first to envoy-security@. Since the issue can be easily mitigated (by removing the entry for the stop_accepting_connections action from the overload manager config), the window in which it can be triggered is small, and the only possible result is a segfault and crash, this can go through the regular bug process.

Concretely, this is what happens:

  1. An LDS update comes that removes a listener.
  2. This results in a call to ListenerManagerImpl::removeListener,
  3. which calls drainListener, which calls stopListener,
  4. which calls Worker::stopListener.
  5. WorkerImpl::stopListener calls ConnectionHandler::stopListeners,
  6. which calls ActiveListener::shutdownListener.
  7. ActiveTcpListener::shutdownListener deletes the unique_ptr listener_ member.

Repro steps:
Enable the envoy.overload_actions.stop_accepting_connection overload action.
Run Envoy near the overload threshold such that the action changes state frequently.
Send LDS updates for Envoy via the control server.

Config:
Overload config that was causing the issue:

    overload_manager:
      refresh_interval:
        seconds: 0
        nanos: 250000000
      resource_monitors:
      - name: "envoy.resource_monitors.fixed_heap"
        typed_config:
          "@type": type.googleapis.com/envoy.config.resource_monitor.fixed_heap.v2alpha.FixedHeapConfig
          max_heap_size_bytes: 943718400
      actions:
      - name: "envoy.overload_actions.shrink_heap"
        triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.90
      - name: "envoy.overload_actions.disable_http_keepalive"
        triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.95
      - name: "envoy.overload_actions.stop_accepting_connections"
        triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.97
      - name: "envoy.overload_actions.stop_accepting_requests"
        triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.99

Call Stack:
From examining the core dump with GDB:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055d1d2914fbe in Envoy::Server::ConnectionHandlerImpl::ActiveTcpListener::pauseListening (
    this=0x55d1d6782b40)
    at bazel-out/k8-dbg/bin/source/server/_virtual_includes/connection_handler_lib/server/connection_handler_impl.h:133
133	bazel-out/k8-dbg/bin/source/server/_virtual_includes/connection_handler_lib/server/connection_handler_impl.h: No such file or directory.
[Current thread is 1 (Thread 0x7fd37c444700 (LWP 23))]
Missing separate debuginfos, use: yum debuginfo-install cesp-envoy-debug-1.14.4-1.x86_64
(gdb) bt
#0  0x000055d1d2914fbe in Envoy::Server::ConnectionHandlerImpl::ActiveTcpListener::pauseListening (
    this=0x55d1d6782b40)
    at bazel-out/k8-dbg/bin/source/server/_virtual_includes/connection_handler_lib/server/connection_handler_impl.h:133
#1  0x000055d1d290a27a in Envoy::Server::ConnectionHandlerImpl::disableListeners (this=0x55d1d5711b30)
    at source/server/connection_handler_impl.cc:74
#2  0x000055d1d29000a6 in Envoy::Server::WorkerImpl::stopAcceptingConnectionsCb (this=0x55d1d56ed380, 
    state=Envoy::Server::OverloadActionState::Active) at source/server/worker_impl.cc:125
#3  0x000055d1d29015de in Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0::operator()(Envoy::Server::OverloadActionState) const (this=0x55d1d56df778, 
    state=Envoy::Server::OverloadActionState::Active) at source/server/worker_impl.cc:34
#4  0x000055d1d29015a1 in std::__1::__invoke<Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0&, Envoy::Server::OverloadActionState> (__f=..., 
    __args=@0x7fd37c42fc64: Envoy::Server::OverloadActionState::Active)
    at /opt/llvm/bin/../include/c++/v1/type_traits:3530
#5  0x000055d1d2901542 in std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0&, Envoy::Server::OverloadActionState>(Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0&, Envoy::Server::OverloadActionState&&) (
    __args=@0x7fd37c42fc64: Envoy::Server::OverloadActionState::Active, 
    __args=@0x7fd37c42fc64: Envoy::Server::OverloadActionState::Active)
    at /opt/llvm/bin/../include/c++/v1/__functional_base:348
#6  0x000055d1d2901502 in std::__1::__function::__alloc_func<Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0, std::__1::allocator<Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0>, void (Envoy::Server::OverloadActionState)>::operator()(Envoy::Server::OverloadActionState&&) (this=0x55d1d56df778, 
    __arg=@0x7fd37c42fc64: Envoy::Server::OverloadActionState::Active)
    at /opt/llvm/bin/../include/c++/v1/functional:1533
#7  0x000055d1d2900643 in std::__1::__function::__func<Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0, std::__1::allocator<Envoy::Server::WorkerImpl::WorkerImpl(Envoy::ThreadLocal::Instance&, Envoy::ListenerHooks&, std::__1::unique_ptr<Envoy::Event::Dispatcher, std::__1::default_delete<Envoy::Event::Dispatcher> >&&, std::__1::unique_ptr<Envoy::Network::ConnectionHandler, std::__1::default_delete<Envoy::Network::ConnectionHandler> >, Envoy::Server::OverloadManager&, Envoy::Api::Api&, std::__1::basic--Type <RET> for more, q to quit, c to continue without paging-- 

c:OverloadActionState)>::operator()(Envoy::Server::OverloadActionState&&) (this=0x55d1d56df770, __arg=@0x7fd37c42fc64: Envoy::Server::OverloadActionState::Active) at /opt/llvm/bin/../include/c++/v1/functional:1707
#8  0x000055d1d2f2e08d in std::__1::__function::__value_func<void (Envoy::Server::OverloadActionState)>::operator()(Envoy::Server::OverloadActionState&&) const (this=0x55d1d56df770, 
    __args=@0x7fd37c42fc64: Envoy::Server::OverloadActionState::Active) at /opt/llvm/bin/../include/c++/v1/functional:1860
#9  0x000055d1d2f2e02f in std::__1::function<void (Envoy::Server::OverloadActionState)>::operator()(Envoy::Server::OverloadActionState) const (this=0x55d1d56df770, __arg=Envoy::Server::OverloadActionState::Active)
    at /opt/llvm/bin/../include/c++/v1/functional:2419
#10 0x000055d1d2f1b875 in Envoy::Server::OverloadManagerImpl::updateResourcePressure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, double)::$_2::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&) const::{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&) const::{lambda()#1}::operator()() const (this=0x7fd37c42fd98) at source/server/overload_manager_impl.cc:217
#11 0x000055d1d2f1b82d in std::__1::__invoke<Envoy::Server::OverloadManagerImpl::updateResourcePressure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, double)::$_2::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&) const::{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&) const::{lambda()#1}&> (__f=...) at /opt/llvm/bin/../include/c++/v1/type_traits:3530
#12 0x000055d1d2f1b7dd in std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::OverloadManagerImpl::updateResourcePressure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, double)::$_2::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&) const::{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&) const::{lambda()#1}&>(Envoy::Server::OverloadManagerImpl::updateResourcePressure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, double)::$_2::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&) const::{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&) const::{lambda()#1}&) (__args=...)
    at /opt/llvm/bin/../include/c++/v1/__functional_base:348
#13 0x000055d1d2f1b7ad in std::__1::__function::__alloc_func<Envoy::Server::OverloadManagerImpl::updateResourcePressure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, double)::$_2::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&) const::{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&) const::{lambda()#1}, std::__1::allocator<{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}>, void ()>::operator()() (this=0x7fd37c42fd98) at /opt/llvm/bin/../include/c++/v1/functional:1533
#14 0x000055d1d2f1a8de in std::__1::__function::__func<Envoy::Server::OverloadManagerImpl::updateResourcePressure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, double)::$_2::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&) const::{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}::operator()(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&) const::{lambda()#1}, std::__1::allocator<{lambda(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, Envoy::Server::OverloadManagerImpl::ActionCallback>&)#1}>, void ()>::operator()() 
(this=0x7fd37c42fd90) at /opt/llvm/bin/../include/c++/v1/functional:1707
#15 0x000055d1d0be5435 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x7fd37c42fd90) at /opt/llvm/bin/../include/c++/v1/functional:1860

#16 0x000055d1d0be50d5 in std::__1::function<void ()>::operator()() const (this=0x7fd37c42fd90) at /opt/llvm/bin/../include/c++/v1/functional:2419
#17 0x000055d1d292aa29 in Envoy::Event::DispatcherImpl::runPostCallbacks (this=0x55d1d576fc80) at source/common/event/dispatcher_impl.cc:225
#18 0x000055d1d292d438 in Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1::operator()() const (this=0x55d1d57a4a48) at source/common/event/dispatcher_impl.cc:43
#19 0x000055d1d292d3fd in std::__1::__invoke<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&> (__f=...) at /opt/llvm/bin/../include/c++/v1/type_traits:3530
#20 0x000055d1d292d3ad in std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&>(Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&) (__args=...) at /opt/llvm/bin/../include/c++/v1/__functional_base:348
#21 0x000055d1d292d37d in std::__1::__function::__alloc_func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1>, void ()>::operator()() (this=0x55d1d57a4a48) at /opt/llvm/bin/../include/c++/v1/functional:1533
#22 0x000055d1d292c4ce in std::__1::__function::__func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1>, void ()>::operator()() (this=0x55d1d57a4a40) at /opt/llvm/bin/../include/c++/v1/functional:1707
#23 0x000055d1d0be5435 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x55d1d57a4a40) at /opt/llvm/bin/../include/c++/v1/functional:1860
#24 0x000055d1d0be50d5 in std::__1::function<void ()>::operator()() const (this=0x55d1d57a4a40) at /opt/llvm/bin/../include/c++/v1/functional:2419
#25 0x000055d1d296a18d in Envoy::Event::TimerImpl::TimerImpl(Envoy::CSmartPtr<event_base, &event_base_free>&, std::__1::function<void ()>, Envoy::Event::Dispatcher&)::$_0::operator()(int, short, void*) const (this=0xffffffff, 
    arg=0x55d1d57a49c0) at source/common/event/timer_impl.cc:15
#26 0x000055d1d296a136 in Envoy::Event::TimerImpl::TimerImpl(Envoy::CSmartPtr<event_base, &event_base_free>&, std::__1::function<void ()>, Envoy::Event::Dispatcher&)::$_0::__invoke(int, short, void*) (arg=0x55d1d57a49c0)
    at source/common/event/timer_impl.cc:15
#27 0x000055d1d343f961 in event_process_active_single_queue (base=0x55d1d5451340, activeq=0x55d1d5431e90, max_to_process=2147483647, endtime=0x0)
    at /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/sandbox/processwrapper-sandbox/31/execroot/envoy/external/com_github_libevent_libevent/event.c:1713
#28 0x000055d1d343a05a in event_process_active (base=0x55d1d5451340) at /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/sandbox/processwrapper-sandbox/31/execroot/envoy/external/com_github_libevent_libevent/event.c:1805
#29 0x000055d1d3438f2c in event_base_loop (base=0x55d1d5451340, flags=0)
    at /build/tmp/_bazel_bazel/b570b5ccd0454dc9af9f65ab1833764d/sandbox/processwrapper-sandbox/31/execroot/envoy/external/com_github_libevent_libevent/event.c:2047
--Type <RET> for more, q to quit, c to continue without paging--
#30 0x000055d1d2968cd8 in Envoy::Event::LibeventScheduler::run (this=0x55d1d576fcd0, mode=Envoy::Event::Dispatcher::RunType::Block) at source/common/event/libevent_scheduler.cc:47
#31 0x000055d1d292a93a in Envoy::Event::DispatcherImpl::run (this=0x55d1d576fc80, type=Envoy::Event::Dispatcher::RunType::Block) at source/common/event/dispatcher_impl.cc:197
#32 0x000055d1d28ffef1 in Envoy::Server::WorkerImpl::threadRoutine (this=0x55d1d56ed380, guard_dog=...) at source/server/worker_impl.cc:110
#33 0x000055d1d2905aec in Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3::operator()() const (this=0x55d1d70743d8) at source/server/worker_impl.cc:75
#34 0x000055d1d2905aad in std::__1::__invoke<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3&> (__f=...) at /opt/llvm/bin/../include/c++/v1/type_traits:3530
#35 0x000055d1d2905a5d in std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3&>(Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3&) (__args=...)
    at /opt/llvm/bin/../include/c++/v1/__functional_base:348
#36 0x000055d1d2905a2d in std::__1::__function::__alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3, std::__1::allocator<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3>, void ()>::operator()() (this=0x55d1d70743d8) at /opt/llvm/bin/../include/c++/v1/functional:1533
#37 0x000055d1d2904b5e in std::__1::__function::__func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3, std::__1::allocator<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3>, void ()>::operator()()
    (this=0x55d1d70743d0) at /opt/llvm/bin/../include/c++/v1/functional:1707
#38 0x000055d1d0be5435 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x55d1d70743d0) at /opt/llvm/bin/../include/c++/v1/functional:1860

#39 0x000055d1d0be50d5 in std::__1::function<void ()>::operator()() const (this=0x55d1d70743d0) at /opt/llvm/bin/../include/c++/v1/functional:2419
#40 0x000055d1d35b4672 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const (this=0x55d1d70743c0, arg=0x55d1d70743c0) at source/common/common/posix/thread_impl.cc:33
#41 0x000055d1d35b4645 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) (arg=0x55d1d70743c0) at source/common/common/posix/thread_impl.cc:32
#42 0x00007fd38201a2de in start_thread () from /lib64/libpthread.so.0
#43 0x00007fd381d4be83 in clone () from /lib64/libc.so.6
@akonradi akonradi added bug triage Issue requires triage labels Oct 12, 2020
@akonradi akonradi mentioned this issue Oct 12, 2020
Merged
@mattklein123 mattklein123 added area/overload_manager and removed triage Issue requires triage labels Oct 12, 2020
@antoniovicente
Copy link
Contributor

antoniovicente commented Oct 13, 2020 •
edited
Loading

Seems like this crash was introduced by #10443, worth some backports to 1.14, 1.15 and 1.16?

@antoniovicente antoniovicente added the backport/review Request to backport to stable releases label Oct 13, 2020
@akonradi
Copy link
Contributor Author

I'd push for backporting given that it's a known issue and how simple the fix is.

@cpakulski cpakulski mentioned this issue Nov 3, 2020
Merged
@ggreenway ggreenway added backport/approved Approved backports to stable releases and removed backport/review Request to backport to stable releases labels Nov 3, 2020
This was referenced Nov 4, 2020
Merged
Merged
lizan pushed a commit that referenced this issue Nov 11, 2020
@cpakulski
backport to 1.14: Prevent SEGFAULT when disabling listener (#13515) (#…
1160bd6 
…13945)

This prevents the stop_listening overload action from causing
segmentation faults that can occur if the action is enabled after the
listener has already shut down.

Risk Level: low
Testing: unit tests
Docs Changes: none
Release Notes: Yes
Fixes #13514

Signed-off-by: Alex Konradi <[email protected]>
Signed-off-by: Christoph Pakulski <[email protected]>
@cpakulski cpakulski mentioned this issue Nov 16, 2020
Closed
@cpakulski
Copy link
Contributor

Backported to releases 1.14, 1.15 and 1.16. Removing backport/approved label.

@cpakulski cpakulski removed the backport/approved Approved backports to stable releases label Dec 16, 2020
@vinzo99
Copy link

vinzo99 commented Dec 17, 2020

Does this mean the backport fix is already available in 1.14.6 / 1.15.3 / 1.16.2 ? Or do we have to wait for 1.14.7 / 1.15.4 / 1.16.3 ?
Thanks

@akonradi
Copy link
Contributor Author

The fix is in 1.14.6, 1.15.3, and 1.16.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/overload_manager bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

overload: prevent segfault when disabling listener akonradi/envoy
6 participants
@akonradi @mattklein123 @ggreenway @antoniovicente @cpakulski @vinzo99