Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ext_proc fuzzer test trigger ENVOY_BUG when clear route cache #27657

Merged

Conversation

yanjunxiang-google
Copy link
Contributor

@yanjunxiang-google yanjunxiang-google commented May 26, 2023

This PR is to address an ENVOY_BUG crash exposed by ext_proc fuzzer test reported by:

https://clusterfuzz.corp.google.com/testcase-detail/4756668218736640

The steps to trigger this ENVOY_BUG is below:

  1. When handling upstream response body message, ext_proc filter sends the response body to the ext_proc server.
  2. The ext_proc server sends back a body response with header mutation and set the clear_route_cache flag.
  3. Such clear_route_cache flag will trigger function call into connection_manager to clear route cache, which is set to be blocked in the stage when connection_manager is processing the upstream header response. Thus the ENVOY_BUG is fired.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

@KBaichoo
Copy link
Contributor

/assign @tyxia

As codeowner

/wait

For CI

Copy link
Member

@tyxia tyxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! I think it is the right fix overall.

Also, cc @wbpcode, I think before PR #26045, clearRouteCache on response path to downstream client is just a no-op (since route selection is not needed on response path ) rather than triggering the ENVOY_BUG.

source/extensions/filters/http/ext_proc/ext_proc.h Outdated Show resolved Hide resolved
source/extensions/filters/http/ext_proc/processor_state.cc Outdated Show resolved Hide resolved
source/extensions/filters/http/ext_proc/processor_state.cc Outdated Show resolved Hide resolved
Signed-off-by: Yanjun Xiang <[email protected]>
@yanjunxiang-google
Copy link
Contributor Author

/assign @yanavlasov @htuch @mpwarres @stevenzzzz

@yanjunxiang-google
Copy link
Contributor Author

This is the OSS fuzzer report:
https://clusterfuzz.corp.google.com/testcase-detail/4756668218736640

Here is the crash trace back decode:

Command: /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test

  | Time ran: 57.5866904258728
  |  
  | INFO: google3-libFuzzer:
  | google3 flags should be passed as --flag=value
  | libFuzzer flags should be passed as -flag=value
  | INFO: Running in fuzzing mode, google3 logging disabled
  | INFO: Use --minloglevel=0 to re-enable logging
  | INFO: found LLVMFuzzerCustomMutator (0x5aa5109285c0). Disabling -len_control by default.
  | INFO: Running with entropic power schedule (0xFF, 100).
  | INFO: Seed:4231534795
  | INFO: Loaded 1 modules (1767040 inline 8-bit counters): 1767040 [0x5aa51a28ec50, 0x5aa51a43e2d0),
  | INFO: Loaded 1 PC tables (1767040 PCs): 1767040 [0x5aa51a43e2d0,0x5aa51bf34ad0),
  | /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test: Running 1 inputs 100 time(s) each.
  | Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test
  | MemorySanitizer:DEADLYSIGNAL
  | ==1458622==ERROR: MemorySanitizer: ABRT on unknown address 0x0539001641be (pc 0x79b7c07162fa bp 0x79b7b4756b60 sp 0x79b7b47569b8 T1458646)
  | #0 0x79b7c07162fa in raise (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d)
  | #1 0x5aa516aa81a3 in Envoy::Http::ConnectionManagerImpl::ActiveStream::routeCacheBlocked() const third_party/envoy/src/source/common/http/conn_manager_impl.h:310:7
  | #2 0x5aa516ab95a2 in Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc:1966:7
  | #3 0x5aa516ab99f3 in non-virtual thunk to Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc
  | #4 0x5aa510a2ad96 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::clearRouteCache(envoy::service::ext_proc::v3::CommonResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:375:47
  | #5 0x5aa510a2f2f1 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::handleBodyResponse(envoy::service::ext_proc::v3::BodyResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:328:7
  | #6 0x5aa510a153b6 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc:628:41
  | #7 0x5aa510a18c93 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc
  | #8 0x5aa510a01537 in onReceiveMessage third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc:53:14
  | #9 0x5aa510a01537 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorStreamImpl::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc
  | #10 0x5aa510a01fb8 in Envoy::Grpc::AsyncStreamCallbacksenvoy::service::ext_proc::v3::ProcessingResponse::onReceiveMessageRaw(std::__msan::unique_ptr<Envoy::Buffer::Instance, std::__msan::default_deleteEnvoy::Buffer::Instance>&&) third_party/envoy/src/source/common/grpc/typed_async_client.h:98:5
  | #11 0x5aa51314e664 in Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc:153:21
  | #12 0x5aa51314e836 in non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc
  | #13 0x5aa51316009e in Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/async_client_impl.cc:142:21
  | #14 0x5aa51379db66 in Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc:1606:15
  | #15 0x5aa51379e016 in non-virtual thunk to Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc
  | #16 0x5aa5137c7ad9 in Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.cc:311:11
  | #17 0x5aa5137dc48f in Envoy::Router::UpstreamRequestFilterManagerCallbacks::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.h:273:23
  | #18 0x5aa516af1223 in Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState) third_party/envoy/src/source/common/http/filter_manager.cc:1356:29
  | #19 0x5aa516af19d9 in encodeData third_party/envoy/src/source/common/http/filter_manager.cc:463:11
  | #20 0x5aa516af19d9 in non-virtual thunk to Envoy::Http::ActiveStreamDecoderFilter::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/filter_manager.cc
  | #21 0x5aa5137ebda8 in Envoy::Router::UpstreamCodecFilter::CodecBridge::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_codec_filter.cc:162:23
  | #22 0x5aa5130bc9b4 in Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_wrappers.h:32:12
  | #23 0x5aa51701a99b in Envoy::Http::Http2::ConnectionImpl::StreamImpl::decodeData() third_party/envoy/src/source/common/http/http2/codec_impl.cc:519:17
  | #24 0x5aa51703799f in Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1175:13
  | #25 0x5aa517057040 in operator() third_party/envoy/src/source/common/http/http2/codec_impl.cc:1695:64
  | #26 0x5aa517057040 in Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_5::__invoke(nghttp2_session*, nghttp2_frame const*, void*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1694:19
  | #27 0x5aa517380334 in external_quiche_http2::adapter::CallbackVisitor::OnDataForStream(int, std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/callback_visitor.cc:259:14
  | #28 0x5aa51734e773 in external_quiche_http2::adapter::callbacks::OnDataChunk(nghttp2_session*, unsigned char, int, unsigned char const*, unsigned long, void*) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_callbacks.cc:277:32
  | #29 0x5aa5173b2b51 in nghttp2_session_mem_recv third_party/nghttp2/src/lib/nghttp2_session.c:7170:18
  | #30 0x5aa51734aefd in external_quiche_http2::adapter::NgHttp2Session::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_session.cc:35:10
  | #31 0x5aa517344b19 in external_quiche_http2::adapter::NgHttp2Adapter::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_adapter.cc:74:45
  | #32 0x5aa51703170f in Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc:956:20
  | #33 0x5aa5170342b0 in virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc
  | #34 0x5aa513801292 in Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/codec_client.cc:173:33
  | #35 0x5aa5138073e1 in Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_client.h:211:15
  | #36 0x5aa5171af927 in Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) third_party/envoy/src/source/common/network/filter_manager_impl.cc:95:48
  | #37 0x5aa5171b0097 in Envoy::Network::FilterManagerImpl::onRead() third_party/envoy/src/source/common/network/filter_manager_impl.cc:105:3
  | #38 0x5aa5171887a0 in Envoy::Network::ConnectionImpl::onRead(unsigned long) third_party/envoy/src/source/common/network/connection_impl.cc:345:19
  | #39 0x5aa51719926f in Envoy::Network::ConnectionImpl::onReadReady() third_party/envoy/src/source/common/network/connection_impl.cc:651:5
  | #40 0x5aa5171929b8 in Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) third_party/envoy/src/source/common/network/connection_impl.cc:602:5
  | #41 0x5aa5171a7f05 in operator() third_party/envoy/src/source/common/network/connection_impl.cc:94:54
  | #42 0x5aa5171a7f05 in __invoke<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
  | #43 0x5aa5171a7f05 in __call<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
  | #44 0x5aa5171a7f05 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
  | #45 0x5aa5171a7f05 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__msan::unique_ptr<Envoy::Network::ConnectionSocket, std::__msan::default_deleteEnvoy::Network::ConnectionSocket>&&, std::__msan::unique_ptr<Envoy::Network::TransportSocket, std::__msan::default_deleteEnvoy::Network::TransportSocket>&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
  | #46 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
  | #47 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
  | #48 0x5aa517130576 in operator() third_party/envoy/src/source/common/event/dispatcher_impl.cc:184:9
  | #49 0x5aa517130576 in __invoke<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
  | #50 0x5aa517130576 in __call<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
  | #51 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
  | #52 0x5aa517130576 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__msan::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
  | #53 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
  | #54 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
  | #55 0x5aa51713710d in Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(unsigned int) third_party/envoy/src/source/common/event/file_event_impl.cc:161:3
  | #56 0x5aa5171377fb in operator() third_party/envoy/src/source/common/event/file_event_impl.cc:82:16
  | #57 0x5aa5171377fb in Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) third_party/envoy/src/source/common/event/file_event_impl.cc:66:7
  | #58 0x5aa5175af2c7 in event_persist_closure third_party/libevent/src/event.c:1639:9
  | #59 0x5aa5175af2c7 in event_process_active_single_queue third_party/libevent/src/event.c:1698:4
  | #60 0x5aa5175a2132 in event_process_active third_party/libevent/src/event.c:1799:9
  | #61 0x5aa5175a2132 in event_base_loop third_party/libevent/src/event.c:2041:12
  | #62 0x5aa517574e53 in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/libevent_scheduler.cc:60:3
  | #63 0x5aa51712739a in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/dispatcher_impl.cc:299:19
  | #64 0x5aa511f0bbce in Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&) third_party/envoy/src/source/server/worker_impl.cc:148:16
  | #65 0x5aa511f0d318 in operator() third_party/envoy/src/source/server/worker_impl.cc:115:42
  | #66 0x5aa511f0d318 in __invoke<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
  | #67 0x5aa511f0d318 in __call<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
  | #68 0x5aa511f0d318 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
  | #69 0x5aa511f0d318 in void std::__msan::__function::__policy_invoker<void ()>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&)::$_0, void ()>>(std::__msan::__function::__policy_storage const*) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
  | #70 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
  | #71 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
  | #72 0x5aa5178a74d2 in operator() third_party/envoy/src/source/common/common/posix/thread_impl.cc:49:11
  | #73 0x5aa5178a74d2 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__msan::function<void ()>, std::__msan::optionalEnvoy::Thread::Options const&)::'lambda'(void*)::__invoke(void*) third_party/envoy/src/source/common/common/posix/thread_impl.cc:48:9
  | #74 0x79b7c0838d62 in start_thread (/usr/grte/v5/lib64/libpthread.so.0+0xbd62) (BuildId: 598ce51a74ecc39ad11a39e9c549e191)
  |  
  | MemorySanitizer can not provide additional info.
  | SUMMARY: MemorySanitizer: ABRT (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d) in raise
  | ==1458622==ABORTING

Command: /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test
Time ran: 57.5866904258728
INFO: google3-libFuzzer:
google3 flags should be passed as --flag=value
libFuzzer flags should be passed as -flag=value
INFO: Running in fuzzing mode, google3 logging disabled
INFO: Use --minloglevel=0 to re-enable logging
INFO: found LLVMFuzzerCustomMutator (0x5aa5109285c0). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed:4231534795
INFO: Loaded 1 modules (1767040 inline 8-bit counters): 1767040 [0x5aa51a28ec50, 0x5aa51a43e2d0),
INFO: Loaded 1 PC tables (1767040 PCs): 1767040 [0x5aa51a43e2d0,0x5aa51bf34ad0),
/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test: Running 1 inputs 100 time(s) each.
Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test
MemorySanitizer:DEADLYSIGNAL
==1458622==ERROR: MemorySanitizer: ABRT on unknown address 0x0539001641be (pc 0x79b7c07162fa bp 0x79b7b4756b60 sp 0x79b7b47569b8 T1458646)
#0 0x79b7c07162fa in raise (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d)
#1 0x5aa516aa81a3 in Envoy::Http::ConnectionManagerImpl::ActiveStream::routeCacheBlocked() const third_party/envoy/src/source/common/http/conn_manager_impl.h:310:7
#2 0x5aa516ab95a2 in Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc:1966:7
#3 0x5aa516ab99f3 in non-virtual thunk to Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc
#4 0x5aa510a2ad96 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::clearRouteCache(envoy::service::ext_proc::v3::CommonResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:375:47
#5 0x5aa510a2f2f1 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::handleBodyResponse(envoy::service::ext_proc::v3::BodyResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:328:7
#6 0x5aa510a153b6 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc:628:41
#7 0x5aa510a18c93 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc
#8 0x5aa510a01537 in onReceiveMessage third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc:53:14
#9 0x5aa510a01537 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorStreamImpl::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc
#10 0x5aa510a01fb8 in Envoy::Grpc::AsyncStreamCallbacksenvoy::service::ext_proc::v3::ProcessingResponse::onReceiveMessageRaw(std::__msan::unique_ptr<Envoy::Buffer::Instance, std::__msan::default_deleteEnvoy::Buffer::Instance>&&) third_party/envoy/src/source/common/grpc/typed_async_client.h:98:5
#11 0x5aa51314e664 in Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc:153:21
#12 0x5aa51314e836 in non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc
#13 0x5aa51316009e in Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/async_client_impl.cc:142:21
#14 0x5aa51379db66 in Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc:1606:15
#15 0x5aa51379e016 in non-virtual thunk to Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc
#16 0x5aa5137c7ad9 in Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.cc:311:11
#17 0x5aa5137dc48f in Envoy::Router::UpstreamRequestFilterManagerCallbacks::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.h:273:23
#18 0x5aa516af1223 in Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState) third_party/envoy/src/source/common/http/filter_manager.cc:1356:29
#19 0x5aa516af19d9 in encodeData third_party/envoy/src/source/common/http/filter_manager.cc:463:11
#20 0x5aa516af19d9 in non-virtual thunk to Envoy::Http::ActiveStreamDecoderFilter::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/filter_manager.cc
#21 0x5aa5137ebda8 in Envoy::Router::UpstreamCodecFilter::CodecBridge::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_codec_filter.cc:162:23
#22 0x5aa5130bc9b4 in Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_wrappers.h:32:12
#23 0x5aa51701a99b in Envoy::Http::Http2::ConnectionImpl::StreamImpl::decodeData() third_party/envoy/src/source/common/http/http2/codec_impl.cc:519:17
#24 0x5aa51703799f in Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1175:13
#25 0x5aa517057040 in operator() third_party/envoy/src/source/common/http/http2/codec_impl.cc:1695:64
#26 0x5aa517057040 in Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_5::__invoke(nghttp2_session*, nghttp2_frame const*, void*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1694:19
#27 0x5aa517380334 in external_quiche_http2::adapter::CallbackVisitor::OnDataForStream(int, std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/callback_visitor.cc:259:14
#28 0x5aa51734e773 in external_quiche_http2::adapter::callbacks::OnDataChunk(nghttp2_session*, unsigned char, int, unsigned char const*, unsigned long, void*) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_callbacks.cc:277:32
#29 0x5aa5173b2b51 in nghttp2_session_mem_recv third_party/nghttp2/src/lib/nghttp2_session.c:7170:18
#30 0x5aa51734aefd in external_quiche_http2::adapter::NgHttp2Session::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_session.cc:35:10
#31 0x5aa517344b19 in external_quiche_http2::adapter::NgHttp2Adapter::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_adapter.cc:74:45
#32 0x5aa51703170f in Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc:956:20
#33 0x5aa5170342b0 in virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc
#34 0x5aa513801292 in Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/codec_client.cc:173:33
#35 0x5aa5138073e1 in Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_client.h:211:15
#36 0x5aa5171af927 in Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) third_party/envoy/src/source/common/network/filter_manager_impl.cc:95:48
#37 0x5aa5171b0097 in Envoy::Network::FilterManagerImpl::onRead() third_party/envoy/src/source/common/network/filter_manager_impl.cc:105:3
#38 0x5aa5171887a0 in Envoy::Network::ConnectionImpl::onRead(unsigned long) third_party/envoy/src/source/common/network/connection_impl.cc:345:19
#39 0x5aa51719926f in Envoy::Network::ConnectionImpl::onReadReady() third_party/envoy/src/source/common/network/connection_impl.cc:651:5
#40 0x5aa5171929b8 in Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) third_party/envoy/src/source/common/network/connection_impl.cc:602:5
#41 0x5aa5171a7f05 in operator() third_party/envoy/src/source/common/network/connection_impl.cc:94:54
#42 0x5aa5171a7f05 in __invoke<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
#43 0x5aa5171a7f05 in __call<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
#44 0x5aa5171a7f05 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
#45 0x5aa5171a7f05 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__msan::unique_ptr<Envoy::Network::ConnectionSocket, std::__msan::default_deleteEnvoy::Network::ConnectionSocket>&&, std::__msan::unique_ptr<Envoy::Network::TransportSocket, std::__msan::default_deleteEnvoy::Network::TransportSocket>&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
#46 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
#47 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
#48 0x5aa517130576 in operator() third_party/envoy/src/source/common/event/dispatcher_impl.cc:184:9
#49 0x5aa517130576 in __invoke<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
#50 0x5aa517130576 in __call<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
#51 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
#52 0x5aa517130576 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__msan::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
#53 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
#54 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
#55 0x5aa51713710d in Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(unsigned int) third_party/envoy/src/source/common/event/file_event_impl.cc:161:3
#56 0x5aa5171377fb in operator() third_party/envoy/src/source/common/event/file_event_impl.cc:82:16
#57 0x5aa5171377fb in Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) third_party/envoy/src/source/common/event/file_event_impl.cc:66:7
#58 0x5aa5175af2c7 in event_persist_closure third_party/libevent/src/event.c:1639:9
#59 0x5aa5175af2c7 in event_process_active_single_queue third_party/libevent/src/event.c:1698:4
#60 0x5aa5175a2132 in event_process_active third_party/libevent/src/event.c:1799:9
#61 0x5aa5175a2132 in event_base_loop third_party/libevent/src/event.c:2041:12
#62 0x5aa517574e53 in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/libevent_scheduler.cc:60:3
#63 0x5aa51712739a in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/dispatcher_impl.cc:299:19
#64 0x5aa511f0bbce in Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&) third_party/envoy/src/source/server/worker_impl.cc:148:16
#65 0x5aa511f0d318 in operator() third_party/envoy/src/source/server/worker_impl.cc:115:42
#66 0x5aa511f0d318 in __invoke<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
#67 0x5aa511f0d318 in __call<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
#68 0x5aa511f0d318 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
#69 0x5aa511f0d318 in void std::__msan::__function::__policy_invoker<void ()>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&)::$_0, void ()>>(std::__msan::__function::__policy_storage const*) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
#70 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
#71 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
#72 0x5aa5178a74d2 in operator() third_party/envoy/src/source/common/common/posix/thread_impl.cc:49:11
#73 0x5aa5178a74d2 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__msan::function<void ()>, std::__msan::optionalEnvoy::Thread::Options const&)::'lambda'(void*)::__invoke(void*) third_party/envoy/src/source/common/common/posix/thread_impl.cc:48:9
#74 0x79b7c0838d62 in start_thread (/usr/grte/v5/lib64/libpthread.so.0+0xbd62) (BuildId: 598ce51a74ecc39ad11a39e9c549e191)
MemorySanitizer can not provide additional info.
SUMMARY: MemorySanitizer: ABRT (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d) in raise
==1458622==ABORTING

@yanjunxiang-google
Copy link
Contributor Author

This is the fuzzer test case:

ext_proc_data: "scterpc_cre:csp_yilooo0\n\000*!pV1:ae!FoFFF,F\n"

@wbpcode
Copy link
Member

wbpcode commented May 31, 2023

clearRouteCache on response path to downstream client is just a no-op (since route selection is not needed on response path

Nope, the refresh in response phase still may changes the roue and effects the per filter config/metadata... etc that filters may need to use. Apparently it's wrong, so, we should never do this refresh in the response phase.

@tyxia
Copy link
Member

tyxia commented May 31, 2023

clearRouteCache on response path to downstream client is just a no-op (since route selection is not needed on response path

Nope, the refresh in response phase still may changes the roue and effects the per filter config/metadata... etc that filters may need to use. Apparently it's wrong, so, we should never do this refresh in the response phase.

Yes, per filter config can be used in response path. What I meant about "no-op" is that before your PR #26045, such behavior will not trigger the Envoy_Bug. Thanks for sharing more context!

Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait

source/extensions/filters/http/ext_proc/processor_state.cc Outdated Show resolved Hide resolved
Signed-off-by: Yanjun Xiang <[email protected]>
@repokitteh-read-only
Copy link

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #27657 was synchronize by yanjunxiang-google.

see: more, trace.

Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait

api/envoy/service/ext_proc/v3/external_processor.proto Outdated Show resolved Hide resolved
Signed-off-by: Yanjun Xiang <[email protected]>
Copy link
Member

@tyxia tyxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, modulo nit.

Thanks!

Signed-off-by: Yanjun Xiang <[email protected]>
Signed-off-by: Yanjun Xiang <[email protected]>
tyxia
tyxia previously approved these changes Jun 5, 2023
Copy link
Member

@tyxia tyxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@yanjunxiang-google
Copy link
Contributor Author

@envoyproxy/api-shepherds PTAL

Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait-any

api/envoy/service/ext_proc/v3/external_processor.proto Outdated Show resolved Hide resolved
@yanjunxiang-google
Copy link
Contributor Author

Kind ping!

@stevenzzzz
Copy link
Contributor

LGTM

@yanjunxiang-google
Copy link
Contributor Author

Kind ping!

@yanavlasov yanavlasov merged commit ce456d5 into envoyproxy:main Jun 12, 2023
@yanjunxiang-google yanjunxiang-google deleted the ext_proc_clear_route_cache branch June 13, 2023 01:24
asheryerm pushed a commit to asheryerm/envoy that referenced this pull request Jul 5, 2023
…roxy#27657)

* ext_proc fuzzer test trigger ENVOY_BUG when clear route cache for upstream response

Signed-off-by: Yanjun Xiang <[email protected]>
Signed-off-by: asheryer <[email protected]>
reskin89 pushed a commit to reskin89/envoy that referenced this pull request Jul 11, 2023
…roxy#27657)

* ext_proc fuzzer test trigger ENVOY_BUG when clear route cache for upstream response

Signed-off-by: Yanjun Xiang <[email protected]>
Signed-off-by: Ryan Eskin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants