Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

yanjunxiang-google · 2023-05-08T19:21:36Z

Properly handle dataplane exception when received response timeout Duration is too large.

Also adding PGVs for ext_proc filter config.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

…void ext_proc filter fuzzer crash due to duration config out-of-bounds. Signed-off-by: Yanjun Xiang <[email protected]>

repokitteh-read-only · 2023-05-08T19:21:49Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @mattklein123
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #27260 was opened by yanjunxiang-google.

see: more, trace.

yanjunxiang-google · 2023-05-08T19:21:52Z

The crash traceback is:
./bazel-bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz crash-caae576f1c5a5c4bd6f831dd7d159b2504f476b0
INFO: found LLVMFuzzerCustomMutator (0x298ec70). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3719293196
INFO: Loaded 1 modules (1566644 inline 8-bit counters): 1566644 [0xae23d00, 0xafa24b4),
INFO: Loaded 1 PC tables (1566644 PCs): 1566644 [0xafa24b8,0xc789ff8),
./bazel-bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz: Running 1 inputs 1 time(s) each.
Running: crash-caae576f1c5a5c4bd6f831dd7d159b2504f476b0
terminate called after throwing an instance of 'Envoy::DurationUtil::OutOfRangeException'
what(): Duration out-of-range: seconds: 137438953472
nanos: 655360

==1818001== ERROR: libFuzzer: deadly signal
error: failed to decompress '.debug_aranges', zlib is not available
error: failed to decompress '.debug_info', zlib is not available
error: failed to decompress '.debug_abbrev', zlib is not available
error: failed to decompress '.debug_line', zlib is not available
error: failed to decompress '.debug_str', zlib is not available
error: failed to decompress '.debug_line_str', zlib is not available
error: failed to decompress '.debug_loclists', zlib is not available
error: failed to decompress '.debug_rnglists', zlib is not available
#0 0x2953121 in __sanitizer_print_stack_trace /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
#1 0x2870528 in fuzzer::PrintStackTrace() /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
#2 0x28551e3 in fuzzer::Fuzzer::CrashCallback() /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3
#3 0x7f2193c1bf8f (/lib/x86_64-linux-gnu/libc.so.6+0x3bf8f) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#4 0x7f2193c6accb (/lib/x86_64-linux-gnu/libc.so.6+0x8accb) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#5 0x7f2193c1bef1 in raise (/lib/x86_64-linux-gnu/libc.so.6+0x3bef1) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#6 0x7f2193c06471 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x26471) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#7 0x7f219409d918 (/lib/x86_64-linux-gnu/libstdc++.so.6+0x9d918) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#8 0x7f21940a8e19 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa8e19) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#9 0x7f21940a8e84 in std::terminate() (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa8e84) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#10 0x7f21940a90d7 in __cxa_throw (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa90d7) (BuildId: da7a43b9a9187f8d6fd68c66590dfe908423be34)
#11 0x69b6048 in Envoy::(anonymous namespace)::validateDuration(google::protobuf::Duration const&, long) /proc/self/cwd/source/common/protobuf/utility.cc:654:5
#12 0x69a82f2 in Envoy::DurationUtil::durationToMilliseconds(google::protobuf::Duration const&) /proc/self/cwd/source/common/protobuf/utility.cc:669:3
#13 0x2b6b64b in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__1::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__1::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse >&&) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:562:18
#14 0x2a8d7ed in std::__1::__function::__func<Envoy::Extensions::HttpFilters::ExtProc::UnitTestFuzz::TestOneProtoInput(envoy::extensions::filters::http::ext_proc::unit_test_fuzz::ExtProcUnitTestCase const&)::$_0::operator()(Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorCallbacks&, envoy::config::core::v3::GrpcService const&, Envoy::StreamInfo::StreamInfo const&) const::'lambda'(envoy::service::ext_proc::v3::ProcessingRequest&&, bool), std::__1::allocator<Envoy::Extensions::HttpFilters::ExtProc::UnitTestFuzz::TestOneProtoInput(envoy::extensions::filters::http::ext_proc::unit_test_fuzz::ExtProcUnitTestCase const&)::$_0::operator()(Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorCallbacks&, envoy::config::core::v3::GrpcService const&, Envoy::StreamInfo::StreamInfo const&) const::'lambda'(envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>, void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::operator()(envoy::service::ext_proc::v3::ProcessingRequest&&, bool&&) /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:81:31
#15 0x2a7af17 in std::__1::__function::__value_func<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::operator()(envoy::service::ext_proc::v3::ProcessingRequest&&, bool&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
#16 0x2a7adeb in std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::operator()(envoy::service::ext_proc::v3::ProcessingRequest&&, bool) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
#17 0x2a7acb2 in decltype(std::forward<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&>(fp)(std::get<0ul>(std::forward<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(fp0)), std::get<1ul>(std::forward<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(fp0)))) testing::internal::ApplyImpl<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>, 0ul, 1ul>(std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>&&, testing::internal::IndexSequence<0ul, 1ul>) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/internal/gmock-internal-utils.h:420:10
#18 0x2a7abf0 in decltype(ApplyImpl(std::forward<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&>(fp), std::forward<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(fp0), (testing::internal::MakeIndexSequence<std::tuple_size<std::__1::remove_reference<std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >::type>::value>)())) testing::internal::Apply<std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool> >(std::__1::function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>&&) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/internal/gmock-internal-utils.h:429:10
#19 0x2a7a30d in testing::Action<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::Perform(std::__1::tuple<envoy::service::ext_proc::v3::ProcessingRequest&&, bool>) const /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:497:12
#20 0x2a7b013 in testing::internal::ActionResultHolder* testing::internal::ActionResultHolder::PerformAction<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>(testing::Action<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)> const&, testing::internal::Function<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::ArgumentTuple&&) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1441:12
#21 0x2a78c32 in testing::internal::FunctionMocker<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::UntypedPerformAction(void const*, void*) const /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1556:12
#22 0x8da6fa0 in testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*) /proc/self/cwd/external/com_google_googletest/googlemock/src/gmock-spec-builders.cc:452:24
#23 0x2a84c05 in testing::internal::FunctionMocker<void (envoy::service::ext_proc::v3::ProcessingRequest&&, bool)>::Invoke(envoy::service::ext_proc::v3::ProcessingRequest&&, bool) /proc/self/cwd/external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1593:15
#24 0x2a782ec in Envoy::Extensions::HttpFilters::ExtProc::UnitTestFuzz::MockStream::send(envoy::service::ext_proc::v3::ProcessingRequest&&, bool) /proc/self/cwd/./test/extensions/filters/http/ext_proc/unit_test_fuzz/mocks.h:20:3
#25 0x2b5a2c5 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onHeaders(Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState&, Envoy::Http::RequestOrResponseHeaderMap&, bool) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:176:12
#26 0x2b5b8d8 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:194:23
#27 0x2ad1855 in Envoy::Http::FilterHeadersStatus Envoy::Extensions::HttpFilters::HttpFilterFuzzer::sendHeadersEnvoy::Http::StreamDecoderFilter(Envoy::Http::StreamDecoderFilter*, test::fuzz::HttpData const&, bool) /proc/self/cwd/./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:145:46
#28 0x2995b55 in void Envoy::Extensions::HttpFilters::HttpFilterFuzzer::runDataEnvoy::Http::StreamDecoderFilter(Envoy::Http::StreamDecoderFilter*, test::fuzz::HttpData const&) /proc/self/cwd/./test/extensions/filters/http/common/fuzz/http_filter_fuzzer.h:102:31
#29 0x298ffeb in LLVMFuzzerTestOneInput /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:88:10
#30 0x2856783 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
#31 0x2841212 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
#32 0x2846abc in fuzzer::FuzzerDriver(int*, char***, int ()(unsigned char const, unsigned long)) /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
#33 0x2870ce2 in main /local/mnt/workspace/bcain_clang_hu-bcain-lv_22036/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
#34 0x7f2193c07189 (/lib/x86_64-linux-gnu/libc.so.6+0x27189) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#35 0x7f2193c07244 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x27244) (BuildId: e144007f35d794adf218479af5ddcb2a11a2c583)
#36 0x28373ad in _start (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/51ff81aa23c8ee714a5106cc912b2104/execroot/envoy/bazel-out/k8-dbg/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x28373ad)

NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal

yanjunxiang-google · 2023-05-08T19:23:58Z

/assign @yanavlasov @adisuissa @htuch

…imeout Signed-off-by: Yanjun Xiang <[email protected]>

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto

source/extensions/filters/http/ext_proc/ext_proc.cc

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto

adisuissa

Thanks!
I think the 1hr limit should be ok for most use-cases.
Left a small comment on the handling of the override as part of the response case.

source/extensions/filters/http/ext_proc/ext_proc.cc

yanavlasov

/wait-any

yanavlasov

/wait

source/extensions/filters/http/ext_proc/ext_proc.cc

Signed-off-by: Yanjun Xiang <[email protected]>

yanavlasov

/wait

source/common/protobuf/utility.cc

Signed-off-by: Yanjun Xiang <[email protected]>

source/common/protobuf/utility.cc

Signed-off-by: Yanjun Xiang <[email protected]>

adisuissa

Thanks! A few minor comments, but overall looks good.

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto

source/common/protobuf/utility.cc

source/extensions/filters/http/ext_proc/ext_proc.cc

Signed-off-by: Yanjun Xiang <[email protected]>

adisuissa

Thanks!
Let's revert the descriptions updates as they don't add anything, and we can approve this.

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto

…Duration is too large (envoyproxy#27260) * Adding ext_proc filter config and response timeout Duration PGVs to avoid ext_proc filter fuzzer crash due to duration config out-of-bounds. Signed-off-by: Yanjun Xiang <[email protected]>

…Duration is too large (envoyproxy#27260) * Adding ext_proc filter config and response timeout Duration PGVs to avoid ext_proc filter fuzzer crash due to duration config out-of-bounds. Signed-off-by: Yanjun Xiang <[email protected]> Signed-off-by: Ryan Eskin <[email protected]>

Adding ext_proc filter config and response timeout Duration PGVs to a…

adb6e5a

…void ext_proc filter fuzzer crash due to duration config out-of-bounds. Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google requested a review from snowp as a code owner May 8, 2023 19:21

repokitteh-read-only bot added the api label May 8, 2023

repokitteh-read-only bot assigned mattklein123 May 8, 2023

repokitteh-read-only bot assigned adisuissa, htuch and yanavlasov May 8, 2023

Adding exception handling for response with large override_message_t…

2554501

…imeout Signed-off-by: Yanjun Xiang <[email protected]>

htuch reviewed May 9, 2023

View reviewed changes

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved

htuch previously approved these changes May 9, 2023

View reviewed changes

yanjunxiang-google commented May 9, 2023

View reviewed changes

source/extensions/filters/http/ext_proc/ext_proc.cc Outdated Show resolved Hide resolved

yanjunxiang-google changed the title ~~Adding ext_proc filter config and response timeout Duration PGVs~~ Envoy ext_proc filter throw exception when received response timeout Duration is too large May 9, 2023

yanjunxiang-google commented May 9, 2023

View reviewed changes

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved

adisuissa reviewed May 9, 2023

View reviewed changes

source/extensions/filters/http/ext_proc/ext_proc.cc Outdated Show resolved Hide resolved

mattklein123 removed their assignment May 9, 2023

yanavlasov reviewed May 9, 2023

View reviewed changes

repokitteh-read-only bot added waiting:any and removed waiting:any labels May 9, 2023

yanavlasov requested changes May 9, 2023

View reviewed changes

source/extensions/filters/http/ext_proc/ext_proc.cc Outdated Show resolved Hide resolved

repokitteh-read-only bot added the waiting label May 9, 2023

addressing comments

f796853

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google dismissed htuch’s stale review via f796853 May 10, 2023 01:10

repokitteh-read-only bot removed the waiting label May 10, 2023

yanavlasov requested changes May 10, 2023

View reviewed changes

source/common/protobuf/utility.cc Outdated Show resolved Hide resolved

repokitteh-read-only bot added the waiting label May 10, 2023

adding unit test for the newly added NoThrow utility function

b3187e1

Signed-off-by: Yanjun Xiang <[email protected]>

repokitteh-read-only bot removed the waiting label May 10, 2023

addressing comments

217b185

Signed-off-by: Yanjun Xiang <[email protected]>

adisuissa reviewed May 10, 2023

View reviewed changes

source/common/protobuf/utility.cc Outdated Show resolved Hide resolved

addressing comments

9ccba29

Signed-off-by: Yanjun Xiang <[email protected]>

adisuissa reviewed May 11, 2023

View reviewed changes

addressing comments

1b25122

Signed-off-by: Yanjun Xiang <[email protected]>

yanavlasov approved these changes May 12, 2023

View reviewed changes

yanavlasov enabled auto-merge (squash) May 12, 2023 15:04

yanavlasov merged commit b14b3e8 into envoyproxy:main May 12, 2023

adisuissa reviewed May 12, 2023

View reviewed changes

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved

api/envoy/extensions/filters/http/ext_proc/v3/ext_proc.proto Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

yanjunxiang-google commented May 8, 2023 •

edited

Loading

repokitteh-read-only bot commented May 8, 2023

yanjunxiang-google commented May 8, 2023

yanjunxiang-google commented May 8, 2023

adisuissa left a comment

yanavlasov left a comment

yanavlasov left a comment

yanavlasov left a comment

adisuissa left a comment

adisuissa left a comment

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

Conversation

yanjunxiang-google commented May 8, 2023 • edited Loading

repokitteh-read-only bot commented May 8, 2023

yanjunxiang-google commented May 8, 2023

yanjunxiang-google commented May 8, 2023

adisuissa left a comment

Choose a reason for hiding this comment

yanavlasov left a comment

Choose a reason for hiding this comment

yanavlasov left a comment

Choose a reason for hiding this comment

yanavlasov left a comment

Choose a reason for hiding this comment

adisuissa left a comment

Choose a reason for hiding this comment

adisuissa left a comment

Choose a reason for hiding this comment

yanjunxiang-google commented May 8, 2023 •

edited

Loading