aws: Memory overwrite protobuf issue triggered by PackFrom

[nbaws]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
[email protected] where the issue will be triaged appropriately.

Title: Memory overwrite occurring during debug build/testing triggered by protobuf PackFrom

Description:

This line

envoy/source/extensions/common/aws/utility.cc

Line 367 in f0c2329

socket->mutable_typed_config()->PackFrom(tls_socket);

Causes a memory overwrite corrupting test cases from here:

envoy/test/extensions/common/aws/credentials_provider_impl_test.cc

Line 2325 in f0c2329

TEST_F(WebIdentityCredentialsProviderTest, FailedFetchingDocument) {

Repro steps:
Needs lldb to repro.

In most cases

envoy/test/extensions/common/aws/credentials_provider_impl_test.cc

Line 2162 in f0c2329

: api_(Api::createApiForTest(time_system_)), raw_metadata_fetcher_(new MockMetadataFetcher) {

raw_metadata_fetcher_ will be overwritten and cause a test crash.

I'm not familiar enough with the internals of protobuf to know whether the protobuf setup in utility.cc is correct, however it appears to be common from a quick scan through other code.
I'm also not sure if this line is specifically the issue, or its prior to this line in the cluster setup that is the problem.

Would really appreciate some guidance on how to troubleshoot this any further. Unfortunately this crash is highly memory dependent and doesn't occur from a main build, so this is reproducible only in my fork.

Call Stack:

This call stack is generated by watchpoint on raw_metadata_fetcher_

Process exited with code 9.
Console is in 'commands' mode, prefix expressions with '?'.
Launching: /build/.cache/bazel/_bazel_vscode/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-dbg/bin/test/extensions/common/aws/credentials_provider_impl_test -l trace
Launched process 56433
w s e -- 0x26a0bfd60000
Watchpoint created: Watchpoint 1: addr = 0x26a0bfd60000 size = 8 state = enabled type = w

    watchpoint spec = '0x26a0bfd60000'
    new value: 95075568

Watchpoint 1 hit:
old value: 95075568
new value: 0
bt
* thread #1, name = 'credentials_pro', stop reason = watchpoint 1
  * frame #0: 0x00007ffff7dd8df4 libc.so.6`___lldb_unnamed_symbol3474 + 100
    frame #1: 0x00000000055b7064 credentials_provider_impl_test`std::__1::__unique_if<google::protobuf::internal::FlatAllocator>::__unique_single std::__1::make_unique<google::protobuf::internal::FlatAllocator>() at unique_ptr.h:725:32
    frame #2: 0x00000000055664fd credentials_provider_impl_test`google::protobuf::DescriptorBuilder::BuildFile(this=0x000026a0bfd48000, original_proto=0x000026a0bfd600e0) at descriptor.cc:5631:16
    frame #3: 0x000000000555bd15 credentials_provider_impl_test`google::protobuf::DescriptorPool::BuildFileFromDatabase(this=0x000026a0bfc04000, proto=0x000026a0bfd600e0) const at descriptor.cc:4553:13
    frame #4: 0x0000000005559bea credentials_provider_impl_test`google::protobuf::DescriptorPool::TryFindFileInFallbackDatabase(this=0x000026a0bfc04000, name="envoy/extensions/transport_sockets/tls/v3/tls.proto") const at descriptor.cc:2565:7
    frame #5: 0x0000000005559a28 credentials_provider_impl_test`google::protobuf::DescriptorPool::FindFileByName(this=0x000026a0bfc04000, name="envoy/extensions/transport_sockets/tls/v3/tls.proto") const at descriptor.cc:2145:7
    frame #6: 0x00000000056930e2 credentials_provider_impl_test`google::protobuf::(anonymous namespace)::AssignDescriptorsImpl(table=0x0000000005c87b10, eager=false) at generated_message_reflection.cc:3626:50
    frame #7: 0x0000000005699174 credentials_provider_impl_test`google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2::operator()(this=((credentials_provider_impl_test`descriptor_table_envoy_2fextensions_2ftransport_5fsockets_2ftls_2fv3_2ftls_2eproto_getter() at tls.pb.cc:525))) const at generated_message_reflection.cc:3707:5
    frame #8: 0x0000000005699135 credentials_provider_impl_test`decltype(static_cast<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2>(fp)()) std::__1::__invoke<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2>(__f=((credentials_provider_impl_test`descriptor_table_envoy_2fextensions_2ftransport_5fsockets_2ftls_2fv3_2ftls_2eproto_getter() at tls.pb.cc:525))) at type_traits:3640:23
    frame #9: 0x000000000569910d credentials_provider_impl_test`std::__1::invoke_result<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2>::type std::__1::invoke<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2>(__f=((credentials_provider_impl_test`descriptor_table_envoy_2fextensions_2ftransport_5fsockets_2ftls_2fv3_2ftls_2eproto_getter() at tls.pb.cc:525))) at invoke.h:93:12
    frame #10: 0x000000000569905a credentials_provider_impl_test`void absl::lts_20230802::base_internal::CallOnceImpl<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2>(control=0x0000000005cbe7ac, scheduling_mode=SCHEDULE_COOPERATIVE_AND_KERNEL, fn=((credentials_provider_impl_test`descriptor_table_envoy_2fextensions_2ftransport_5fsockets_2ftls_2fv3_2ftls_2eproto_getter() at tls.pb.cc:525))) at call_once.h:178:5
    frame #11: 0x0000000005692f0d credentials_provider_impl_test`void absl::lts_20230802::call_once<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* (*)(), absl::lts_20230802::once_flag*, google::protobuf::Metadata const&)::$_2>(flag=0x0000000005cbe7ac, fn=((credentials_provider_impl_test`descriptor_table_envoy_2fextensions_2ftransport_5fsockets_2ftls_2fv3_2ftls_2eproto_getter() at tls.pb.cc:525))) at call_once.h:210:5
    frame #12: 0x0000000005692e89 credentials_provider_impl_test`google::protobuf::internal::AssignDescriptors(table=(credentials_provider_impl_test`descriptor_table_envoy_2fextensions_2ftransport_5fsockets_2ftls_2fv3_2ftls_2eproto_getter() at tls.pb.cc:525), once=0x0000000005cbe7ac, metadata=0x0000000005cbe730) at generated_message_reflection.cc:3704:3
    frame #13: 0x0000000004e23b06 credentials_provider_impl_test`envoy::extensions::transport_sockets::tls::v3::UpstreamTlsContext::GetMetadata(this=0x00007fffffffbb20) const at tls.pb.cc:924:10
    frame #14: 0x0000000002c07306 credentials_provider_impl_test`google::protobuf::Message::GetDescriptor(this=0x00007fffffffbb20) const at message.h:354:52
    frame #15: 0x0000000005555c6a credentials_provider_impl_test`google::protobuf::internal::AnyMetadata::PackFrom(this=0x000026a0bfdb8e68, arena=0x0000000000000000, message=0x00007fffffffbb20, type_url_prefix="type.googleapis.com/") at any.cc:29:26
    frame #16: 0x0000000005555c25 credentials_provider_impl_test`google::protobuf::internal::AnyMetadata::PackFrom(this=0x000026a0bfdb8e68, arena=0x0000000000000000, message=0x00007fffffffbb20) at any.cc:23:10
    frame #17: 0x00000000023cfb1b credentials_provider_impl_test`google::protobuf::Any::PackFrom(this=0x000026a0bfdb8e40, message=0x00007fffffffbb20) at any.pb.h:139:34
    frame #18: 0x00000000023c8fb3 credentials_provider_impl_test`Envoy::Extensions::Common::Aws::Utility::addInternalClusterStatic(cm=0x000026a0bfd6dfc0, cluster_name="credentials_provider_cluster", cluster_type=Cluster_DiscoveryType_LOGICAL_DNS, uri="sts.region.amazonaws.com:443") at utility.cc:367:41
    frame #19: 0x0000000002381d7b credentials_provider_impl_test`Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2::operator()(this=(0x000026a0bfdd7b20)) const at credentials_provider_impl.cc:124:12
    frame #20: 0x0000000002381cc5 credentials_provider_impl_test`decltype(static_cast<Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2&>(fp)()) std::__1::__invoke<Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2&>(__f=(0x000026a0bfdd7b20)) at type_traits:3640:23
    frame #21: 0x0000000002381c8d credentials_provider_impl_test`std::__1::invoke_result<Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2&>::type std::__1::invoke<Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2&>(__f=(0x000026a0bfdd7b20)) at invoke.h:93:12
    frame #22: 0x0000000002381c2d credentials_provider_impl_test`void absl::lts_20230802::internal_any_invocable::InvokeR<void, Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2&, void>(f=(0x000026a0bfdd7b20)) at any_invocable.h:132:3
    frame #23: 0x0000000002381bc2 credentials_provider_impl_test`void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(Envoy::Api::Api&, Envoy::OptRef<Envoy::Server::Configuration::ServerFactoryContext>, std::__1::function<std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>> (Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&)> const&, std::__1::function<std::__1::unique_ptr<Envoy::Extensions::Common::Aws::MetadataFetcher, std::__1::default_delete<Envoy::Extensions::Common::Aws::MetadataFetcher>> (Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>)>, std::__1::basic_string_view<char, std::__1::char_traits<char>>, envoy::config::cluster::v3::Cluster_DiscoveryType, std::__1::basic_string_view<char, std::__1::char_traits<char>>, Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::$_2&>(state=0x00007fffffffc170) at any_invocable.h:310:10
    frame #24: 0x0000000004039b52 credentials_provider_impl_test`absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()(this=0x00007fffffffc170) at any_invocable.h:868:1
    frame #25: 0x0000000003ff7f2c credentials_provider_impl_test`Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1::operator()(this=(), cb=<unavailable>) const at mocks.cc:31:72
    frame #26: 0x0000000003ff7ec6 credentials_provider_impl_test`decltype(static_cast<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1&>(fp)(static_cast<absl::lts_20230802::AnyInvocable<void ()>>(fp0))) std::__1::__invoke<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1&, absl::lts_20230802::AnyInvocable<void ()>>(__f=({...}), __args=0x00007fffffffc2c0) at type_traits:3640:23
    frame #27: 0x0000000003ff7e62 credentials_provider_impl_test`void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1&, absl::lts_20230802::AnyInvocable<void ()>>(__args=({...}), __args=0x00007fffffffc2c0) at invoke.h:61:9
    frame #28: 0x0000000003ff7e22 credentials_provider_impl_test`std::__1::__function::__alloc_func<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1, std::__1::allocator<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1>, void (absl::lts_20230802::AnyInvocable<void ()>)>::operator()(this=0x000026a0bfd62a58, __arg=0x00007fffffffc2c0) at function.h:180:16
    frame #29: 0x0000000003ff6f91 credentials_provider_impl_test`std::__1::__function::__func<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1, std::__1::allocator<Envoy::Event::MockDispatcher::MockDispatcher(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)::$_1>, void (absl::lts_20230802::AnyInvocable<void ()>)>::operator()(this=0x000026a0bfd62a50, __arg=0x00007fffffffc2c0) at function.h:354:12
    frame #30: 0x000000000402119a credentials_provider_impl_test`std::__1::__function::__value_func<void (absl::lts_20230802::AnyInvocable<void ()>)>::operator()(this=0x000026a0bfd62a50, __args=0x00007fffffffc2c0) const at function.h:507:16
    frame #31: 0x0000000004021120 credentials_provider_impl_test`std::__1::function<void (absl::lts_20230802::AnyInvocable<void ()>)>::operator()(this=0x000026a0bfd62a50, __arg=<unavailable>) const at function.h:1184:12
    frame #32: 0x000000000402109b credentials_provider_impl_test`decltype(std::forward<std::__1::function<void (absl::lts_20230802::AnyInvocable<void ()>)> const&>(fp)(std::get<0ul>(std::forward<std::__1::tuple<absl::lts_20230802::AnyInvocable<void ()>>>(fp0)))) testing::internal::ApplyImpl<std::__1::function<void (absl::lts_20230802::AnyInvocable<void ()>)> const&, std::__1::tuple<absl::lts_20230802::AnyInvocable<void ()>>, 0ul>(f=0x000026a0bfd62a50, args=size=1, (null)=IndexSequence<0UL> @ 0x00007fffffffc2f8) at gmock-internal-utils.h:420:10
    frame #33: 0x0000000004020fd2 credentials_provider_impl_test`decltype(ApplyImpl(std::forward<std::__1::function<void (absl::lts_20230802::AnyInvocable<void ()>)> const&>(fp), std::forward<std::__1::tuple<absl::lts_20230802::AnyInvocable<void ()>>>(fp0), (testing::internal::MakeIndexSequence<std::tuple_size<std::__1::remove_reference<std::__1::tuple<absl::lts_20230802::AnyInvocable<void ()>>>::type>::value>)())) testing::internal::Apply<std::__1::function<void (absl::lts_20230802::AnyInvocable<void ()>)> const&, std::__1::tuple<absl::lts_20230802::AnyInvocable<void ()>>>(f=0x000026a0bfd62a50, args=size=1) at gmock-internal-utils.h:429:10
    frame #34: 0x0000000004020cd0 credentials_provider_impl_test`testing::Action<void (absl::lts_20230802::AnyInvocable<void ()>)>::Perform(this=0x000026a0bfd62a50, args=size=1) const at gmock-actions.h:497:12
    frame #35: 0x0000000004020a46 credentials_provider_impl_test`testing::internal::FunctionMocker<void (absl::lts_20230802::AnyInvocable<void ()>)>::PerformDefaultAction(this=0x000026a0bfd74470, args=size=1, call_description="Function call: post") const at gmock-spec-builders.h:1517:32
    frame #36: 0x0000000004020975 credentials_provider_impl_test`testing::internal::ActionResultHolder<void>* testing::internal::ActionResultHolder<void>::PerformDefaultAction<void (absl::lts_20230802::AnyInvocable<void ()>)>(func_mocker=0x000026a0bfd74470, args=size=1, call_description="Function call: post") at gmock-spec-builders.h:1432:18
    frame #37: 0x000000000402053d credentials_provider_impl_test`testing::internal::FunctionMocker<void (absl::lts_20230802::AnyInvocable<void ()>)>::UntypedPerformDefaultAction(this=0x000026a0bfd74470, untyped_args=0x00007fffffffcb80, call_description="Function call: post") const at gmock-spec-builders.h:1542:12
    frame #38: 0x0000000005818dd2 credentials_provider_impl_test`testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(this=0x000026a0bfd74470, untyped_args=0x00007fffffffcb80) at gmock-spec-builders.cc:409:20
    frame #39: 0x000000000251e4f1 credentials_provider_impl_test`testing::internal::FunctionMocker<void (absl::lts_20230802::AnyInvocable<void ()>)>::Invoke(this=0x000026a0bfd74470, args=<unavailable>) at gmock-spec-builders.h:1593:15
    frame #40: 0x000000000251c854 credentials_provider_impl_test`Envoy::Event::MockDispatcher::post(this=0x000026a0bfd74070, gmock_a0=testing::internal::ElemFromList<0, absl::lts_20230802::AnyInvocable<void ()> >::type @ 0x00007fffffffcd40) at mocks.h:138:3
    frame #41: 0x000000000236eb9d credentials_provider_impl_test`Envoy::Extensions::Common::Aws::MetadataCredentialsProviderBase::MetadataCredentialsProviderBase(this=0x000026a0bfdd7b20, api=0x000026a0bfc15500, context=(ptr_ = 0x000026a0bfd70cb0), fetch_metadata_using_curl=0x00007fffffffd1a0, create_metadata_fetcher_cb=Envoy::Extensions::Common::Aws::CreateMetadataFetcherCb @ 0x00007fffffffcff0, cluster_name="credentials_provider_cluster", cluster_type=Cluster_DiscoveryType_LOGICAL_DNS, uri="sts.region.amazonaws.com:443", receiver_state=Ready, initialization_timer=(__rep_ = 2)) at credentials_provider_impl.cc:123:38
    frame #42: 0x000000000237862b credentials_provider_impl_test`Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider::WebIdentityCredentialsProvider(this=0x000026a0bfdd7b20, api=0x000026a0bfc15500, context=(ptr_ = 0x000026a0bfd70cb0), fetch_metadata_using_curl=0x00007fffffffd1a0, create_metadata_fetcher_cb=Envoy::Extensions::Common::Aws::CreateMetadataFetcherCb @ 0x00007fffffffd170, token_file_path="/tmp/envoy_test_tmp.A4S5Ep/web_token_file", sts_endpoint="sts.region.amazonaws.com:443", role_arn="aws:iam::123456789012:role/arn", role_session_name="role-session-name", receiver_state=Ready, initialization_timer=(__rep_ = 2), cluster_name="credentials_provider_cluster") at credentials_provider_impl.cc:645:7
    frame #43: 0x00000000022f1fa2 credentials_provider_impl_test`Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider* std::__1::construct_at<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider, Envoy::Api::Api&, testing::NiceMock<Envoy::Server::Configuration::MockServerFactoryContext>&, Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&), Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, char const (&) [29], char const (&) [31], char const (&) [18], Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>&, char const (&) [29], Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider*>(__location=0x000026a0bfdd7b20, __args=0x000026a0bfc15500, __args=0x000026a0bfd70cb0, __args=(0x000026a0bfd6c000), __args=(0x000026a0bfd6c000), __args="/tmp/envoy_test_tmp.A4S5Ep/web_token_file", __args={\xddL1\0\0\0\0\0}, __args={\xd2\x86;\0\0\0\0\0}, __args={\07#\0\0\0\0\0}, __args=0x00007fffffffd7ac, __args=0x00007fffffffd7b8, __args={\x92\x956\0\0\0\0\0}) at construct_at.h:38:50
    frame #44: 0x00000000022f1bdc credentials_provider_impl_test`void std::__1::allocator_traits<std::__1::allocator<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider>>::construct<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider, Envoy::Api::Api&, testing::NiceMock<Envoy::Server::Configuration::MockServerFactoryContext>&, Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&), Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, char const (&) [29], char const (&) [31], char const (&) [18], Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>&, char const (&) [29], void, void>((null)=0x00007fffffffd348, __p=0x000026a0bfdd7b20, __args=0x000026a0bfc15500, __args=0x000026a0bfd70cb0, __args=(0x000026a0bfd6c000), __args=(0x000026a0bfd6c000), __args="/tmp/envoy_test_tmp.A4S5Ep/web_token_file", __args={\xddL1\0\0\0\0\0}, __args={\xd2\x86;\0\0\0\0\0}, __args={\07#\0\0\0\0\0}, __args=0x00007fffffffd7ac, __args=0x00007fffffffd7b8, __args={\x92\x956\0\0\0\0\0}) at allocator_traits.h:298:9
    frame #45: 0x00000000022f1829 credentials_provider_impl_test`std::__1::__shared_ptr_emplace<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider, std::__1::allocator<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider>>::__shared_ptr_emplace<Envoy::Api::Api&, testing::NiceMock<Envoy::Server::Configuration::MockServerFactoryContext>&, Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&), Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, char const (&) [29], char const (&) [31], char const (&) [18], Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>&, char const (&) [29]>(this=0x000026a0bfdd7b00, __a=allocator<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider> @ 0x00007fffffffd388, __args=0x000026a0bfc15500, __args=0x000026a0bfd70cb0, __args=(0x000026a0bfd6c000), __args=(0x000026a0bfd6c000), __args="/tmp/envoy_test_tmp.A4S5Ep/web_token_file", __args={\xddL1\0\0\0\0\0}, __args={\xd2\x86;\0\0\0\0\0}, __args={\07#\0\0\0\0\0}, __args=0x00007fffffffd7ac, __args=0x00007fffffffd7b8, __args={\x92\x956\0\0\0\0\0}) at shared_ptr.h:293:9
    frame #46: 0x00000000022f151e credentials_provider_impl_test`std::__1::shared_ptr<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider> std::__1::allocate_shared<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider, std::__1::allocator<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider>, Envoy::Api::Api&, testing::NiceMock<Envoy::Server::Configuration::MockServerFactoryContext>&, Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&), Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, char const (&) [29], char const (&) [31], char const (&) [18], Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>&, char const (&) [29], void>(__a=0x00007fffffffd568, __args=0x000026a0bfc15500, __args=0x000026a0bfd70cb0, __args=(0x000026a0bfd6c000), __args=(0x000026a0bfd6c000), __args="/tmp/envoy_test_tmp.A4S5Ep/web_token_file", __args={\xddL1\0\0\0\0\0}, __args={\xd2\x86;\0\0\0\0\0}, __args={\07#\0\0\0\0\0}, __args=0x00007fffffffd7ac, __args=0x00007fffffffd7b8, __args={\x92\x956\0\0\0\0\0}) at shared_ptr.h:954:55
    frame #47: 0x00000000022f1336 credentials_provider_impl_test`std::__1::shared_ptr<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider> std::__1::make_shared<Envoy::Extensions::Common::Aws::WebIdentityCredentialsProvider, Envoy::Api::Api&, testing::NiceMock<Envoy::Server::Configuration::MockServerFactoryContext>&, Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Http::Message<Envoy::Http::RequestHeaderMap, Envoy::Http::RequestTrailerMap>&), Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>)::'lambda'(Envoy::Upstream::ClusterManager&, std::__1::basic_string_view<char, std::__1::char_traits<char>>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, char const (&) [29], char const (&) [31], char const (&) [18], Envoy::Extensions::Common::Aws::MetadataFetcher::MetadataReceiver::ReceiverState&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l>>&, char const (&) [29], void>(__args=0x000026a0bfc15500, __args=0x000026a0bfd70cb0, __args=(0x000026a0bfd6c000), __args=(0x000026a0bfd6c000), __args="/tmp/envoy_test_tmp.A4S5Ep/web_token_file", __args={\xddL1\0\0\0\0\0}, __args={\xd2\x86;\0\0\0\0\0}, __args={\07#\0\0\0\0\0}, __args=0x00007fffffffd7ac, __args=0x00007fffffffd7b8, __args={\x92\x956\0\0\0\0\0}) at shared_ptr.h:963:12
    frame #48: 0x0000000002232dd6 credentials_provider_impl_test`Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest::setupProvider(this=0x000026a0bfd6c000, receiver_state=Ready, initialization_timer=(__rep_ = 2)) at credentials_provider_impl_test.cc:1768:17
    frame #49: 0x0000000002210c3a credentials_provider_impl_test`Envoy::Extensions::Common::Aws::WebIdentityCredentialsProviderTest_FailedFetchingDocument_Test::TestBody(this=0x000026a0bfd6c000) at credentials_provider_impl_test.cc:1856:3
    frame #50: 0x00000000058620eb credentials_provider_impl_test`void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=0x000026a0bfd6c000, method=0x00000000000000000000000000000021, location="the test body") at gtest.cc:2580:10
    frame #51: 0x000000000585178a credentials_provider_impl_test`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x000026a0bfd6c000, method=0x00000000000000000000000000000021, location="the test body") at gtest.cc:2616:14
    frame #52: 0x000000000583d2d3 credentials_provider_impl_test`testing::Test::Run(this=0x000026a0bfd6c000) at gtest.cc:2655:5
    frame #53: 0x000000000583dc84 credentials_provider_impl_test`testing::TestInfo::Run(this=0x000026a0bfc50700) at gtest.cc:2832:11
    frame #54: 0x000000000583e3cb credentials_provider_impl_test`testing::TestSuite::Run(this=0x000026a0bfc50800) at gtest.cc:2986:28
    frame #55: 0x000000000584a31d credentials_provider_impl_test`testing::internal::UnitTestImpl::RunAllTests(this=0x000026a0bfc4c000) at gtest.cc:5697:44
    frame #56: 0x000000000586577b credentials_provider_impl_test`bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x000026a0bfc4c000, method=(credentials_provider_impl_test`testing::internal::UnitTestImpl::RunAllTests() at gtest.cc:5588), location="auxiliary test code (environments or event listeners)") at gtest.cc:2580:10
    frame #57: 0x000000000585397a credentials_provider_impl_test`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x000026a0bfc4c000, method=(credentials_provider_impl_test`testing::internal::UnitTestImpl::RunAllTests() at gtest.cc:5588), location="auxiliary test code (environments or event listeners)") at gtest.cc:2616:14
    frame #58: 0x0000000005849ed8 credentials_provider_impl_test`testing::UnitTest::Run(this=0x0000000005cc8bd8) at gtest.cc:5280:10
    frame #59: 0x000000000406f3c1 credentials_provider_impl_test`RUN_ALL_TESTS() at gtest.h:2485:46
    frame #60: 0x000000000406ea55 credentials_provider_impl_test`Envoy::TestRunner::runTests(argc=3, argv=0x00007fffffffe648) at test_runner.cc:171:10

    frame #61: 0x000000000406c30f credentials_provider_impl_test`main(argc=3, argv=0x00007fffffffe648) at main.cc:34:10
    frame #62: 0x00007ffff7c71083 libc.so.6`__libc_start_main + 243
    frame #63: 0x00000000021f2fee credentials_provider_impl_test`_start + 46

[nbaws]

Not consistently reproducible. Will reopen if I can narrow this down further.