dns: fix exception unsafe behavior in c-ares callbacks.

[htuch]

Previously, we were taking an exception due to the late validation of update_merge_window duration
in ClusterManagerImpl::scheduleUpdate, which happened under a c-ares strict DNS host resolution
callback. There are several related issues here:

1. c-ares is exception unsafe, see ares_gethostbyname() and exception safety c-ares/c-ares#219.
2. We should be validating Durations with PGV, see
   Validate Duration fields in PGV bufbuild/protoc-gen-validate#97.
3. We should defer the c-ares resolution callbacks to be outside the c-ares callback context for
   exception safety.

This PR addresses (3) by moving callbacks, even when they are "immediate", to a dispatcher post, so
that we never take an exception under a c-ares callback.

A workaround for (2) is provided, in lieu of bufbuild/protoc-gen-validate#97,
which is blocked on our ability to bump PGV version in Envoy, see
lyft/protoc-gen-star#28.

Fixes oss-fuzz issue https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9868.

Risk level: Medium (DNS clusters will have some timing changes).
Testing: Updated DNS implementation unit tests, server fuzz corpus entry added.

Signed-off-by: Harvey Tuch [email protected]

[ggreenway]

With the original exception, who is supposed to catch it? What does the callstack roughly look like when this happens?

[htuch]

@ggreenway the server initialization code was supposed to catch it; here's the trace:

Direct leak of 8 byte(s) in 1 object(s) allocated from:
--
  | #0 0x4e4858 in __interceptor___strdup _asan_rtl_
  | #1 0x26fefd4 in fake_hostent /builder/home/.cache/bazel/_bazel_root/4e9824db8e7d11820cfa25090ed4ed10/external/envoy_deps_cache_b22e04bff96538ea37e715942da6315c/cares.dep.build/c-ares-cares-1_14_0/ares_gethostbyname.c:290:20
  | #2 0x26fefd4 in ares_gethostbyname /builder/home/.cache/bazel/_bazel_root/4e9824db8e7d11820cfa25090ed4ed10/external/envoy_deps_cache_b22e04bff96538ea37e715942da6315c/cares.dep.build/c-ares-cares-1_14_0/ares_gethostbyname.c:98
  | #3 0x8c0a25 in Envoy::Network::DnsResolverImpl::resolve(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Network::DnsLookupFamily, std::__1::function<void (std::__1::list<std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Address::Instance const> > >&&)>) /source/common/network/dns_impl.cc:196:25
  | #4 0x14d1cf4 in Envoy::Upstream::StrictDnsClusterImpl::ResolveTarget::startResolve() /source/common/upstream/upstream_impl.cc:1059:42
  | #5 0x14d18ae in Envoy::Upstream::StrictDnsClusterImpl::startPreInit() /source/common/upstream/upstream_impl.cc:1015:13
  | #6 0x14c74a2 in Envoy::Upstream::ClusterImplBase::initialize(std::__1::function<void ()>) /source/common/upstream/upstream_impl.cc:516:3
  | #7 0x922b26 in Envoy::Upstream::ClusterManagerInitHelper::addCluster(Envoy::Upstream::Cluster&) /source/common/upstream/cluster_manager_impl.cc:50:13
  | #8 0x92628e in Envoy::Upstream::ClusterManagerImpl::ClusterManagerImpl(envoy::config::bootstrap::v2::Bootstrap const&, Envoy::Upstream::ClusterManagerFactory&, Envoy::Stats::Store&, Envoy::ThreadLocal::Instance&, Envoy::Runtime::Loader&, Envoy::Runtime::RandomGenerator&, Envoy::LocalInfo::LocalInfo const&, Envoy::AccessLog::AccessLogManager&, Envoy::Event::Dispatcher&, Envoy::Server::Admin&, Envoy::SystemTimeSource&, Envoy::MonotonicTimeSource&) /source/common/upstream/cluster_manager_impl.cc:292:18
  | #9 0x93d06a in Envoy::Upstream::ProdClusterManagerFactory::clusterManagerFromProto(envoy::config::bootstrap::v2::Bootstrap const&, Envoy::Stats::Store&, Envoy::ThreadLocal::Instance&, Envoy::Runtime::Loader&, Envoy::Runtime::RandomGenerator&, Envoy::LocalInfo::LocalInfo const&, Envoy::AccessLog::AccessLogManager&, Envoy::Server::Admin&) /source/common/upstream/cluster_manager_impl.cc:1179:32
  | #10 0x25ec765 in Envoy::Server::Configuration::MainImpl::initialize(envoy::config::bootstrap::v2::Bootstrap const&, Envoy::Server::Instance&, Envoy::Upstream::ClusterManagerFactory&) /source/server/configuration_impl.cc:56:46
  | #11 0x724ae6 in Envoy::Server::InstanceImpl::initialize(Envoy::Server::Options&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::ComponentFactory&) /source/server/server.cc:280:16
  | #12 0x71ffeb in Envoy::Server::InstanceImpl::InstanceImpl(Envoy::Server::Options&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::TestHooks&, Envoy::Server::HotRestart&, Envoy::Stats::StoreRoot&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, Envoy::ThreadLocal::Instance&) /source/server/server.cc:75:5
  | #13 0x5d0dcc in std::__1::__unique_if<Envoy::Server::InstanceImpl>::__unique_single std::__1::make_unique<Envoy::Server::InstanceImpl, testing::NiceMock<Envoy::Server::MockOptions>&, std::__1::shared_ptr<Envoy::Network::Address::Ipv4Instance>, Envoy::DefaultTestHooks&, testing::NiceMock<Envoy::Server::MockHotRestart>&, Envoy::Stats::TestIsolatedStoreImpl&, Envoy::Thread::MutexBasicLockable&, Envoy::Server::TestComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGeneratorImpl, std::__1::default_delete<Envoy::Runtime::RandomGeneratorImpl> >, Envoy::ThreadLocal::InstanceImpl&>(testing::NiceMock<Envoy::Server::MockOptions>&&&, std::__1::shared_ptr<Envoy::Network::Address::Ipv4Instance>&&, Envoy::DefaultTestHooks&&&, testing::NiceMock<Envoy::Server::MockHotRestart>&&&, Envoy::Stats::TestIsolatedStoreImpl&&&, Envoy::Thread::MutexBasicLockable&&&, Envoy::Server::TestComponentFactory&&&, std::__1::unique_ptr<Envoy::Runtime::RandomGeneratorImpl, std::__1::default_delete<Envoy::Runtime::RandomGeneratorImpl> >&&, Envoy::ThreadLocal::InstanceImpl&&&) /usr/local/include/c++/v1/memory:3114:32
  | #14 0x5cfcc7 in Envoy::Server::TestOneProtoInput(envoy::config::bootstrap::v2::Bootstrap const&) /test/server/server_fuzz_test.cc:39:19
  | #15 0x5cf694 in LLVMFuzzerTestOneInput /test/server/server_fuzz_test.cc:19:1
  | #16 0x38234e2 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:575:15
  | #17 0x380202a in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
  | #18 0x380d6f5 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:715:9
  | #19 0x37f3c3c in main /src/libfuzzer/FuzzerMain.cpp:20:10
  | #20 0x7ff42b63f82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291

[ggreenway]

So with this fix, where does the exception get caught? If we post() it, it won't get caught by server initialization. So where does it go?

[htuch]

There are two things in this PR:

1. We post to defer it happening under c-ares callback. This then gets picked up by the server's run() and Envoy would take an "unexpected exception, let's shutdown" resolution.
2. I validate and check for the problem exception condition much earlier (search for the TODO). We should be checking for this condition well before we start doing DNS resolution.

[ggreenway]

So let's say that you hadn't done (2) above so the exception could still be thrown at that time, but you moved it to be post()'d. Isn't that still an exception-unsafe context? I don't think there's any top-level try/catch. So it seems like whether you move it to be post()'d, or call it from the same place it is now, you'd need a try/catch around the call if you want to protect against future incorrect throw's. So what value is there in moving it to be post()'d?

[htuch]

@ggreenway we have top-level try-catch for this, see https://github.com/envoyproxy/envoy/blob/master/source/exe/main_common.cc#L136. The idea in this PR is that it's better to exit via this path if we take an accidental exception vs. the existing situation, where we have a who-knows-what-c-ares is doing scenario.

[ggreenway]

Oh right, I was thinking this was on a worker thread, but this always happens on the main thread.