Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dns cache: add DNS query timeout option #17207

Merged
merged 4 commits into from
Jul 7, 2021
Merged

dns cache: add DNS query timeout option #17207

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
baef43e
dns cache: add DNS query timeout option
mattklein123 Jun 30, 2021
71de6be
Merge remote-tracking branch 'origin/main' into dns_timeout
mattklein123 Jul 1, 2021
e822fdf
comments
mattklein123 Jul 1, 2021
7a2708a
Merge remote-tracking branch 'origin/main' into dns_timeout
mattklein123 Jul 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api/envoy/extensions/common/dynamic_forward_proxy/v3/dns_cache.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,5 +119,5 @@ message DnsCacheConfig {
// used by the underlying DNS implementation (e.g., c-areas and Apple DNS) which are opaque.
// Setting this timeout will ensure that queries succeed or fail within the specified time frame
// and are then retried using the standard refresh rates. Defaults to 5s if not set.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it a bit more clear if the original request is canceled or if we retry in parallel like hedging? I recall (possibly erroneously) that was the latency-improving timeout folks often use on mobile of "if DNS does not return in this time use stale result" which implies to me 5s happens frequently enough we don't want to cancel. Alternately if platform implementation details would result in the OS getting the results for the original request that's Ok too but I'm not sure how to regression test that. cc @RyanTheOptimist @DavidSchinazi for thoughts as this is more their area than mine.

Copy link
Member Author

@mattklein123 mattklein123 Jul 1, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is a good call out. The current implementation will actually cancel and not hedge. Part of the impetus for this change is we think the iOS/apple resolver is occasionally getting wedged, so we are going to accompany this with some changes to that resolver to hopefully make it better handle timeouts (cc @junr03). One thing we might want to do here is actually modify the cancel() API to take a reason parameter, so that the impl can do different things if it's a normal cancellation vs. a timeout. I think in the case of timeout we might want to have the impls tare down the DNS connections and make new ones. Let me do that in this change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "hedge" mean in this context?

I'm not sure about 5s being the right value. I think that's probably too short for a client network stack which need to operate in congested lossy networks. I could look at some telemetry from Chrome around DNS resolution times if that would help? (Of course, I don't know much about the forward proxy deployments so maybe that's the right value for them)

FWIW, the way the stale DNS racing works in Chrome with QUIC is as follows. First host resolution is attempted. If no "Fresh" (presumably cached) result is available but a stale result is (this is all synchronous and does not involve waiting for network events) then the QUIC connection is started using the stale DNS entry. At this point we are running the QUIC handshake in parallel with the DNS request. We only use the resulting QUIC connection to send requests on if the DNS comes back and matches the IP connected to.

To do this, Chrome's resolver basically has two APIs. One for getting a fresh address (which will be async if the address is not cached) and another for getting a stale address (which is always synchronous but may return nothing).

Does that help?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could look at some telemetry from Chrome around DNS resolution times if that would help?

Yeah that would be super useful. My feeling though is this is an OK default for server and we can tune it for envoy-mobile config.

Does that help?

Yes definitely. I think this is what we need to move towards. cc @junr03

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm good with the default being server side as long as we make sure that @goaway and @junr03 and co know to tweak their defaults when they pick up the change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh and @RyanTheOptimist hedging is an option in router config where you can configure a retry in parallel - 2 requests to 2 upstreams, and cancel the slow one once the first response headers coming back. My point was I was ok with 5s for mobile if we were kicking off a second attempt not canceling the first but I'm also Ok with just upping the default on mobile.

google.protobuf.Duration dns_query_timeout = 11;
google.protobuf.Duration dns_query_timeout = 11 [(validate.rules).duration = {gt {}}];
}
2 changes: 1 addition & 1 deletion api/envoy/extensions/common/dynamic_forward_proxy/v4alpha/dns_cache.proto
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions docs/root/configuration/http/http_filters/dynamic_forward_proxy_filter.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ namespace.
dns_query_attempt, Counter, Number of DNS query attempts.
dns_query_success, Counter, Number of DNS query successes.
dns_query_failure, Counter, Number of DNS query failures.
dns_query_timeout, Counter, Number of DNS query :ref:`timeouts <envoy_v3_api_field_extensions.common.dynamic_forward_proxy.v3.DnsCacheConfig.dns_query_timeout>`.
host_address_changed, Counter, Number of DNS queries that resulted in a host address change.
host_added, Counter, Number of hosts that have been added to the cache.
host_removed, Counter, Number of hosts that have been removed from the cache.
Expand Down
12 changes: 11 additions & 1 deletion envoy/network/dns.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,20 @@ class ActiveDnsQuery {
public:
virtual ~ActiveDnsQuery() = default;

enum class CancelReason {
// The caller no longer needs the answer to the query.
QueryAbandoned,
// The query timed out from the perspective of the caller. The DNS implementation may take
// a different action in this case (e.g., destroying existing DNS connections) in an effort
// to get an answer to future queries.
Timeout
};

/**
* Cancel an outstanding DNS request.
* @param reason supplies the cancel reason.
*/
virtual void cancel() PURE;
virtual void cancel(CancelReason reason) PURE;
};

/**
Expand Down
2 changes: 1 addition & 1 deletion generated_api_shadow/envoy/extensions/common/dynamic_forward_proxy/v3/dns_cache.proto
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion generated_api_shadow/envoy/extensions/common/dynamic_forward_proxy/v4alpha/dns_cache.proto
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion source/common/network/apple_dns_impl.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,9 @@ AppleDnsResolverImpl::PendingResolution::~PendingResolution() {
}
}

void AppleDnsResolverImpl::PendingResolution::cancel() {
void AppleDnsResolverImpl::PendingResolution::cancel(Network::ActiveDnsQuery::CancelReason) {
// TODO(mattklein123): If cancel reason is timeout, do something more aggressive about destroying
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be inclined to remove the argument if it's not used anywhere, unless you actually plan on following up in the not too distant future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intention here was to follow up with at least the Apple DNS provider (and maybe c-ares) to blow away the channels if we have a timeout, to make it more likely we can work around dead connections. cc @junr03 wdyt?

Copy link
Member

@junr03 junr03 Jul 7, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattklein123 For the Apple implementation post #17226 there is nothing to blow away as each PendingResolution is getting its own socket (and registered file event with the event loop).

On the c-ares implementation, where the connection is shared, the distinction would be useful. A timeout based cancelation could be treated the same as a c-ares error and used to blow up the existing channel.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let's keep it, and we can make the c-ares change for Android, which I think would be useful IMO.

// and recreating the DNS system to maximize the chance of success in following queries.
ENVOY_LOG(debug, "Cancelling PendingResolution for {}", dns_name_);
ASSERT(owned_);
if (pending_cb_) {
Expand Down
2 changes: 1 addition & 1 deletion source/common/network/apple_dns_impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ class AppleDnsResolverImpl : public DnsResolver, protected Logger::Loggable<Logg
~PendingResolution();

// Network::ActiveDnsQuery
void cancel() override;
void cancel(Network::ActiveDnsQuery::CancelReason reason) override;

static DnsResponse buildDnsResponse(const struct sockaddr* address, uint32_t ttl);
// Wrapper for the API call.
Expand Down
3 changes: 2 additions & 1 deletion source/common/network/dns_impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,10 @@ class DnsResolverImpl : public DnsResolver, protected Logger::Loggable<Logger::I
: parent_(parent), callback_(callback), dispatcher_(dispatcher), channel_(channel),
dns_name_(dns_name) {}

void cancel() override {
void cancel(CancelReason) override {
// c-ares only supports channel-wide cancellation, so we just allow the
// network events to continue but don't invoke the callback on completion.
// TODO(mattklein123): Potentially use timeout to destroy and recreate the channel.
cancelled_ = true;
}

Expand Down
2 changes: 1 addition & 1 deletion source/common/upstream/logical_dns_cluster.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ void LogicalDnsCluster::startPreInit() { startResolve(); }

LogicalDnsCluster::~LogicalDnsCluster() {
if (active_dns_query_) {
active_dns_query_->cancel();
active_dns_query_->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);
}
}

Expand Down
2 changes: 1 addition & 1 deletion source/common/upstream/strict_dns_cluster.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ StrictDnsClusterImpl::ResolveTarget::ResolveTarget(

StrictDnsClusterImpl::ResolveTarget::~ResolveTarget() {
if (active_query_) {
active_query_->cancel();
active_query_->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);
}
}

Expand Down
2 changes: 1 addition & 1 deletion source/extensions/clusters/redis/redis_cluster.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ RedisCluster::DnsDiscoveryResolveTarget::DnsDiscoveryResolveTarget(RedisCluster&

RedisCluster::DnsDiscoveryResolveTarget::~DnsDiscoveryResolveTarget() {
if (active_query_) {
active_query_->cancel();
active_query_->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);
}
// Disable timer for mock tests.
if (resolve_timer_) {
Expand Down
11 changes: 6 additions & 5 deletions source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ DnsCacheImpl::DnsCacheImpl(
DnsCacheImpl::~DnsCacheImpl() {
for (const auto& primary_host : primary_hosts_) {
if (primary_host.second->active_query_ != nullptr) {
primary_host.second->active_query_->cancel();
primary_host.second->active_query_->cancel(
Network::ActiveDnsQuery::CancelReason::QueryAbandoned);
}
}

Expand Down Expand Up @@ -210,6 +211,9 @@ void DnsCacheImpl::startCacheLoad(const std::string& host, uint16_t default_port
}

DnsCacheImpl::PrimaryHostInfo& DnsCacheImpl::getPrimaryHost(const std::string& host) {
// Functions modify primary_hosts_ are only called in the main thread so we
// know it is safe to use the PrimaryHostInfo pointers outside of the lock.
ASSERT(main_thread_dispatcher_.isThreadSafe());
absl::ReaderMutexLock reader_lock{&primary_hosts_lock_};
const auto primary_host_it = primary_hosts_.find(host);
ASSERT(primary_host_it != primary_hosts_.end());
Expand All @@ -222,7 +226,7 @@ void DnsCacheImpl::onResolveTimeout(const std::string& host) {
auto& primary_host = getPrimaryHost(host);
ENVOY_LOG(debug, "host='{}' resolution timeout", host);
stats_.dns_query_timeout_.inc();
primary_host.active_query_->cancel();
primary_host.active_query_->cancel(Network::ActiveDnsQuery::CancelReason::Timeout);
finishResolve(host, Network::DnsResolver::ResolutionStatus::Failure, {});
}

Expand All @@ -233,10 +237,7 @@ void DnsCacheImpl::onReResolve(const std::string& host) {
// use-after-free issues
PrimaryHostInfoPtr host_to_erase;

// Functions like this one that modify primary_hosts_ are only called in the main thread so we
// know it is safe to use the PrimaryHostInfo pointers outside of the lock.
auto& primary_host = getPrimaryHost(host);

const std::chrono::steady_clock::duration now_duration =
main_thread_dispatcher_.timeSource().monotonicTime().time_since_epoch();
auto last_used_time = primary_host.host_info_->lastUsedTime();
Expand Down
4 changes: 2 additions & 2 deletions test/common/network/apple_dns_impl_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ TEST_F(AppleDnsImplTest, InvalidConfigOptions) {
TEST_F(AppleDnsImplTest, DestructPending) {
ActiveDnsQuery* query = resolveWithUnreferencedParameters("", DnsLookupFamily::V4Only, 0);
ASSERT_NE(nullptr, query);
query->cancel();
query->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);
}

TEST_F(AppleDnsImplTest, LocalLookup) {
Expand Down Expand Up @@ -194,7 +194,7 @@ TEST_F(AppleDnsImplTest, Cancel) {
DnsResolver::ResolutionStatus::Success, true));

ASSERT_NE(nullptr, query);
query->cancel();
query->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);

dispatcher_->run(Event::Dispatcher::RunType::Block);
}
Expand Down
4 changes: 2 additions & 2 deletions test/common/network/dns_impl_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -572,7 +572,7 @@ INSTANTIATE_TEST_SUITE_P(IpVersions, DnsImplTest,
TEST_P(DnsImplTest, DestructPending) {
ActiveDnsQuery* query = resolveWithUnreferencedParameters("", DnsLookupFamily::V4Only, false);
ASSERT_NE(nullptr, query);
query->cancel();
query->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);
// Also validate that pending events are around to exercise the resource
// reclamation path.
EXPECT_GT(peer_->events().size(), 0U);
Expand Down Expand Up @@ -806,7 +806,7 @@ TEST_P(DnsImplTest, Cancel) {
{"201.134.56.7"}, {}, absl::nullopt));

ASSERT_NE(nullptr, query);
query->cancel();
query->cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned);

dispatcher_->run(Event::Dispatcher::RunType::Block);
}
Expand Down
2 changes: 1 addition & 1 deletion test/common/upstream/logical_dns_cluster_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ class LogicalDnsClusterTest : public Event::TestUsingSimulatedTime, public testi
logical_host->createConnection(dispatcher_, nullptr, nullptr);

// Make sure we cancel.
EXPECT_CALL(active_dns_query_, cancel());
EXPECT_CALL(active_dns_query_, cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
expectResolve(Network::DnsLookupFamily::V4Only, expected_address);
resolve_timer_->invokeCallback();

Expand Down
24 changes: 16 additions & 8 deletions test/common/upstream/upstream_impl_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -457,8 +457,10 @@ TEST_F(StrictDnsClusterImplTest, Basic) {
resolver2.expectResolve(*dns_resolver_);
resolver2.timer_->invokeCallback();

EXPECT_CALL(resolver1.active_dns_query_, cancel());
EXPECT_CALL(resolver2.active_dns_query_, cancel());
EXPECT_CALL(resolver1.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
EXPECT_CALL(resolver2.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
}

// Verifies that host removal works correctly when hosts are being health checked
Expand Down Expand Up @@ -854,9 +856,12 @@ TEST_F(StrictDnsClusterImplTest, LoadAssignmentBasic) {
resolver3.expectResolve(*dns_resolver_);
resolver3.timer_->invokeCallback();

EXPECT_CALL(resolver1.active_dns_query_, cancel());
EXPECT_CALL(resolver2.active_dns_query_, cancel());
EXPECT_CALL(resolver3.active_dns_query_, cancel());
EXPECT_CALL(resolver1.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
EXPECT_CALL(resolver2.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
EXPECT_CALL(resolver3.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
}

TEST_F(StrictDnsClusterImplTest, LoadAssignmentBasicMultiplePriorities) {
Expand Down Expand Up @@ -993,9 +998,12 @@ TEST_F(StrictDnsClusterImplTest, LoadAssignmentBasicMultiplePriorities) {
resolver3.expectResolve(*dns_resolver_);
resolver3.timer_->invokeCallback();

EXPECT_CALL(resolver1.active_dns_query_, cancel());
EXPECT_CALL(resolver2.active_dns_query_, cancel());
EXPECT_CALL(resolver3.active_dns_query_, cancel());
EXPECT_CALL(resolver1.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
EXPECT_CALL(resolver2.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
EXPECT_CALL(resolver3.active_dns_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
}

// Verifies that specifying a custom resolver when using STRICT_DNS fails
Expand Down
3 changes: 1 addition & 2 deletions test/extensions/clusters/redis/redis_cluster_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -544,10 +544,9 @@ class RedisClusterTest : public testing::Test,
Network::DnsResolver::ResolveCb) -> Network::ActiveDnsQuery* {
return &active_dns_query_;
}));
;
resolver_target.startResolveDns();

EXPECT_CALL(active_dns_query_, cancel());
EXPECT_CALL(active_dns_query_, cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
}

Stats::TestUtil::TestStore stats_store_;
Expand Down
5 changes: 3 additions & 2 deletions test/extensions/common/dynamic_forward_proxy/dns_cache_impl_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ TEST_F(DnsCacheImplTest, ResolveTimeout) {
checkStats(1 /* attempt */, 0 /* success */, 0 /* failure */, 0 /* address changed */,
1 /* added */, 0 /* removed */, 1 /* num hosts */);

EXPECT_CALL(resolver_->active_query_, cancel());
EXPECT_CALL(resolver_->active_query_, cancel(Network::ActiveDnsQuery::CancelReason::Timeout));
EXPECT_CALL(*timeout_timer, disableTimer());
EXPECT_CALL(update_callbacks_, onDnsHostAddOrUpdate(_, _)).Times(0);
EXPECT_CALL(callbacks, onLoadDnsCacheComplete(DnsHostInfoAddressIsNull()));
Expand Down Expand Up @@ -783,7 +783,8 @@ TEST_F(DnsCacheImplTest, CancelActiveQueriesOnDestroy) {
EXPECT_NE(result.handle_, nullptr);
EXPECT_EQ(absl::nullopt, result.host_info_);

EXPECT_CALL(resolver_->active_query_, cancel());
EXPECT_CALL(resolver_->active_query_,
cancel(Network::ActiveDnsQuery::CancelReason::QueryAbandoned));
dns_cache_.reset();
}

Expand Down
2 changes: 1 addition & 1 deletion test/mocks/network/mocks.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ class MockActiveDnsQuery : public ActiveDnsQuery {
~MockActiveDnsQuery() override;

// Network::ActiveDnsQuery
MOCK_METHOD(void, cancel, ());
MOCK_METHOD(void, cancel, (CancelReason reason));
};

class MockDnsResolver : public DnsResolver {
Expand Down