Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for client-side QUIC 0-RTT #16260

Merged
merged 9 commits into from
May 5, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions source/common/quic/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,17 @@ envoy_cc_library(
],
)

envoy_cc_library(
name = "envoy_quic_session_cache_lib",
srcs = ["envoy_quic_session_cache.cc"],
hdrs = ["envoy_quic_session_cache.h"],
external_deps = ["quiche_quic_platform"],
tags = ["nofips"],
deps = [
"@com_googlesource_quiche//:quic_core_crypto_crypto_handshake_lib",
],
)

envoy_cc_library(
name = "spdy_server_push_utils_for_envoy_lib",
srcs = ["spdy_server_push_utils_for_envoy.cc"],
Expand Down Expand Up @@ -150,6 +161,7 @@ envoy_cc_library(
":envoy_quic_connection_helper_lib",
":envoy_quic_proof_verifier_lib",
":envoy_quic_server_session_lib",
":envoy_quic_session_cache_lib",
":envoy_quic_utils_lib",
"//include/envoy/http:codec_interface",
"//include/envoy/registry",
Expand Down
8 changes: 5 additions & 3 deletions source/common/quic/client_connection_factory_impl.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#include "common/quic/client_connection_factory_impl.h"

#include "common/quic/envoy_quic_session_cache.h"
#include "common/quic/quic_transport_socket_factory.h"

namespace Envoy {
Expand All @@ -20,9 +21,10 @@ PersistentQuicInfoImpl::PersistentQuicInfoImpl(
: conn_helper_(dispatcher), alarm_factory_(dispatcher, *conn_helper_.GetClock()),
server_id_{getConfig(transport_socket_factory).serverNameIndication(),
static_cast<uint16_t>(server_addr->ip()->port()), false},
crypto_config_(
std::make_unique<quic::QuicCryptoClientConfig>(std::make_unique<EnvoyQuicProofVerifier>(
stats_scope, getConfig(transport_socket_factory), time_source))) {
crypto_config_(std::make_unique<quic::QuicCryptoClientConfig>(
std::make_unique<EnvoyQuicProofVerifier>(stats_scope, getConfig(transport_socket_factory),
time_source),
std::make_unique<EnvoyQuicSessionCache>(time_source))) {
quiche::FlagRegistry::getInstance();
}

Expand Down
168 changes: 168 additions & 0 deletions source/common/quic/envoy_quic_session_cache.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
#include "common/quic/envoy_quic_session_cache.h"

namespace Envoy {
namespace Quic {
namespace {

constexpr size_t MaxSessionCacheEntries = 1024;
alyssawilk marked this conversation as resolved.
Show resolved Hide resolved

// Returns false if the SSL |session| doesn't exist or it is expired at |system_time|.
bool isSessionValid(SSL_SESSION* session, SystemTime system_time) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits: if you make this a private member function, it can directly access system_time_ and no need to pass in system_time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no system_time_ member. I think you might be confusing it with time_source_ which is different - we want to avoid computing the time from the time source too often.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I meant to access time_source_ directly. time_source_.systemTime() is always called before isSessionValid(). For readability, I think either renaming system_time to now here or making this function a class member.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I've renamed system_time to now.

if (session == nullptr) {
return false;
}
const time_t now = std::chrono::system_clock::to_time_t(system_time);
if (now < 0) {
return false;
}
const uint64_t now_u64 = static_cast<uint64_t>(now);
const uint64_t session_time = SSL_SESSION_get_time(session);
const uint64_t session_expiration = session_time + SSL_SESSION_get_timeout(session);
DavidSchinazi marked this conversation as resolved.
Show resolved Hide resolved
// now_u64 may be slightly behind because of differences in how time is calculated at this layer
// versus BoringSSL. Add a second of wiggle room to account for this.
return session_time <= now_u64 + 1 && now_u64 < session_expiration;
}

bool doApplicationStatesMatch(const quic::ApplicationState* state, quic::ApplicationState* other) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits: making other const as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if (state == other) {
return true;
}
Comment on lines +30 to +32
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering why we compare the pointer of two different objects. But it seems to be only possible when both state and other are nullptr? I slightly prefer to make it explicit instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is needed to avoid a dereference when they're both nullptr. I personally prefer this style of comparison because it is correct for all inputs. The fact that callers won't ever send the same non-null pointer as both parameters is an implementation details that doesn't need to be visible in this function.

if ((state != nullptr && other == nullptr) || (state == nullptr && other != nullptr)) {
return false;
}
return (*state == *other);
}

} // namespace

EnvoyQuicSessionCache::EnvoyQuicSessionCache(TimeSource& time_source) : time_source_(time_source) {}

EnvoyQuicSessionCache::~EnvoyQuicSessionCache() = default;

void EnvoyQuicSessionCache::Insert(const quic::QuicServerId& server_id,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The duplication between this and QuicClientSessionCache is a bit of a bummer.

Optionally, as a follow-up, maybe we can implement more of the base logic in the quiche class and either template away time or have pure virtuals for time bits. but that'd be a follow-up no matter what and should wait until we've sorted out any upstream scalability problems.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's QuicClientSessionCache?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, yes we're going to have duplication between Chrome and Envoy :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, but modulo "can we refactor away the time abstraction" seems reasonable to stick this one in quiche at some point.

bssl::UniquePtr<SSL_SESSION> session,
const quic::TransportParameters& params,
const quic::ApplicationState* application_state) {
auto it = cache_.find(server_id);
if (it == cache_.end()) {
// New server ID, add a new entry.
createAndInsertEntry(server_id, std::move(session), params, application_state);
return;
}
Entry& entry = it->second;
if (params == *entry.params &&
doApplicationStatesMatch(application_state, entry.application_state.get())) {
// The states are both the same, so we only need to insert the session.
entry.pushSession(std::move(session));
return;
}
// States are different, replace the entry.
cache_.erase(it);
createAndInsertEntry(server_id, std::move(session), params, application_state);
}

std::unique_ptr<quic::QuicResumptionState>
EnvoyQuicSessionCache::Lookup(const quic::QuicServerId& server_id, const SSL_CTX* /*ctx*/) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drive by question - why is SSL_CTX unused here and in quiche? Should we clean up upstream?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some clients may want to use different CCL_CTX'es. We could remove it but it isn't causing any issues. Either way that would be a QUICHE CL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, sorry, quiche == upstream in this context. Not something you'd do in situ but if it's not used anywhere might be worth adding our quiche clean up list.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SG

auto it = cache_.find(server_id);
if (it == cache_.end()) {
return nullptr;
}
Entry& entry = it->second;
const SystemTime system_time = time_source_.systemTime();
if (!isSessionValid(entry.peekSession(), system_time)) {
cache_.erase(it);
return nullptr;
}
auto state = std::make_unique<quic::QuicResumptionState>();
state->tls_session = entry.popSession();
if (entry.params != nullptr) {
state->transport_params = std::make_unique<quic::TransportParameters>(*entry.params);
}
if (entry.application_state != nullptr) {
state->application_state = std::make_unique<quic::ApplicationState>(*entry.application_state);
}
return state;
}

void EnvoyQuicSessionCache::ClearEarlyData(const quic::QuicServerId& server_id) {
auto it = cache_.find(server_id);
if (it == cache_.end()) {
return;
}
for (bssl::UniquePtr<SSL_SESSION>& session : it->second.sessions) {
if (session != nullptr) {
session.reset(SSL_SESSION_copy_without_early_data(session.get()));
}
}
}

void EnvoyQuicSessionCache::prune() {
quic::QuicServerId oldest_id;
uint64_t oldest_expiration = std::numeric_limits<uint64_t>::max();
const SystemTime system_time = time_source_.systemTime();
auto it = cache_.begin();
while (it != cache_.end()) {
Entry& entry = it->second;
SSL_SESSION* session = entry.peekSession();
if (!isSessionValid(session, system_time)) {
it = cache_.erase(it);
DavidSchinazi marked this conversation as resolved.
Show resolved Hide resolved
} else {
if (cache_.size() >= MaxSessionCacheEntries) {
// Only save oldest if we are at the size limit.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/s/save oldest/track the oldest session/ ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

const uint64_t session_expiration =
SSL_SESSION_get_time(session) + SSL_SESSION_get_timeout(session);
if (session_expiration < oldest_expiration) {
oldest_expiration = session_expiration;
oldest_id = it->first;
}
}
++it;
}
}
if (cache_.size() >= MaxSessionCacheEntries) {
cache_.erase(oldest_id);
}
}

void EnvoyQuicSessionCache::createAndInsertEntry(const quic::QuicServerId& server_id,
bssl::UniquePtr<SSL_SESSION> session,
const quic::TransportParameters& params,
const quic::ApplicationState* application_state) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit confusing about the life time of application_state. Can you comment that it's now owned?

Copy link
Contributor Author

@DavidSchinazi DavidSchinazi May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The input parameter isn't owned. The fact that it's a const pointer makes the ownership model clear to me. Am I missing something?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, typo "not" for "now". If we create ApplicationState instance for each entry here, why does doApplicationStatesMatch() compare pointers?
And Lookup() also create ApplicationState for return value. Probably QUICHE somehow requires duplication this object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, typo "not" for "now".

I don't think we need to comment that a const pointer is not owned, as that's very common usage.

If we create ApplicationState instance for each entry here, why does doApplicationStatesMatch() compare pointers?

doApplicationStatesMatch() dereferences the pointers and compares the values. It looks at pointers because they can be nullptr.

And Lookup() also create ApplicationState for return value. Probably QUICHE somehow requires duplication this object.

I'm not sure what you're asking? QUICHE passes us a const pointer and we need to make a copy that we own.

prune();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we walk the entire list on insertion? Maybe make that clear in prune comments()
Also definitely worth a comment up by that 1024 that it should stay small and why. eek!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

O(1024) is negligible compared to the cryptographic operations required to create a new connection.

Added comments though

ASSERT(cache_.size() < MaxSessionCacheEntries);
Entry entry;
entry.pushSession(std::move(session));
entry.params = std::make_unique<quic::TransportParameters>(params);
if (application_state != nullptr) {
entry.application_state = std::make_unique<quic::ApplicationState>(*application_state);
}
cache_.insert(std::make_pair(server_id, std::move(entry)));
}

size_t EnvoyQuicSessionCache::size() const { return cache_.size(); }

EnvoyQuicSessionCache::Entry::Entry() = default;
EnvoyQuicSessionCache::Entry::Entry(Entry&&) noexcept = default;
EnvoyQuicSessionCache::Entry::~Entry() = default;

void EnvoyQuicSessionCache::Entry::pushSession(bssl::UniquePtr<SSL_SESSION> session) {
if (sessions[0] != nullptr) {
sessions[1] = std::move(sessions[0]);
}
sessions[0] = std::move(session);
}

bssl::UniquePtr<SSL_SESSION> EnvoyQuicSessionCache::Entry::popSession() {
if (sessions[0] == nullptr) {
return nullptr;
}
bssl::UniquePtr<SSL_SESSION> session = std::move(sessions[0]);
sessions[0] = std::move(sessions[1]);
sessions[1] = nullptr;
return session;
}

SSL_SESSION* EnvoyQuicSessionCache::Entry::peekSession() { return sessions[0].get(); }

} // namespace Quic
} // namespace Envoy
62 changes: 62 additions & 0 deletions source/common/quic/envoy_quic_session_cache.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#pragma once

#include "envoy/common/time.h"

#include "quiche/quic/core/crypto/quic_crypto_client_config.h"

namespace Envoy {
namespace Quic {

// Implementation of quic::SessionCache using an Envoy time source.
class EnvoyQuicSessionCache : public quic::SessionCache {
public:
explicit EnvoyQuicSessionCache(TimeSource& time_source);
~EnvoyQuicSessionCache() override;

// From quic::SessionCache.
void Insert(const quic::QuicServerId& server_id, bssl::UniquePtr<SSL_SESSION> session,
const quic::TransportParameters& params,
const quic::ApplicationState* application_state) override;
std::unique_ptr<quic::QuicResumptionState> Lookup(const quic::QuicServerId& server_id,
const SSL_CTX* ctx) override;
void ClearEarlyData(const quic::QuicServerId& server_id) override;

// Returns number of entries in the cache.
size_t size() const;

private:
struct Entry {
Entry();
Entry(Entry&&) noexcept;
~Entry();

// Adds a new |session| onto sessions, dropping the oldest one if two are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto | here and below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

// already stored.
void pushSession(bssl::UniquePtr<SSL_SESSION> session);

// Retrieves the latest session from the entry, meanwhile removing it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retrieves and removes the latest session...?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

bssl::UniquePtr<SSL_SESSION> popSession();

SSL_SESSION* peekSession();

bssl::UniquePtr<SSL_SESSION> sessions[2];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why size of 2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because that's sufficient in practice, it's the value Chrome uses.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind commenting about the reason? Otherwise a queue-like data structure might be more preferred.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mind adding a bit more detail for those of with less QUIC-savvy? is it that session information changes infrequently? If so why do we need two?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

std::unique_ptr<quic::TransportParameters> params;
std::unique_ptr<quic::ApplicationState> application_state;
};

// Remove all entries that are no longer valid. If after that we are still at the size limit, also
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optionally
If after that we are still at the size limit, also remove ->
If all entries were valid but the cache is at its size limit, instead removes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

// remove the oldest entry.
void prune();

// Creates a new entry and insert into |cache_|.
void createAndInsertEntry(const quic::QuicServerId& server_id,
bssl::UniquePtr<SSL_SESSION> session,
const quic::TransportParameters& params,
const quic::ApplicationState* application_state);

std::map<quic::QuicServerId, Entry> cache_;
alyssawilk marked this conversation as resolved.
Show resolved Hide resolved
TimeSource& time_source_;
};

} // namespace Quic
} // namespace Envoy
10 changes: 10 additions & 0 deletions test/common/quic/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@ envoy_cc_test(
],
)

envoy_cc_test(
name = "envoy_quic_session_cache_test",
srcs = ["envoy_quic_session_cache_test.cc"],
external_deps = ["quiche_quic_platform"],
tags = ["nofips"],
deps = [
"//source/common/quic:envoy_quic_session_cache_lib",
],
)

envoy_cc_test(
name = "envoy_quic_proof_verifier_test",
srcs = ["envoy_quic_proof_verifier_test.cc"],
Expand Down
Loading