sds: additional support for symlink-based key rotation. (envoyproxy#1…

…3721)

There are a few limitations in our existing support for symlink-based
key rotation:

We don't atomically resolve symlinks, so a single snapshot might have
inconsistent symlink resolutions for different watched files.
Watches are on parent directories, e.g. for /foo/bar/baz on /foo/bar,
which doesn't support common key rotation schemes were /foo/new/baz
is rotated via a mv -Tf /foo/new /foo/bar.
The solution is to provide a structured WatchedDirectory for Secrets to
opt into when monitoring DataSources. SDS will used WatchedDirectory
to setup the inotify watch instead of the DataSource path. On update, it will
read key/cert twice, verifying file content hash consistency.

Risk level: Low (opt-in feature)
Testing: Unit and integration tests added.

Fixes envoyproxy#13663
Fixes envoyproxy#10979
Fixes envoyproxy#13370

Signed-off-by: Harvey Tuch <[email protected]>
Signed-off-by: Qin Qin <[email protected]>

api/envoy/config/core/v3/base.proto

@@ -313,6 +313,13 @@ message HeaderMap {
   repeated HeaderValue headers = 1;
 }
 
+// A directory that is watched for changes, e.g. by inotify on Linux. Move/rename
+// events inside this directory trigger the watch.
+message WatchedDirectory {
+  // Directory path to watch.
+  string path = 1 [(validate.rules).string = {min_len: 1}];
+}
+
 // Data source consisting of either a file or an inline value.
 message DataSource {
   option (udpa.annotations.versioning).previous_message_type = "envoy.api.v2.core.DataSource";

api/envoy/extensions/transport_sockets/tls/v3/common.proto

@@ -128,16 +128,36 @@ message PrivateKeyProvider {
   }
 }
 
-// [#next-free-field: 7]
+// [#next-free-field: 8]
 message TlsCertificate {
   option (udpa.annotations.versioning).previous_message_type = "envoy.api.v2.auth.TlsCertificate";
 
   // The TLS certificate chain.
+  //
+  // If *certificate_chain* is a filesystem path, a watch will be added to the
+  // parent directory for any file moves to support rotation. This currently
+  // only applies to dynamic secrets, when the *TlsCertificate* is delivered via
+  // SDS.
   config.core.v3.DataSource certificate_chain = 1;
 
   // The TLS private key.
+  //
+  // If *private_key* is a filesystem path, a watch will be added to the parent
+  // directory for any file moves to support rotation. This currently only
+  // applies to dynamic secrets, when the *TlsCertificate* is delivered via SDS.
   config.core.v3.DataSource private_key = 2 [(udpa.annotations.sensitive) = true];
 
+  // If specified, updates of file-based *certificate_chain* and *private_key*
+  // sources will be triggered by this watch. The certificate/key pair will be
+  // read together and validated for atomic read consistency (i.e. no
+  // intervening modification occurred between cert/key read, verified by file
+  // hash comparisons). This allows explicit control over the path watched, by
+  // default the parent directories of the filesystem paths in
+  // *certificate_chain* and *private_key* are watched if this field is not
+  // specified. This only applies when a *TlsCertificate* is delivered by SDS
+  // with references to filesystem paths.
+  config.core.v3.WatchedDirectory watched_directory = 7;
+
   // BoringSSL private key method provider. This is an alternative to :ref:`private_key
   // <envoy_api_field_extensions.transport_sockets.tls.v3.TlsCertificate.private_key>` field. This can't be
   // marked as ``oneof`` due to API compatibility reasons. Setting both :ref:`private_key
@@ -191,7 +211,7 @@ message TlsSessionTicketKeys {
       [(validate.rules).repeated = {min_items: 1}, (udpa.annotations.sensitive) = true];
 }
 
-// [#next-free-field: 11]
+// [#next-free-field: 12]
 message CertificateValidationContext {
   option (udpa.annotations.versioning).previous_message_type =
       "envoy.api.v2.auth.CertificateValidationContext";
@@ -233,8 +253,21 @@ message CertificateValidationContext {
   //
   // See :ref:`the TLS overview <arch_overview_ssl_enabling_verification>` for a list of common
   // system CA locations.
+  //
+  // If *trusted_ca* is a filesystem path, a watch will be added to the parent
+  // directory for any file moves to support rotation. This currently only
+  // applies to dynamic secrets, when the *CertificateValidationContext* is
+  // delivered via SDS.
   config.core.v3.DataSource trusted_ca = 1;
 
+  // If specified, updates of a file-based *trusted_ca* source will be triggered
+  // by this watch. This allows explicit control over the path watched, by
+  // default the parent directory of the filesystem path in *trusted_ca* is
+  // watched if this field is not specified. This only applies when a
+  // *CertificateValidationContext* is delivered by SDS with references to
+  // filesystem paths.
+  config.core.v3.WatchedDirectory watched_directory = 11;
+
   // An optional list of base64-encoded SHA-256 hashes. If specified, Envoy will verify that the
   // SHA-256 of the DER-encoded Subject Public Key Information (SPKI) of the presented certificate
   // matches one of the specified values.

docs/root/configuration/security/secret.rst

@@ -33,6 +33,26 @@ SDS Configuration
 
 *SdsSecretConfig* is used in two fields in :ref:`CommonTlsContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CommonTlsContext>`. The first field is *tls_certificate_sds_secret_configs* to use SDS to get :ref:`TlsCertificate <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.TlsCertificate>`. The second field is *validation_context_sds_secret_config* to use SDS to get :ref:`CertificateValidationContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CertificateValidationContext>`.
 
+.. _sds_key_rotation:
+
+Key rotation
+------------
+
+It's usually preferrable to perform key rotation via gRPC SDS, but when this is not possible or
+desired (e.g. during bootstrap of SDS credentials), SDS allows for filesystem rotation when secrets
+refer to filesystem paths. This currently is supported for the following secret types:
+
+* :ref:`TlsCertificate <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.TlsCertificate>`
+* :ref:`CertificateValidationContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CertificateValidationContext>`
+
+By default, directories containing secrets are watched for filesystem move events. Explicit control over
+the watched directory is possible by specifying a *watched_directory* path in :ref:`TlsCertificate
+<envoy_v3_api_field_extensions.transport_sockets.tls.v3.TlsCertificate.watched_directory>` and
+:ref:`CertificateValidationContext
+<envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.watched_directory>`.
+
+An example of key rotation is provided :ref:`below <xds_certificate_rotation>`.
+
 Example one: static_resource
 -----------------------------
 
@@ -211,9 +231,11 @@ In contrast, :ref:`sds_server_example` requires a restart to reload xDS certific
           "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext"
           common_tls_context:
             tls_certificate_sds_secret_configs:
+              name: tls_sds
               sds_config:
                 path: /etc/envoy/tls_certificate_sds_secret.yaml
             validation_context_sds_secret_config:
+              name: validation_context_sds
               sds_config:
                 path: /etc/envoy/validation_context_sds_secret.yaml
 
@@ -239,6 +261,39 @@ Path to CA certificate bundle for validating the xDS server certificate is given
           trusted_ca:
             filename: /certs/cacert.pem
 
+In the above example, a watch will be established on ``/certs``. File movement in this directory
+will trigger an update. An alternative common key rotation scheme that provides improved atomicity
+is to establish an active symlink ``/certs/current`` and use an atomic move operation to replace the
+symlink. The watch in this case needs to be on the certificate's grandparent directory. Envoy
+supports this scheme via the use of *watched_directory*. Continuing the above examples:
+
+.. code-block:: yaml
+
+    resources:
+      - "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
+        tls_certificate:
+          certificate_chain:
+            filename: /certs/current/sds_cert.pem
+          private_key:
+            filename: /certs/current/sds_key.pem
+          watched_directory:
+            path: /certs
+
+.. code-block:: yaml
+
+    resources:
+      - "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
+        validation_context:
+          trusted_ca:
+            filename: /certs/current/cacert.pem
+          watched_directory:
+            path: /certs
+
+Secret rotation can be performed with:
+
+.. code-block:: bash
+
+  ln -s <path to new secrets> /certs/new && mv -Tf /certs/new /certs/current
 
 Statistics
 ----------
@@ -261,3 +316,12 @@ For upstream clusters, they are in the *cluster.<CLUSTER_NAME>.client_ssl_socket
 
      ssl_context_update_by_sds, Total number of ssl context has been updated.
      upstream_context_secrets_not_ready, Total number of upstream connections reset due to empty ssl certificate.
+
+SDS has a :ref:`statistics <subscription_statistics>` tree rooted in the *sds.<SECRET_NAME>.*
+namespace. In addition, the following statistics are tracked in this namespace:
+
+.. csv-table::
+     :header: Name, Description
+     :widths: 1, 2
+
+     key_rotation_failed, Total number of filesystem key rotations that failed outside of an SDS update.

docs/root/version_history/current.rst

@@ -63,6 +63,9 @@ New Features
 * overload: add :ref:`envoy.overload_actions.reduce_timeouts <config_overload_manager_overload_actions>` overload action to enable scaling timeouts down with load.
 * ratelimit: added support for use of various :ref:`metadata <envoy_v3_api_field_config.route.v3.RateLimit.Action.metadata>` as a ratelimit action.
 * ratelimit: added :ref:`disable_x_envoy_ratelimited_header <envoy_v3_api_msg_extensions.filters.http.ratelimit.v3.RateLimit>` option to disable `X-Envoy-RateLimited` header.
+* sds: improved support for atomic :ref:`key rotations <xds_certificate_rotation>` and added configurable rotation triggers for
+  :ref:`TlsCertificate <envoy_v3_api_field_extensions.transport_sockets.tls.v3.TlsCertificate.watched_directory>` and
+  :ref:`CertificateValidationContext <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.watched_directory>`.
 * tcp: added a new :ref:`envoy.overload_actions.reject_incoming_connections <config_overload_manager_overload_actions>` action to reject incoming TCP connections.
 * tls: added support for RSA certificates with 4096-bit keys in FIPS mode.
 * tracing: added SkyWalking tracer.