googleforgames · markmandel · Mar 9, 2021 · Mar 6, 2021 · Mar 9, 2021 · Mar 9, 2021
@@ -1,7 +1,7 @@
 # CaptureBytes
 
 The `CaptureBytes` filter's job is to find a series of bytes within a packet, and capture it into
-[Filter dynamic metadata]`(TODO: add link to dynamic metadata docs)`, so that it can be utilised by filters further
+[Filter Dynamic Metadata][filter-dynamic-metadata], so that it can be utilised by filters further
 down the chain.
 
 This is often used as a way of retrieving authentication tokens from a packet, and used in combination with
@@ -71,3 +71,6 @@ properties:
 * `quilkin_filter_CaptureBytes_packets_dropped`  
   A counter of the total number of packets that have been dropped due to their length being less than the configured
   `size`.
+
+
+[filter-dynamic-metadata]: ./filter.md#filter-dynamic-metadata
@@ -59,6 +59,27 @@ The above example creates a filter chain comprising a [Debug](debug.md) filter f
 
 > The sequence determines the filter chain order so its ordering matters - the chain starts with the filter corresponding the first filter config and ends with the filter corresponding the last filter config in the sequence.
 
+### Filter Dynamic Metadata
+
+A filter within the filter chain can share data within another filter further along in the filter chain by propagating the desired data alongside the packet being processed.
+This enables sharing dynamic information at runtime, e.g information about the current packet that might be useful to other filters that process that packet.
+
+At packet processing time each packet is associated with _filter dynamic metadata_ (a set of key-value pairs). Each key is a unique string while value is an arbitrary value.
+When a filter processes a packet, it can choose to consult the associated dynamic metadata for more information or itself add/update or remove key-values from the set.
+
+As an example, the built-in [CaptureBytes] filter is one such filter that populates a packet's filter metadata.
+[CaptureBytes] extracts information (a configurable byte sequence) from each packet and appends it to the packet's dynamic metadata for other filters to leverage.
+On the other hand, the built-in [TokenRouter] filter selects what endpoint to route a packet by consulting the packet's dynamic metadata for a routing token.
+Consequently, we can build a filter chain with a [CaptureBytes] filter preceeding a [TokenRouter] filter, both configured to write and read the same key in the dynamic metadata entry. The effect would be that packets are routed to upstream endpoints based on token information extracted from their contents.
+
+#### Well Known Dynamic Metadata
+
+The following metadata are currently used by Quilkin core and built-in filters.
+
+| Name | Type | Description |
+|------|------|-------------|
+| `quilkin.dev/captured_bytes` | `Vec<u8>` | The default key under which the [CaptureBytes] filter puts the byte slices it extracts from each packet. |
+
 ### Built-in filters <a name="built-in-filters"></a>
 Quilkin includes several filters out of the box.
 
@@ -91,3 +112,6 @@ properties:
 
 required: [ 'name', 'config' ]
 ```
+
+[CaptureBytes]: ./capture_bytes.md
+[TokenRouter]: ./token_router.md
@@ -3,8 +3,8 @@
 The `TokenRouter` filter's job is to provide a mechanism to declare which Endpoints a packet should be sent to.   
 
 This Filter provides this functionality by comparing a byte array token found in the
-[Filter dynamic metadata]`(TODO: add link to dynamic metadata docs)` from a previous Filter, and comparing it to
-Endpoint's connection_id values, and sending packets to those Endpoints only if there is a match.
+[Filter Dynamic Metadata][filter-dynamic-metadata] from a previous Filter, and comparing it to
+[Endpoint's tokens][endpoint-tokens], and sending packets to those Endpoints only if there is a match.
 
 #### Filter name
 ```text
@@ -101,3 +101,6 @@ static:
 
 On the game client side the [ConcatenateBytes](./concatenate_bytes.md) filter could also be used to add authentication
 tokens to outgoing packets.
+
+[filter-dynamic-metadata]: ./filter.md#filter-dynamic-metadata
+[endpoint-tokens]: ../../proxy#upstream-endpoint
@@ -92,6 +92,23 @@ definitions:
     type: array
     description: |
       A list of upstream endpoints to forward packets to.
+    items:
+      type: object
+        description: |
+          An upstream endpoint
+        properties:
+          address:
+            type: string
+            description: |
+              Socket address of the endpoint. This must be of the ´IP:Port` form e.g `192.168.1.1:7001`
+            metadata:
+              type: object
+              description: |
+                Arbitrary key value pairs that is associated with the endpoint.
+                These are visible to Filters when processing packets and can be used to provide more context about endpoints (e.g whether or not to route a packet to an endpoint).
+                Keys must be of type string otherwise the configuration is rejected.
+      required:
+        - address
 ```
 
 [examples]: ../examples

@@ -1,9 +1,45 @@
 ### Proxy
 
+#### Concepts
+
+##### Upstream Endpoint
+
+An Upstream Endpoint represents a server that Quilkin forwards packets to.
+It is represented by an IP address and port. An upstream endpoint can optionally be associated with a (potentially empty) set of tokens as well as metadata.
+
+###### Endpoint Metadata
+
+Arbitrary key value pairs that is associated with the endpoint.
+These are visible to Filters when processing packets and can be used to provide more context about endpoints (e.g whether or not to route a packet to an endpoint).
+Keys must be of type string otherwise the configuration is rejected.
+
+Metadata associated with an endpoint contain arbitrary key value pairs which [Filters][filters-doc] can consult when processing packets (e.g they can contain information that determine whether or not to route a particular packet to an endpoint).
+
+In fact, the tokens associated with an endpoint are simply a special piece of metadata well known to Quilkin and is used by the built-in [TokenRouter] filter to route packets.
+Such well known values are placed within an object in the endpoint metadata, under the special key `quilkin.dev`. Currently, only the `tokens` entry is in use.
+
+As an example, the following shows the configuration for an endpoint with its metadata:
+```yaml
+static:
+  endpoints:
+    - address: 127.0.0.1:26000
+      metadata:
+        canary: false
+        quilkin.dev: # This object is extracted by Quilkin and is usually reserved for built-in features
+          tokens:
+            - MXg3aWp5Ng== # base64 for 1x7ijy6
+            - OGdqM3YyaQ== # base64 for 8gj3v2i
+```
+
+An endpoint's metadata can be specified alongside the endpoint in [static configuration][proxy-configuration] or using the [xDS endpoint metadata][xds-endpoint-metadata] field when using [dynamic configuration][dynamic-configuration-doc] via xDS.
+
+##### Session
+
+A session represents ongoing communication flow between a client and an [Upstream Endpoint][endpoint]. See the [Session documentation][sessions-doc] for more information.
 
 #### Metrics
 
-The proxy exposes the following core metrics:
+The proxy exposes the following general metrics (See the metrics sub-sections for metrics specific to other Quilkin components, e.g for metrics related to packet flow see [sessions metrics][session-metrics], or metrics exported by individual filters can be found in the documentation for each filter):
 
 - `quilkin_proxy_packets_dropped_total{reason}` (Counter)
 
@@ -12,4 +48,19 @@ The proxy exposes the following core metrics:
   * `reason = NoConfiguredEndpoints`
     - `NoConfiguredEndpoints`: No upstream endpoints were available to send the packet to. This can occur e.g if the endpoints cluster was scaled down to zero and the proxy is configured via a control plane.
 
-[session-metrics]: ./session.md
+- `quilkin_cluster_active` (Gauge)
+
+  The number of currently active clusters.
+
+- `quilkin_endpoints_active` (Gauge)
+
+  The number of currently active upstream endpoints. Note that this tracks the number of endpoints that the proxy knows of rather than those that it is connected to (see [Session Metrics][session-metrics] instead for those)
+
+[sessions-doc]: ./session.md
+[session-metrics]: ./session.md#metrics
+[filters-doc]: ./extensions/filters/filters.md
+[endpoint]: #upstream-endpoint
+[proxy-configuration]: ./proxy-configuration.md
+[xds-endpoint-metadata]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#envoy-v3-api-field-config-endpoint-v3-lbendpoint-metadata
+[dynamic-configuration-doc]: ./xds.md
+[TokenRouter]: ./extensions/filters/token_router.md
@@ -0,0 +1,82 @@
+### Dynamic Configuration using xDS Management Servers
+
+In addition to static configuration provided upon startup, a Quiklin proxy's configuration can also be updated at runtime. The proxy can be configured on startup to talk to a set of management servers which provide it with updates throughout its lifecycle.
+
+Communication between the proxy and management server uses the [xDS gRPC protocol][xDS], similar to an [envoy proxy]. xDS is one of the standard configuration mechanisms for software proxies and as a result, Quilkin can be setup to discover configuration resources from any API compatible server. Also, given that the protocol is [well specified][xDS-protocol], it is similarly straight-forward to implement a custom server to suit any deployment's needs.
+
+> The [go-control-plane] project provides production ready implementations of the API on top of which custom servers can be built relatively easily.
+
+As described within the [xDS-api] documentation, the xDS API comprises a set of resource discovery APIs, each serving a specific set of configuration resource types, while the protocol itself comes in several [variants][xds-variants].
+Quilkin implements the **Aggregated Discovery Service (ADS)** _State of the World (SotW)_ variant with gRPC.
+
+#### Supported APIs
+
+Since the range of resources configurable by the xDS API extends that of Quilkin's domain (i.e being UDP based, Quilkin does not have a need for HTTP/TCP resources), only a subset of the API is supported. The following lists these relevant parts and any limitation to the provided support as a result:
+
+- **Cluster Discovery Service [(CDS)][CDS]**: Provides information about known clusters and their membership information.
+  * The proxy uses these resources to discover clusters and their endpoints.
+  * While cluster topology information like [locality] can be provided in the configuration, the proxy currently does not use this information (support may be included in the future however).
+  * Any [load balancing information][lbpolicy] included in this resource is ignored. For load balancing, use [Quilkin filters][filters-doc] instead.
+  * Only [cluster discovery type] `STATIC` and `EDS` is supported. Configuration including other discovery types e.g `LOGICAL_DNS` is rejected.
+
+- **Endpoint Discovery Service [(EDS)][EDS]**: Provides information about endpoints.
+  * The proxy uses these resources to discover information about endpoints like their IP addresses.
+  * Endpoints may provide [Endpoint Metadata][endpoint-metadata] via the [metadata][xds-endpoint-metadata] field. These metadata will be visible to filters as part of the corresponding endpoints information when processing packets.
+  * Only [socket addresses] are supported on an endpoint's address configuration - i.e an IP address and port number combination. Configuration including any other type of addressing e.g named pipes will be rejected.
+  * Any [load balancing information][clapolicy] included in this resource is ignored. For load balancing, use [Quilkin filters][filters-doc] instead.
+
+- **Listener Discovery Service [(LDS)][LDS]**: Provides information about [Filters and Filter Chains][filters-doc].
+  * Only the `name` and `filter_chains` fields in the [Listener resource][listener-resource] are used by the proxy. The rest are ignored.
+  * Since Quilkin only uses one filter chain per proxy, at most one filter chain can be provided in the resource. Otherwise the configuration is rejected.
+  * Only the list of [filters][xds-filters] specified in the [filter chain][xds-filter-chain] is used by the proxy - i.e other fields like `filter_chain_match` are ignored. This list also specifies the order that the corresponding filter chain will be constructed.
+  * gRPC proto configuration for Quilkin's built-in filters [can be found here][filter-protos]. They are equivalent to the filter's static configuration.
+
+
+#### Metrics
+
+Quilkin exposes the following metrics around the management servers and its resources:
+
+- `quilkin_xds_connected_state` (Gauge)
+
+  A boolean that indicates whether or not the proxy is currently connected to a management server. A value `1` means that the proxy is connected while `0` means that it is not connected to any server at that point in time.
+
+- `quilkin_xds_update_attempt_total` (Counter)
+
+  The total number of attempts made by a management server to configure the proxy. This is equivalent to the total number of configuration updates received by the proxy from a management server.
+
+- `quilkin_xds_update_success_total` (Counter)
+
+  The total number of successful attempts made by a management server to configure the proxy. This is equivalent to the total number of configuration updates received by the proxy from a management server and was successfully applied by the proxy.
+
+- `quilkin_xds_update_failure_total` (Counter)
+
+  The total number of unsuccessful attempts made by a management server to configure the proxy. This is equivalent to the total number of configuration updates received by the proxy from a management server and was rejected by the proxy (e.g due to a bad/inconsistent configuration).
+
+- `quilkin_xds_requests_total` (Counter)
+
+  The total number of [DiscoveryRequest]s made by the proxy to management servers. This tracks messages flowing in the direction from the proxy to the management server.
+
+
+[xDS]: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#xds-rest-and-grpc-protocol
+[envoy proxy]: https://www.envoyproxy.io/docs/envoy/latest/
+[xDS-protocol]: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#the-xds-transport-protocol
+[go-control-plane]: https://github.com/envoyproxy/go-control-plane
+[xDS-api]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration
+[CDS]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration#cds
+[EDS]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration#eds
+[LDS]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration#lds
+[cluster discovery type]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#enum-config-cluster-v3-cluster-discoverytype
+[lbpolicy]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#enum-config-cluster-v3-cluster-lbpolicy
+[clapolicy]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint.proto#config-endpoint-v3-clusterloadassignment-policy
+[locality]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#config-core-v3-locality
+[socket addresses]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#config-core-v3-address
+[filters-doc]: ./extensions/filters/filters.md
+[listener-resource]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/listener/v3/listener.proto#config-listener-v3-listener
+[xds-filters]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/listener/v3/listener_components.proto#envoy-v3-api-msg-config-listener-v3-filter
+[xds-filter-chain]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/listener/v3/listener_components.proto#config-listener-v3-filterchain
+[DiscoveryRequest]: https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/discovery.proto#envoy-api-msg-discoveryrequest
+[xds-variants]: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#variants-of-the-xds-transport-protocol
+[filter-protos]: ../proto/quilkin/extensions/filters
+[filters-doc]: ./extensions/filters/filters.md
+[xds-endpoint-metadata]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#envoy-v3-api-msg-config-core-v3-metadata
+[endpoint-metadata]: ./proxy.md#endpoint-metadata
@@ -26,10 +26,14 @@ static: # Provide static configuration of endpoints
   endpoints: # array of potential endpoints to send on traffic to
     - name: Game Server No. 1
       address: 127.0.0.1:26000
-      connection_ids:
-        - MXg3aWp5Ng== # the connection byte array to route to, encoded as base64 (string value: 1x7ijy6)
-        - OGdqM3YyaQ== # (string value: 8gj3v2i)
+      metadata: # Metadata associated with the endpoint
+        quilkin.dev:
+          tokens:
+            - MXg3aWp5Ng== # the connection byte array to route to, encoded as base64 (string value: 1x7ijy6)
+            - OGdqM3YyaQ== # (string value: 8gj3v2i)
     - name: Game Server No. 2
       address: 127.0.0.1:26001
-      connection_ids:
-        - bmt1eTcweA== # (string value: nkuy70x)
+      metadata: # Metadata associated with the endpoint
+        quilkin.dev:
+          tokens:
+            - bmt1eTcweA== # (string value: nkuy70x)
@@ -409,13 +409,15 @@ static:
   endpoints:
     - address: 127.0.0.1:26000
       metadata:
-        tokens:
-          - MXg3aWp5Ng== #1x7ijy6
-          - OGdqM3YyaQ== #8gj3v2i
+        quilkin.dev:
+          tokens:
+            - MXg3aWp5Ng== #1x7ijy6
+            - OGdqM3YyaQ== #8gj3v2i
     - address: 127.0.0.1:26001
       metadata:
-        tokens:
-          - bmt1eTcweA== #nkuy70x";
+        quilkin.dev:
+          tokens:
+            - bmt1eTcweA== #nkuy70x";
         let config = parse_config(yaml);
         assert_static_endpoints(
             &config.source,
@@ -424,9 +426,14 @@ static:
                     "127.0.0.1:26000".parse().unwrap(),
                     Some(
                         serde_yaml::to_value(
-                            vec![("tokens", vec!["MXg3aWp5Ng==", "OGdqM3YyaQ=="])]
-                                .into_iter()
-                                .collect::<HashMap<_, _>>(),
+                            vec![(
+                                "quilkin.dev",
+                                vec![("tokens", vec!["MXg3aWp5Ng==", "OGdqM3YyaQ=="])]
+                                    .into_iter()
+                                    .collect::<HashMap<_, _>>(),
+                            )]
+                            .into_iter()
+                            .collect::<HashMap<_, _>>(),
                         )
                         .unwrap(),
                     ),
@@ -435,9 +442,14 @@ static:
                     "127.0.0.1:26001".parse().unwrap(),
                     Some(
                         serde_yaml::to_value(
-                            vec![("tokens", vec!["bmt1eTcweA=="])]
-                                .into_iter()
-                                .collect::<HashMap<_, _>>(),
+                            vec![(
+                                "quilkin.dev",
+                                vec![("tokens", vec!["bmt1eTcweA=="])]
+                                    .into_iter()
+                                    .collect::<HashMap<_, _>>(),
+                            )]
+                            .into_iter()
+                            .collect::<HashMap<_, _>>(),
                         )
                         .unwrap(),
                     ),