From 648986c999eca7ba381e51413881dde47e07d8fe Mon Sep 17 00:00:00 2001 From: Mark Mandel Date: Thu, 29 Jun 2023 15:25:56 -0700 Subject: [PATCH] Update Docs for 0.6.0 Release This updates documentation for the upcoming release, and fixes up documentation for some of the new areas and concepts. This includes: * Moved "Configuration" into the proxy configuration section, since that is primarily what it covers now (everything else is command line arguments). * We had `xds` as a folder twice for some reason, so deleted a copy. * A bunch of smaller fixes across those areas. Work on #725 --- docs/src/SUMMARY.md | 8 +- docs/src/services/proxy.md | 23 +++--- .../proxy}/configuration.md | 27 ++++--- docs/src/services/xds.md | 11 ++- docs/src/services/xds/providers/agones.md | 11 ++- docs/src/services/xds/providers/filesystem.md | 4 +- docs/src/xds/metrics.md | 67 ----------------- docs/src/xds/providers/agones.md | 75 ------------------- docs/src/xds/providers/filesystem.md | 41 ---------- 9 files changed, 48 insertions(+), 219 deletions(-) rename docs/src/{deployment => services/proxy}/configuration.md (78%) delete mode 100644 docs/src/xds/metrics.md delete mode 100644 docs/src/xds/providers/agones.md delete mode 100644 docs/src/xds/providers/filesystem.md diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 7903b6f57e..ac9f277777 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -3,13 +3,15 @@ [FAQ](./faq.md) # Quickstart Guides + - [Netcat](./deployment/quickstarts/netcat.md) - [Agones + Xonotic (Sidecar)](./deployment/quickstarts/agones-xonotic-sidecar.md) - [Agones + Xonotic (xDS)](./deployment/quickstarts/agones-xonotic-xds.md) # Services + - [Proxy](./services/proxy.md) - - [Control Message Protocol](./services/proxy/qcmp.md) + - [Configuration File](./services/proxy/configuration.md) - [Filters](./services/proxy/filters.md) - [Capture](./services/proxy/filters/capture.md) - [Compress](./services/proxy/filters/compress.md) @@ -24,6 +26,7 @@ - [Timestamp](./services/proxy/filters/timestamp.md) - [Token Router](./services/proxy/filters/token_router.md) - [Writing Custom Filters](./services/proxy/filters/writing_custom_filters.md) + - [Control Message Protocol](./services/proxy/qcmp.md) - [Metrics](./services/proxy/metrics.md) --- @@ -41,9 +44,10 @@ - [Providers]() # SDKs + - [Unreal Engine](./sdks/unreal-engine.md) # Deployment + - [Administration](./deployment/admin.md) - [Architecture Examples](./deployment/examples.md) -- [Configuration](./deployment/configuration.md) diff --git a/docs/src/services/proxy.md b/docs/src/services/proxy.md index d9c26fd630..08adac1e66 100644 --- a/docs/src/services/proxy.md +++ b/docs/src/services/proxy.md @@ -1,8 +1,8 @@ # Proxy -| services | ports | Protocol | -|----------|-------|-----------| -| proxy | 7777 | UDP (IPv4) | +| services | ports | Protocol | +|----------|-------|------------| +| proxy | 7777 | UDP (IPv4) | "Proxy" is the primary Quilkin service, which acts as a non-transparent UDP proxy. @@ -16,7 +16,7 @@ $ quilkin proxy --help ## Endpoints -An Endpoint represents an address that Quilkin forwards packets to that it has recieved from the +An Endpoint represents an address that Quilkin forwards packets to that it has received from the source port. It is represented by an IP address and port. An Endpoint can optionally be associated with an arbitrary set of @@ -25,20 +25,20 @@ It is represented by an IP address and port. An Endpoint can optionally be assoc ## Proxy Filters Filters are the way for a Quilkin proxy to intercept UDP packet traffic from the -source and [Endpoints][Endpoint] in either direction, and be able to inpsect, +source and [Endpoints][Endpoint] in either direction, and be able to inspect, manipulate, and route the packets as desired. -See [Filters][filters-doc] for a deeper dive into Filters, as well as the list of build in Filters that come with +See [Filters] for a deeper dive into Filters, as well as the list of build in Filters that come with Quilkin. ## Endpoint Metadata -Enpoint metadata is an arbitrary set of key value pairs that are associated with an Endpoint. +Endpoint metadata is an arbitrary set of key value pairs that are associated with an Endpoint. These are visible to Filters when processing packets and can be used to provide more context about endpoints (e.g whether or not to route a packet to an endpoint). Keys must be of type string otherwise the configuration is rejected. -Metadata associated with an endpoint contain arbitrary key value pairs which [Filters][filters-doc] can consult when processing packets (e.g they can contain information that determine whether or not to route a particular packet to an endpoint). +Metadata associated with an endpoint contain arbitrary key value pairs which [Filters] can consult when processing packets (e.g they can contain information that determine whether or not to route a particular packet to an endpoint). ### Specialist Endpoint Metadata @@ -73,7 +73,7 @@ Quilkin uses the "Session" concept to track traffic flowing through the proxy be Session serves the same purpose, and can be thought of as a lightweight version of a `TCP` session in that, while a TCP session requires a protocol to establish and teardown: -- A Quilkin session is automatically created upon receiving the first packet from a client via the [Local Port], to be +- A Quilkin session is automatically created upon receiving the first packet from a client via the [Local Port] to be sent to an upstream [Endpoint]. - The session is automatically deleted after a period of inactivity (where no packet was sent between either party) - currently 60 seconds. @@ -83,11 +83,12 @@ downstream endpoint which initiated the communication with Quilkin and the serve that Quilkin proxies traffic to. Sessions are established *after* the filter chain completes. The destination Endpoint of a packet is determined by -the [filter chain][filter-doc], so a Session can only be created after filter chain completion. For example, if the +the [filter chain][Filters], so a Session can only be created after filter chain completion. For example, if the filter chain drops all packets, then no session will ever be created. [Endpoint]: #endpoints -[file-configuration]: ../deployment/configuration.md +[file-configuration]: ./proxy/configuration.md [xds-endpoint-metadata]: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#envoy-v3-api-field-config-endpoint-v3-lbendpoint-metadata [dynamic-configuration-doc]: ./xds.md [TokenRouter]: ./proxy/filters/token_router.md +[Filters]: ./proxy/filters.md \ No newline at end of file diff --git a/docs/src/deployment/configuration.md b/docs/src/services/proxy/configuration.md similarity index 78% rename from docs/src/deployment/configuration.md rename to docs/src/services/proxy/configuration.md index e359365690..e871915a6a 100644 --- a/docs/src/deployment/configuration.md +++ b/docs/src/services/proxy/configuration.md @@ -1,9 +1,13 @@ -# Configuration +# Configuration File -The following is the schema and reference for a Quilkin configuration +While much of Quilkin's proxy configuration can be configured via its +[command line interface](../proxy.md), if you have a larger or more complex configuration +it might be useful to use a configuration file instead. + +The following is the schema and reference for Quilkin's proxy configuration file. See the [examples] folder for example configuration files. -By default Quilkin will look for a configuration file named `quilkin.yaml` in +By default, Quilkin will look for a configuration file named `quilkin.yaml` in its current running directory first, then if not present, in `/etc/quilkin/quilkin.yaml` on UNIX systems. This can be overridden with the `-c/--config` command-line argument, or the `QUILKIN_FILENAME` @@ -12,20 +16,21 @@ environment variable. ## Static Configuration Example of a full configuration for `quilkin proxy` that utlisies a static -endpoint configuration: +endpoint configuration to specify two endpoints with `token` metadata attached to each: ```yaml -{{#include ../../../examples/proxy.yaml:17:100}} +{{#include ../../../../examples/proxy.yaml:17:100}} ``` -## Dynamic Configuration +This is a great use of a static configuration file, as we only get a singular `--to` endpoint address via the +command line arguments. -Example of a full configuration for `quilkin proxy` that utlisies a dynamic -Endpoint configuration through an [xDS management endpoint](../services/xds.md): +We can also configure [Filters](./filters.md) via the configuration file. See that section for documentation. -```yaml -{{#include ../../../examples/control-plane.yaml:17:100}} -``` +## Dynamic Configuration + +If you need to dynamically change either Filters and/or Endpoints at runtime, see the [Control Plane](../xds.md) +documentation on the configuration API surface, and built in dynamic management providers. ## Json Schema diff --git a/docs/src/services/xds.md b/docs/src/services/xds.md index 857bc0c3b7..7371e36037 100644 --- a/docs/src/services/xds.md +++ b/docs/src/services/xds.md @@ -1,12 +1,12 @@ # xDS Control Plane -| services | ports | Protocol | -|----------|-------|-----------| -| xDS | 7800 | gRPC(IPv4) | +| services | ports | Protocol | +|----------|-------|------------| +| xDS | 7800 | gRPC(IPv4) | For multi-cluster integration, Quilkin provides a `manage` service, that can be used with a number of configuration discovery providers to provide cluster -configuration multiple [`proxy`s](./proxy.md). With each provider automationing the +configuration multiple [`proxy`s](./proxy.md). With each provider automating the complexity of a full xDS management control plane via integrations with popular projects and common architecture patterns. @@ -51,8 +51,7 @@ Since the range of resources configurable by the xDS API extends that of Quilkin ## Connecting to an xDS management server Connecting a Quilkin proxy to an xDS management server can be implemented via providing one or more URLs to -the `management_servers` [command line](../../api/quilkin/struct.Proxy.html#structfield.management_server) or -[file configuration](../deployment/configuration.md#dynamic-configuration). +the `management_servers` [command line](../../api/quilkin/struct.Proxy.html#structfield.management_server). [xDS]: https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#xds-rest-and-grpc-protocol diff --git a/docs/src/services/xds/providers/agones.md b/docs/src/services/xds/providers/agones.md index c61a49658c..33f15f6516 100644 --- a/docs/src/services/xds/providers/agones.md +++ b/docs/src/services/xds/providers/agones.md @@ -38,11 +38,12 @@ annotations: ## Filter Configuration -The Agones provider watches for a singular [`ConfigMap`](https://kubernetes.io/docs/concepts/configuration/configmap/) -that has the label of `quilkin.dev/configmap: "true"`, and any changes that happen to it, and use its contents to -send Filter configuration to any connected Quilkin proxies. +The Agones provider watches for a singular [`ConfigMap`](https://kubernetes.io/docs/concepts/configuration/configmap/) +that has the label of `quilkin.dev/configmap: "true"`, and any changes that happen to it, and use its contents to +send [Filter] configuration to any connected Quilkin proxies. -The `ConfigMap` contents should be a valid Quilkin [file configuration](../../../deployment/configuration.md), but with no Endpoint data. +The `ConfigMap` contents should be a valid Quilkin [file configuration][configuration], but with no +Endpoint data. For example: @@ -72,3 +73,5 @@ and [RBAC] Rules, there is an [Agones, xDS and Xonotic example][example]. [Services]: https://kubernetes.io/docs/concepts/services-networking/service/ [RBAC]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ [example]: https://github.com/googleforgames/quilkin/tree/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds +[Filter]: ../../../services/proxy/filters.md +[configuration]: ../../../services/proxy/configuration.md \ No newline at end of file diff --git a/docs/src/services/xds/providers/filesystem.md b/docs/src/services/xds/providers/filesystem.md index ce83f7fdfa..350c1389de 100644 --- a/docs/src/services/xds/providers/filesystem.md +++ b/docs/src/services/xds/providers/filesystem.md @@ -13,7 +13,7 @@ default port is taken up by an existing Quilkin proxy. After running this command, any proxy that connects to port 18000 will receive updates as configured in `config.yaml` file. -You can find the configuration file schema in [Configuration][configuration]. +You can find the configuration file schema in [Configuration File][configuration]. Example: @@ -38,4 +38,4 @@ clusters: # assert_eq!(config.filters.load().len(), 1); ``` -[configuration]: ../../../deployment/configuration.md +[configuration]: ../../../services/proxy/configuration.md diff --git a/docs/src/xds/metrics.md b/docs/src/xds/metrics.md deleted file mode 100644 index 2d6b505888..0000000000 --- a/docs/src/xds/metrics.md +++ /dev/null @@ -1,67 +0,0 @@ -# xDS Metrics - -## Proxy Mode - -Quilkin exposes the following metrics around the management servers and its resources when running as a -[UDP Proxy](../proxy-mode.md): - -- `quilkin_xds_connected_state` (Gauge) - - A boolean that indicates whether or not the proxy is currently connected to a management server. A value `1` means that the proxy is connected while `0` means that it is not connected to any server at that point in time. - -- `quilkin_xds_update_attempt_total` (Counter) - - The total number of attempts made by a management server to configure the proxy. This is equivalent to the total number of configuration updates received by the proxy from a management server. - -- `quilkin_xds_update_success_total` (Counter) - - The total number of successful attempts made by a management server to configure the proxy. This is equivalent to the total number of configuration updates received by the proxy from a management server and was successfully applied by the proxy. - -- `quilkin_xds_update_failure_total` (Counter) - - The total number of unsuccessful attempts made by a management server to configure the proxy. This is equivalent to the total number of configuration updates received by the proxy from a management server and was rejected by the proxy (e.g due to a bad/inconsistent configuration). - -- `quilkin_xds_requests_total` (Counter) - - The total number of [DiscoveryRequest]s made by the proxy to management servers. This tracks messages flowing in the direction from the proxy to the management server. - - -## xDS Provider Mode - -The following metrics are exposed when Quilkin is running as an [xDS provider](builtin.md). - -- `quilkin_management_server_connected_proxies` (Gauge) - - The number of proxies currently connected to the server. -- `quilkin_management_server_discovery_requests_total{request_type}` (Counter) - - The total number of xDS Discovery requests received across all proxies. - - `request_type` = `type.googleapis.com/envoy.config.cluster.v3.Cluster` | `type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment` | `type.googleapis.com/envoy.config.listener.v3.Listener` - Type URL of the requested resource -- `quilkin_management_server_discovery_responses_total` (Counter) - - The total number of xDS Discovery responses sent back across all proxies in response to Discovery Requests. - Each Discovery response sent corresponds to a configuration update for some proxy. - - `request_type` = `type.googleapis.com/envoy.config.cluster.v3.Cluster` | `type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment` | `type.googleapis.com/envoy.config.listener.v3.Listener` - Type URL of the requested resource -- `quilkin_management_server_endpoints_total` (Gauge) - - The number of active endpoints discovered by the server. The number of active endpoints - correlates with the size of the cluster configuration update sent to proxies. -- `quilkin_management_server_snapshot_generation_errors_total` (Counter) - - The total number of errors encountered while generating a configuration snapshot update for a proxy. -- `quilkin_management_server_snapshots_generated_total` (Counter) - - The total number of configuration snapshot generated across all proxies. A snapshot corresponds - to a point in time view of a proxy's configuration. However it does not necessarily correspond - to a proxy update - a proxy only gets the latest snapshot so it might miss intermediate - snapshots if it lags behind. -- `quilkin_management_server_snapshots_cache_size` (Gauge) - - The current number of snapshots in the in-memory snapshot cache. This corresponds 1-1 to - proxies that connect to the server. However the number may be slightly higher than the number - of connected proxies since snapshots for disconnected proxies are only periodically cleared - from the cache. - -[DiscoveryRequest]: https://www.envoyproxy.io/docs/envoy/v1.22.0/api-v3/service/discovery/v3/discovery.proto.html?highlight=discoveryrequest#service-discovery-v3-discoveryrequest diff --git a/docs/src/xds/providers/agones.md b/docs/src/xds/providers/agones.md deleted file mode 100644 index ffba313276..0000000000 --- a/docs/src/xds/providers/agones.md +++ /dev/null @@ -1,75 +0,0 @@ -# Agones xDS Provider - -The [Agones] xDS Provider is built to simplify Quilkin integration with Agones game server hosting on top of -[Kubernetes](https://kubernetes.io). - -This provider watches for changes in Agones -[`GameServer` resources](https://agones.dev/site/docs/getting-started/create-gameserver/) in a cluster, and -utilises that information to provide [Endpoint][Endpoints] information to connected Quilkin proxies. - -> Currently, the Agones provider can only discover resources within the cluster it is running in. - -## Endpoint Configuration - -This provider watches the Kubernetes clusters for `Allocated` -[Agones GameServers](https://agones.dev/site/docs/reference/gameserver/#gameserver-state-diagram) -and exposes their IP address and Port as [Endpoints] to any connected Quilkin proxies. - -> Since an Agones GameServer can have multiple ports exposed, if multiple ports are in -> use, the server will pick the first port in the port list. - -By default the Agones xDS provider will look in the `default` namespace for any `GameServer` resources, but it can be -configured via the `--gameservers-namespace` argument. - -### Access Tokens - -The set of [access tokens](../../proxy/concepts.md#specialist-endpoint-metadata) for the associated Endpoint can be -set by adding a comma separated standard base64 encoded strings. This must be added under an annotation -`quilkin.dev/tokens` in the -[GameServer](https://agones.dev/site/docs/reference/agones_crd_api_reference/#agones.dev/v1.GameServer)'s metadata. - -For example: - -```yaml -annotations: - # Sets two tokens for the corresponding endpoint with values 1x7ijy6 and 8gj3v2i respectively. - quilkin.dev/tokens: MXg3aWp5Ng==,OGdqM3YyaQ== -``` - -## Filter Configuration - -The Agones provider watches for a singular [`ConfigMap`](https://kubernetes.io/docs/concepts/configuration/configmap/) -that has the label of `quilkin.dev/configmap: "true"`, and any changes that happen to it, and use its contents to -send Filter configuration to any connected Quilkin proxies. - -The `ConfigMap` contents should be a valid Quilkin [file configuration](../../file-configuration.md), but with no -Endpoint data. - -For example: - -```yaml -{{#include ../../../../examples/agones-xonotic-xds/xds-control-plane.yaml:config-map}} -``` - -By default the Agones xDS provider will look in the `default` namespace for this `ConfigMap`, but it can be -configured via the `--config-namespace` argument. - -## Usage - -As an example, the following runs the server with subcommnad `manage agones` against a cluster (using default -kubeconfig authentication) where Quilkin pods run in the `quilkin` namespace and `GameServer` pods run in the -`gameservers` namespace: - -```sh -quilkin manage agones --config-namespace quilkin --gameservers-namespace gameservers -``` - -For a full referenmce of deploying this provider in a Kubernetes cluster, with appropriate [Deployments], [Services], -and [RBAC] Rules, there is an [Agones, xDS and Xonotic example][example]. - -[Agones]: https://agones.dev -[Endpoints]: ../../proxy/concepts.md#endpoints -[Deployments]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ -[Services]: https://kubernetes.io/docs/concepts/services-networking/service/ -[RBAC]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ -[example]: https://github.com/googleforgames/quilkin/tree/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds \ No newline at end of file diff --git a/docs/src/xds/providers/filesystem.md b/docs/src/xds/providers/filesystem.md deleted file mode 100644 index c0a76aff4f..0000000000 --- a/docs/src/xds/providers/filesystem.md +++ /dev/null @@ -1,41 +0,0 @@ -# Filesystem xDS Provider - -The filesystem provider watches a configuration file on disk and sends updates to proxies whenever that file changes. - -It can be started with using subcommnad `manage file` as the following: -```sh -quilkin manage --port 18000 file --config-file-path config.yaml -``` - -We run this on port 1800, in this example, in case you are running this locally, and the -default port is taken up by an existing Quilkin proxy. - -After running this command, any proxy that connects to port 18000 will receive updates as configured in `config.yaml` -file. - -You can find the configuration file schema in [File Configuration][file-configuration]. - -Example: - -```rust -# let yaml = " -version: v1alpha1 -filters: - - name: quilkin.filters.debug.v1alpha1.Debug - config: - id: hello -clusters: - cluster-a: - localities: - - endpoints: - - address: 123.0.0.1:29 - metadata: - 'quilkin.dev': - tokens: - - 'MXg3aWp5Ng==' -# "; -# let config = quilkin::config::Config::from_reader(yaml.as_bytes()).unwrap(); -# assert_eq!(config.filters.load().len(), 1); -``` - -[file-configuration]: ../../file-configuration.md