Skip to content

Commit

Permalink
Refactor ClusterMap (#785)
Browse files Browse the repository at this point in the history
  • Loading branch information
@XAMPPRocky
XAMPPRocky authored Sep 20, 2023
1 parent 939429a commit 1d2bec6
Show file tree
Hide file tree
Showing 59 changed files with 744 additions and 1,223 deletions.
6 changes: 2 additions & 4 deletions agones/src/sidecar.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,8 @@ filters:
on_write: DO_NOTHING
bytes: c2lkZWNhcg== # sidecar
clusters:
default:
localities:
- endpoints:
- address: 127.0.0.1:7654
- endpoints:
- address: 127.0.0.1:7654
"#;

let config_map = config_maps
Expand Down
5 changes: 4 additions & 1 deletion agones/src/xds.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,6 @@ filters:
.as_mut()
.map(|annotations| annotations.remove(token_key).unwrap());
gameservers.replace(name.as_str(), &pp, &gs).await.unwrap();

// now we should send a packet, and not get a response.
let mut failed = false;
for i in 0..30 {
Expand All @@ -197,6 +196,10 @@ filters:
break;
}
}
if !failed {
debug_pods(&client, "role=proxy".into()).await;
debug_pods(&client, "role=xds".into()).await;
}
assert!(failed, "Packet should have failed");
}

Expand Down
2 changes: 1 addition & 1 deletion benches/throughput.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ fn run_quilkin(port: u16, endpoint: SocketAddr) {
let runtime = tokio::runtime::Runtime::new().unwrap();
let config = Arc::new(quilkin::Config::default());
config.clusters.modify(|clusters| {
clusters.insert_default(vec![quilkin::endpoint::Endpoint::new(endpoint.into())])
clusters.insert_default([quilkin::endpoint::Endpoint::new(endpoint.into())].into())
});

let proxy = quilkin::cli::Proxy {
Expand Down
1 change: 1 addition & 0 deletions build.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
"proto/data-plane-api/envoy/type/metadata/v3/metadata.proto",
"proto/data-plane-api/envoy/type/tracing/v3/custom_tag.proto",
"proto/quilkin/relay/v1alpha1/relay.proto",
"proto/quilkin/config/v1alpha1/config.proto",
"proto/quilkin/filters/capture/v1alpha1/capture.proto",
"proto/quilkin/filters/compress/v1alpha1/compress.proto",
"proto/quilkin/filters/concatenate_bytes/v1alpha1/concatenate_bytes.proto",
Expand Down
68 changes: 34 additions & 34 deletions docs/src/deployment/quickstarts/agones-xonotic-xds.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@

## 1. Overview

In this quickstart, we'll be setting up an example [Xonotic](https://xonotic.org/) [Agones](https://agones.dev/)
In this quickstart, we'll be setting up an example [Xonotic](https://xonotic.org/) [Agones](https://agones.dev/)
Fleet, that will only be accessible through Quilkin, via utilising the [TokenRouter]
Filter to provide routing and access control to the Allocated `GameServer` instances.
Filter to provide routing and access control to the Allocated `GameServer` instances.

To do this, we'll take advantage of the Quilkin [Agones xDS Provider](../../services/xds/providers/agones.md) to provide
an out-of-the-box control plane for integration between Agones and [Quilkin's xDS configuration API](../../services/xds.md) with
To do this, we'll take advantage of the Quilkin [Agones xDS Provider](../../services/xds/providers/agones.md) to provide
an out-of-the-box control plane for integration between Agones and [Quilkin's xDS configuration API](../../services/xds.md) with
minimal effort.

## 2. Install Quilkin Agones xDS Provider
Expand All @@ -26,11 +26,11 @@ kubectl apply -f https://raw.githubusercontent.com/googleforgames/quilkin/{{GITH

This applies several resources to your cluster:

1. A [ConfigMap] with a [Capture] and [TokenRouter] Filter set up to route packets to Endpoints, to be the base
1. A [ConfigMap] with a [Capture] and [TokenRouter] Filter set up to route packets to Endpoints, to be the base
configuration for all the Quilkin proxies.
2. Appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions for the
2. Appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions for the
`quilkin manage agones` process to inspect Agones resources.
3. A matching [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) that runs the
3. A matching [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) that runs the
`quilkin manage process` xDS control plane and a
[Service](https://kubernetes.io/docs/concepts/services-networking/service/) that the Quilkin proxies can connect to,
to get their Filter and Endpoint configuration from.
Expand All @@ -44,7 +44,7 @@ quilkin-manage-agones-54b787654-9dbvp 1/1 Running 0 76s
```

We can now run `kubectl get service quilkin-manage-agones` and see the
service that is generated in front of the above Deployment for our Quilkin proxies to connect to and receive their
service that is generated in front of the above Deployment for our Quilkin proxies to connect to and receive their
configuration information from.

```shell
Expand All @@ -55,55 +55,55 @@ quilkin-manage-agones ClusterIP 10.104.2.72 <none> 80/TCP 1m23s

## 3. Install Quilkin Proxy Pool

To install the Quilkin Proxy pool which connects to the above xDS provider, we can create a Deployment of Quilkin
To install the Quilkin Proxy pool which connects to the above xDS provider, we can create a Deployment of Quilkin
proxy instances that point to the aforementioned Service, like so:

```shell
kubectl apply -f https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/proxy-pool.yaml
```

This will set up three instances of Quilkin running as `quilkin proxy --management-server http://quilkin-manage-agones:80`
This will set up three instances of Quilkin running as `quilkin proxy --management-server http://quilkin-manage-agones:80`
all connected to the `quilkin-manage-agones` service.

Now we can run `kubectl get pods` until we see that the Pods for the proxy Deployment is up and running.

```shell
$ kubectl get pods
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
quilkin-manage-agones-54b787654-9dbvp 1/1 Running 0 5m7s
quilkin-proxies-78965c446d-dqvjg 1/1 Running 0 6s
quilkin-proxies-78965c446d-fr6zs 1/1 Running 0 6s
quilkin-proxies-78965c446d-m4rr7 1/1 Running 0 6s
```

Let's take this one step further, and check the configuration of the proxies that should have come from the `quilkin
Let's take this one step further, and check the configuration of the proxies that should have come from the `quilkin
manage agones` instance.

In another terminal, run: `kubectl port-forward deployments/quilkin-proxies 8000`, to port forward the
[admin endpoint](../admin.md) locally, which we can then query.

Go back to your original terminal and run `curl -s http://localhost:8000/config`
Go back to your original terminal and run `curl -s http://localhost:8000/config`

> If you have [jq](https://stedolan.github.io/jq/) installed, run `curl -s http://localhost:8000/config | jq` for a
> nicely formatted JSON output.
```shell
$ curl -s http://localhost:8000/config
{"admin":{"address":"0.0.0.0:8000"},"clusters":{},"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-dqvjg","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
{"admin":{"address":"0.0.0.0:8000"},"clusters":{},"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-dqvjg","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
```

This shows us the current configuration of the proxies coming from the xDS server created via `quilkin manage
agones`. The most interesting part that we see here, is that we have a matching set of
[Filters](../../services/proxy/filters.md) that are found in the `ConfigMap` in the
This shows us the current configuration of the proxies coming from the xDS server created via `quilkin manage
agones`. The most interesting part that we see here, is that we have a matching set of
[Filters](../../services/proxy/filters.md) that are found in the `ConfigMap` in the
[xds-control-plane.yaml](https://github.com/googleforgames/quilkin/blob/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/xds-control-plane.yaml)
we installed earlier.

## 4. Create the Agones Fleet

Now we will create an [Agones Fleet](https://agones.dev/site/docs/reference/fleet/) to spin up all our Xonotic
Now we will create an [Agones Fleet](https://agones.dev/site/docs/reference/fleet/) to spin up all our Xonotic
game servers.

Thankfully, Agones Fleets require no specific configuration to work with Quilkin proxies, so this yaml is a
Thankfully, Agones Fleets require no specific configuration to work with Quilkin proxies, so this yaml is a
[standard Agones Fleet configuration](https://github.com/googleforgames/quilkin/blob/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/fleet.yaml)

```shell
Expand All @@ -126,18 +126,18 @@ To let the Quilkin xDS provider know what token will route to which `GameServer`
`quilkin.dev/tokens` annotation to an allocated `GameServer`, with the token content as its value.

> This token would normally get generated by some kind of player authentication service and passed to the client
> via the matchmaking service - but for demonstrative purposes, we've hardcoded it into the example
> via the matchmaking service - but for demonstrative purposes, we've hardcoded it into the example
> `GameServerAllocation`.
Since you can add annotations to `GameServers` at
[allocation time](https://agones.dev/site/docs/reference/gameserverallocation/), we can both allocate a `GameServer`
Since you can add annotations to `GameServers` at
[allocation time](https://agones.dev/site/docs/reference/gameserverallocation/), we can both allocate a `GameServer`
and apply the annotation at the same time!

```shell
kubectl create -f https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/gameserverallocation.yaml
```

If we check our `GameServers` now, we should see that one of them has moved to the `Allocated` state, marking it as
If we check our `GameServers` now, we should see that one of them has moved to the `Allocated` state, marking it as
having players playing on it, and therefore it is protected by Agones until the game session ends.

```shell
Expand All @@ -150,7 +150,7 @@ xonotic-d7rfx-sn5d6 Ready 34.168.170.51 7036 gke-agones-default-534a

> Don't do this more than once, as then multiple allocated `GameServers` will have the same routing token!
If we `kubectl describe gameserver <allocated-gameserver>` and have a look at the annotations section, we
If we `kubectl describe gameserver <allocated-gameserver>` and have a look at the annotations section, we
should see something similar to this:

```shell
Expand All @@ -168,7 +168,7 @@ Kind: GameServer
...
```

Where we can see that there is now an annotation of `quilkin.dev/tokens` with the base64 encoded version of `456` as
Where we can see that there is now an annotation of `quilkin.dev/tokens` with the base64 encoded version of `456` as
our authentication and routing token ("NDU2").

> You should use something more cryptographically random than `456` in your application.
Expand All @@ -177,22 +177,22 @@ Let's run `curl -s http://localhost:8000/config` again, so we can see what has c

```shell
$ curl -s http://localhost:8000/config
{"admin":{"address":"0.0.0.0:8000"},"clusters":{"default":{"localities":[{"locality":null,"endpoints":[{"address":"34.168.170.51:7226","metadata":{"quilkin.dev":{"tokens":["NDU2"]}}}]}]}},"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-tfgsj","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
{"admin":{"address":"0.0.0.0:8000"},"clusters": [{"locality":null,"endpoints":[{"address":"34.168.170.51:7226","metadata":{"quilkin.dev":{"tokens":["NDU2"]}}}]}],"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-tfgsj","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
```

Looking under `clusters` > `localities` > `endpoints` we can see an address and token that matches up with the
Looking under `clusters` > `endpoints` we can see an address and token that matches up with the
`GameServer` record we created above!

The xDS process saw that allocated `GameServer`, turned it into a Quilkin `Endpoint` and applied the set routing
token appropriately -- without you having to write a line of xDS compliant code!
The xDS process saw that allocated `GameServer`, turned it into a Quilkin `Endpoint` and applied the set routing
token appropriately -- without you having to write a line of xDS compliant code!

## Connecting Client Side

Instead of connecting to Xonotic or an Agones `GameServer` directly, we'll want to grab the IP and exposed port of
the `Service` that fronts all our Quilkin proxies and connect to that instead -- but we'll have to append our
the `Service` that fronts all our Quilkin proxies and connect to that instead -- but we'll have to append our
routing token `456` from before, to ensure our traffic gets routed to the correct Xonotic `GameServer` address.

Run `kubectl get service quilkin-proxies` to get the `EXTERNAL-IP` of the Service you created.
Run `kubectl get service quilkin-proxies` to get the `EXTERNAL-IP` of the Service you created.

```shell
$ kubectl get service quilkin-proxies
Expand All @@ -201,14 +201,14 @@ quilkin-proxies LoadBalancer 10.109.0.12 35.246.94.14 7000:30174/UDP
```

We have a [Quilkin config yaml](https://github.com/googleforgames/quilkin/blob/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/client-token.yaml)
file all ready for you, that is configured to append the routing token `456` to each
packet that passes through it, via the power of a
file all ready for you, that is configured to append the routing token `456` to each
packet that passes through it, via the power of a
[ConcatenateBytes](../../services/proxy/filters/concatenate_bytes.md) Filter.

Download `client-token.yaml` locally, so you can edit it:

```shell
curl https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/client-token.yaml --output client-token.yaml
curl https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/client-token.yaml --output client-token.yaml
```

We then take the EXTERNAL-IP and port from the `quilkin-proxies` service, and replace the`${LOADBALANCER_IP}`
Expand Down
Loading

0 comments on commit 1d2bec6

Please sign in to comment.