Implement a control plane

[iffyio]

The XDS control planes out there today seem to be directed at proxying HTTP and TCP traffic so that it seems simpler to roll our own rather than attempt to adopt one of them to work for our use case.
go-control-plane seems to make writing an XDS control plane really easy thankfully - it handles running a GRPC server that speaks the XDS protocol with proxies and is backed by a cache which an implementation needs to populate.

The rough workflow would be that we have our code find out what gameserver/upstreamendpoints are available in the cluster and update the cache. Each quilkin proxy will watch resources by contacting go-control-plane's GRPC server which then feeds it data from the cache whenever it is updated.

Currently wondering what a simple workflow would be with e.g Agones + OpenMatch - say a loop that watches GameServers that have been Allocated and populates the cache with their addresses as Endpoints - and it sounds like one difference in this case is that we'll need to ensure that all connected proxies have acknowledged the update containing a GameServer's address before Open Match lets any game client talk to them otherwise a race condition can cause any initial packets from clients to be dropped since the proxies won't know about the new address.
So that this would need some kind of synchronization between the control plane and Open Match? (Say a CRD that will be updated by the control plane server and watched by the director?)

Would this make sense? Is this missing something? Thoughts?

[markmandel]

I'm thinking it makes sense to break things down into the following scenarios, and work things through to make sure there is the most generic applicability.

1. A game server starts
   • This could come from Agones (in some capacity) to tell the proxies "hey, there's a new endpoint"
   • We may wait for Allocation, or we could have the list of Ready Game Servers available? Tradeoffs both ways I feel.
2. A player is match made to a server instance
   • This would need to come from open match
   • This could in theory also tell the control plane about the new endpoint at the same time, and skip the above section.
3. A Game Server is deleted / possibly reset
   • This would need to come from Agones
   • Could be a good reason to make the control plane specific to "Allocated", since a reset to "Ready" would drop it out of the pool again.
4. A player gets booted from a game for bad behaiviour
   • Not sure where this comes from
   • Sounds like something we don't have to manage

I'm trying to think if Agones player tracking helps here, but I don't think it does.

Open Match lets any game client talk to them otherwise a race condition can cause any initial packets from clients to be dropped since the proxies won't know about the new address.

I wonder if this matters? As long as the reconciliation timeframe is a matter of seconds, as long as the initial connection timeout on the game takes that into account, it's probably not a concern?

One other thought - can Open Match use the control plane to be aware of all the proxies? Since it will need to send that information to the game client as well.

[iffyio]

1 + 3 sounds like primarily the control plane needs to keep track of gameserver churn (from its pov say allocated vs not allocated) and send them as add/delete events to the proxies.
I'm not sure what 2 implies - why is a player being assigned to a server interesting to the control plane?

I wonder if this matters? As long as the reconciliation timeframe is a matter of seconds, as long as the initial connection timeout on the game takes that into account, it's probably not a concern?

I think from a game client's pov this would be a bug or annoyance since we only should drop packets due to network or config issues - requiring clients to update their code might not be easy especially if they're doing fire/forget with no connection/retries which sounds likely since its all udp?

One other thought - can Open Match use the control plane to be aware of all the proxies? Since it will need to send that information to the game client as well.

Yeah it sounds like in some cases this information could be needed. Some cases where it might not be:

• I remember @luna-duclos pondered having load balancers in front of the proxies - in this setup OM would likely care about the lb adresses rather than the proxies'

• If the proxies are in the same k8s cluster as the game service that OM talks to, that service can find the info out for itself by e.g watching the relevant proxy resource?

As a fallback, we could probably add a grpc endpoint to the control plane that streams updates when a proxy joins/leaves

[markmandel]

I'm not sure what 2 implies - why is a player being assigned to a server interesting to the control plane?

Because then their authentication token will need to be added to the endpoint, so they have access to that game server.

I think from a game client's pov this would be a bug or annoyance

I agree - but it might be worth waiting and seeing real data before implementing a solution. It may not actually be a big deal. Or it may. Just thinking it's the sort of thing we should probably have some real data on before making decisions.

since we only should drop packets due to network or config issues - requiring clients to update their code might not be easy especially if they're doing fire/forget with no connection/retries which sounds likely since its all udp?

Since you can't assume that a UDP packet ever reaches it's destination - the packets will have to retry by default until they receive their ACK response. I would expect that determining "no response" is more of a "I didn't get a value back in x time" rather than being able to rely on something stable like a TCP direct connection.

I'm just positing that a delay in our control plane is technically no different from a small network blip, so it may well be handled anyway.

[markmandel]

If the proxies are in the same k8s cluster as the game service that OM talks to, that service can find the info out for itself by e.g watching the relevant proxy resource?

That is possible - but seems unlikely. If we're talking about a global game - proxies are distributed around the world. So I think we should assume separate clusters by default.

[iffyio]

That is possible - but seems unlikely. If we're talking about a global game - proxies are distributed around the world. So I think we should assume separate clusters by default.

this would be the case for us at Embark, we have game server clusters spread out in different regions but a game only takes place in exactly one of those so we don't have a need for inter-cluster/region communication.

to clarify what I have in mind for a setup:

• there's a k8s cluster A containing game servers - e.g likely agones runs there
• proxies run in A (at least logically) forwarding traffic to all GSs in A
• there's a control plane for the proxies in A - ideally this runs in A as well

Ideally all of these physically run in A unless there's a reason not to - the point is that the control plane configures only that set of proxies which in turn talk to only that set of game servers.

In a case like ours at Embark, we can still create a cluster (GS, proxies, control plane) in multiple regions and have them independent of each other.

For use cases that do need to be distributed, the same initial setup can still be used: say for example a game client should talk to a proxy in A which should route to a GS in B:

• Control plane in A needs to learn about GSs in B in order to have proxies route to it - we can add a second k8s client to it that watches the GSs in B

• OM for B needs to learn about proxy in A in order to pass it on to the client - add a second k8s client to OM (or the service it talks to when retrieving GS addresses) that watches proxies in A - the point here is that this is done outside of the control plane since its job is to configure the proxies.

It shouldn't matter where OM runs since it shouldn't need to talk to the control plane.

Because then their authentication token will need to be added to the endpoint, so they have access to that game server.

I think this ties closely to how the control plane discovers game server addresses? At the time of allocating a GS, tokens should be known so that the control plane should be able to watch a single resource that contains both token and address (e.g appending the token to the GameServer, Endpoint or some other object) i.e OM provides tokens when requesting a GS and the service it talks to updates the appropriate object with both info - otherwise we'd need to think about races

[markmandel]

I think this ties closely to how the control plane discovers game server addresses? At the time of allocating a GS, tokens should be known so that the control plane should be able to watch a single resource that contains both token and address

I think we have an interesting difference of opinions 💡 . Correct me if I'm wrong though!

I'm assuming each player has their own identifying token. It sounds like you are thinking there is a single token for each game server?

Is that correct? If so, that explains differences in architectural approaches.

[iffyio]

I think we have an interesting difference of opinions bulb . Correct me if I'm wrong though!

I don't think that's the case, I also had in mind multiple tokens for a gameserver as well - my OM knowledge is still minimal currently but my assumption was that at the time OM decides to allocate a GS, it already knows what players to assign to that particular GS (that should be the case right?) and as a result all tokens should be known or generatable at that point - regardless of if its one token per player. e.g when OM requests that a GS is allocated, it also provides any tokens alongside

[markmandel]

I don't think that's the case, I also had in mind multiple tokens for a gameserver as well

Ah awesome, then we are on the same page!

my assumption was that at the time OM decides to allocate a GS, it already knows what players to assign to that particular GS (that should be the case right?)

Aha! Yes this would be true in some games, but definitely not all. Some games will allow you to backfill game servers that are currently active. For example:

• A player drops out early and needs to be replaced
• A persistent world shard that is always looking for new players
• A long running lobby server where people join to be later match made.

More details on Open Match: googleforgames/open-match#1240

So Open Match (or any matchmaker) will need to be part of the process of allowing access to specific game servers.

[iffyio]

Ah, I see! That's good to know I'll take a look at that issue!

[iffyio]

👍 this clarifies why a player being assigned to a server is important to the control plane! Essentially at anytime we should be able to change the set of tokens that are associated with the game server if I understand correctly.

That should work mostly the same with my previous proposal so that the control plane still watches the associated k8s resources containing the tokens&addresses (not necessarily the same resource) and updates the proxies whenever there are changes.
Changes can be applied by whomever has permissions to update those resource objects in the cluster - from the control plane's pov its not interesting if that's OM or the game service or manually so it's up to the user what setup they'd want.

[markmandel]

That should work mostly the same with my previous proposal so that the control plane still watches the associated k8s resources containing the tokens&addresses (not necessarily the same resource)

You are right - if you stored the tokens on the Agones GameServer (maybe also through something the SDK could change as well, if players disconnect?) it could be the central repository for the information?

Combine this with something like googleforgames/agones#1239 -- and or some of the tools in https://agones.dev/site/docs/third-party-content/libraries-tools/ -- that could work out?

I have some worries about some race condition type stuff though, but they can likely be worked through?

One thought - depending on how much performance we get out of the proxy - it is possible that you could server Game Server clusters A, B and C out of a single proxy cluster -- which is where I like the idea of having some kind of message queue system in between?

[iffyio]

Combine this with something like googleforgames/agones#1239 -- and or some of the tools in https://agones.dev/site/docs/third-party-content/libraries-tools/ -- that could work out?

Yeah, implementation wise I would imagine there would be some sort of provider like interface such that the control plane's actual logic is separate from the source of events - e.g polling k8s api vs some other endpoint. Then adding a new source can be a matter a struct that can poll that source for info and convert it to a set of addresses and tokens that the control plane consumes

I have some worries about some race condition type stuff though, but they can likely be worked through?

😮 which race conditions?

One thought - depending on how much performance we get out of the proxy - it is possible that you could server Game Server clusters A, B and C out of a single proxy cluster -- which is where I like the idea of having some kind of message queue system in between?

I would expect the proxies not to be an issue perf wise but rather latency if the clusters are far enough from them, would be interesting to find out for sure. What did you have in mind re message queue, what's sent on the queues?

[markmandel]

Yeah, implementation wise I would imagine there would be some sort of provider like interface such that the control plane's actual logic is separate from the source of events - e.g polling k8s api vs some other endpoint. Then adding a new source can be a matter a struct that can poll that source for info and convert it to a set of addresses and tokens that the control plane consumes

Sounds like a message queue of some kind would be good. https://github.com/Octops/agones-event-broadcaster may be a good fit.

😮 which race conditions?

For example:

If a game server binary tracks that a client has disconnected / kicked out of a game - it will want to remove the access token from itself, and have that propagate out to the proxy. One way it could do that is through an annotation it edits through the Agones SDK. At the same time a re-allocation happens that adds several new players and updates the same annotation. The SDK may still have the old list rather than the new one, and overwrite the new player connection data. In the SDK, there's no concept of increment/make delta change -- but maybe that's something we can add to Agones itself. If we had a "delta change" operation on the SDK for labels and annotations, Kubernetes generational resource locking would save us here (although, not 100% sure how that would work, but we can probably some up with something 🤔 ).

I would expect the proxies not to be an issue perf wise but rather latency if the clusters are far enough from them, would be interesting to find out for sure.

From a GCP perspective - I could have 3 clusters running in the same GCP region/zone, depending on the size of my game. So latency shouldn't be an issue at that point.

What did you have in mind re message queue, what's sent on the queues?

So the project I mentioned sends out GameServer changes over pubsub (or other message queues) - so our xDS service could subscribe to that, and translate that into our xDS format of choice, to be send out to all proxies.

[iffyio]

If a game server binary tracks that a client has disconnected / kicked out of a game - it will want to remove the access token from itself, and have that propagate out to the proxy. One way it could do that is through an annotation it edits through the Agones SDK. At the same time a re-allocation happens that adds several new players and updates the same annotation. The SDK may still have the old list rather than the new one, and overwrite the new player connection data. In the SDK, there's no concept of increment/make delta change -- but maybe that's something we can add to Agones itself. If we had a "delta change" operation on the SDK for labels and annotations, Kubernetes generational resource locking would save us here (although, not 100% sure how that would work, but we can probably some up with something thinking ).

k8s api's have support for optimistic locking to avoid these types of issues, there's the resourceVersion on an object that changes on each write and k8s rejects patches with an old resourceVersion (can't remember if it was an opt-in or default behavior) so that upon a reject the writer re-reads the object to get the latest version and can retry and update based on that - that should avoid the described issue? e.g the SDK would need to ensure its is writing safely by checking for http 409 responses

So the project I mentioned sends out GameServer changes over pubsub (or other message queues) - so our xDS service could subscribe to that, and translate that into our xDS format of choice, to be send out to all proxies.

Ah yes, this would be possible with the provider interface.

[markmandel]

e.g the SDK would need to ensure its is writing safely by checking for http 409 responses

You are correct about the locking (it is the default, and you can somewhat opt out of it with patch statements, but even then sometimes they fail with a Conflict error.)

It's tricky if you are doing some kind of increment operation - i.e. if you had an annotation of:
"agones.dev/sdk-tokens": "abc,def,hij,klm" and you wanted to add "nop" to the list - you would have to write specific code to manage that delta, so that if "def" was removed before you got there, you re processed the list, and added your specific delta.

The way the SDK SetAnnotation() works, it doesn't handle that delta - it just does straight label value swaps (this is getting into very Agones specific stuff).

One way you could handle this though - is make every connection token unique as an annotation! Then Agones doesn't have to worry about deltas!

So you could do something like:

agones.dev/sdk-token/abc: true
agones.dev/sdk-token/def: true
agones.dev/sdk-token/hij: true

Then to add an extra token of nop you could do SDK.SetAnnotation("token/nop", "true") and there's no race condition.
To remove hij for example, you would set it to false SDK.SetAnnotation("token/hij", "false")

Some extra thoughts:

• labels and annotations can't have a key longer than 63 characters (from memory). I expect tokens to be relatively short, so this likely won't be an issue?
• No matter the approach, this does put extra load on the K8s master api.

[iffyio]

I didn't catch the problem from the example - if the first attempt fails then the writer could call SetAnnotation("abc,hij,kml,nop") the second time around which wouldn't be a delta?

[markmandel]

The way `SDK.SetAnnotation() works, it can only do complete value swaps for labels and annotations - so it's a case of last-one-in-wins - it has no concept of list deltas.

So if while it tries and sets an annotation value of SetAnnotation("abc,hij,kml,nop") because when the game logic looked at the GameServer state it was "abc,hij,kml", and we're adding "nop" - that's fine, as long as no other part of the system adds or removes values from that list.

So either - Agones would need to add the concept of list deltas to it's SDK for Annotations (and is that a good idea, maybe? What format, etc? Maybe you can provide JSON patches to json values stored in annotations?), or you would need to do individual unique keys for the appropriate annotation.

[iffyio]

If I understand correctly, the issue is more of an API limitation on the SDK that currently SetAnnotation doesn't provide a way to retry if it got a conflict response? Also, I'm thinking unique keys for each token would be more awkward to handle vs a single annotation, and as you mentioned it effectively limits the max size of a token

[markmandel]

If I understand correctly, the issue is more of an API limitation on the SDK that currently SetAnnotation doesn't provide a way to retry if it got a conflict response?

It does retry, but it's only ever going to retry with the initial value that was passed.

[iffyio]

I see, can we add support for this to the sdk? e.g a new api where the caller passes in a function to recompute a new value on retry

[markmandel]

I see, can we add support for this to the sdk? e.g a new api where the caller passes in a function to recompute a new value on retry

That's an interesting question. The SDK is a wrapper around a gRPC client, and the gRPC servers talks to K8s -- so it'll be tricky to have a callback here, although not impossible.