Player Tracking for each GameServer

[markmandel]

Is your feature request related to a problem? Please describe.
It would be very useful to track player connection and disconnection against a specific GameServer.

This would allow us to do several things:

• Track if a GameServer is full or not (we should also track GameServer capacity as well)
• Be able to trace and debug a player lifecycle from matchmaker to gameserver connection (we should track an id or token on connection)
• Autoscale Fleets based on how full gameservers are (useful for persistent worlds)

Describe the solution you'd like
(Full design TBD)

• Need a way to specify the capacity of a GameServer, at creation time, but also editable from the SDK at runtime
• SDK methods to track a player connection, and player disconnection with an id/player token
• GameServer CRD Events for player connections and disconnections
• GameServer CRD Status values that track player counts and capacity. Maybe should also have a list of current connected players?
• May want to add some labels to allow for searching for full/empty/partially full game servers through the k8s api

Describe alternatives you've considered
Have a separate system for player tracking - but based on feedback, this is a feature that almost all users should find useful, so it feels like part of Agones, and also allows some useful functionality down the road -- such as autoscaling by player count, or automating backfill operations by searching for non-full game servers.

Additional context
I do have concerns that we are adding more API QPS with this, so we should track performance with these changes.

[cyriltovena]

I also do have a concern for etcd, it could potentially be a lot depending on the design.

A Kubernetes event each time a player join ?

[markmandel]

Yeah agreed - was chatting with some people the other day, talking through this,

The basic idea we ended up thinking, which I think would work better long term (there are some implementation design decisions to work out too) are:

• Only keep the player count and capacity on the GameServer CRD status value.
• The SDK could be smart, and only sync the player count once every second (or configurable interval?), to avoid overloading the system when huge numbers come in. Backfill based on player counts will have race conditions anyway, so games will need to deal with this regardless.
• The an event might just be the update to the player count, not each player, on the same interval as above.

To track players - have a CRD configurable webhook that has player data sent to it on connection / disconnection. Something like the gameserver name, connect/disconnect and the token that is passed through. Then we can build a separate system(s) for storing and tracking connections, and/or respond to events as needed as well.

In theory the sidecar would need to send the webhook request (I think),, but that's doable (and an implementation detail).

This requires a full write up, but I think it sounds like a reasonable approach at first pass. WDYT?

[markmandel]

Design ideas

The following is implementation design for both the configuration, status and game server SDK.

Feedback is much appreciated.

Configuration & Status

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  generateName: "simple-udp-"
spec:
  # new configuration
  alpha:
    players:
      initialCapacity: 10 # sets the initial player capacity. Defaults to 0.
      webhook: # http(s) webhook to send player dis/connection events
        service:
          name: player-tracking-service
          namespace: default
          path: player
  ports:
  - name: default
    portPolicy: Dynamic
    containerPort: 7654
  template:
    spec:
      containers:
      - name: simple-udp
        image: gcr.io/agones-images/udp-server:0.17
status:
  alpha:
    # tracks players
    players:
      count: 6 # current number of players. Only updated one per second, so eventually consistent.
      capacity: 10 # current capacity. Set by "initialCapacity" and can be changed by the SDK

• spec.alpha.players.initialCapacity - the initial player capacity as listed in the status value status.players.capacity
• spec.alpha.players.webhook - a webhook configuration that is called whenever a player connects or disconnects. This sort of data seems ill-suited to CRDs, so Agones never stores these values, it’s the responsibility of another system to track this information as required.
  The webhook POSTs a JSON payload with the following details:
  • GameServer name
  • PlayerID (provided through the SDK)
  • Event: “connect” or “disconnect”
• status.alpha.players.count - The current number of players in the system. This information will be eventually consistent as it is only updated once per second to avoid overloading the system. Since player count race conditions are inevitable, there should always be extra checking of player counts on player connection to a game server.
• status.alpha.players.capacity - The current player capacity of this gameserver. Initially set by the initialCapacity value, but also able to be changed at runtime by the Game Server SDK.

SDK Functionality

SDK.Alpha.PlayerConnect(playerID)

This increases the SDK’s stored player count by one, and passes the playerID to the system such that it can fire the webhook with a connection event to any configured receivers.

status.alpha.players.count is then set to update to the current player count a second from now, unless there is already an update pending.

Will throw an error if the player count is equal to or greater than the capacity.

SDK.Alpha.PlayerDisconnect(playerID) : bool

Decreases the SDK’s stored player count by one, and passes the playerID to the system such that it can fire the webhook with a disconnection event to any configured receivers.

status.alpha.players.count is then set to update to the current player count a second from now, unless there is already an update pending.

Does nothing, and returns false if the playerID was not previously added. Returns true otherwise.

SDK.Alpha.SetPlayerCapacity(int capacity)

Update the status.alpha.capacity value with a new capacity.

SDK.Alpha.GetPlayerCapacity() : int

Retrieves the current capacity. This is always accurate, even if the value hasn’t been updated to the GameServer status yet.

SDK.Alpha.GetPlayerCount() : int

Retrieves the current player count. This is always accurate, even if the value hasn’t been updated to the GameServer status yet.

[aLekSer]

I have some questions regarding PlayerConnect:

SDK.Alpha.PlayerConnect(playerID)
This increases the SDK’s stored player count by one, and passes the playerID to the system such that it can fire the webhook with a connection event to any configured receivers.

Should we wait for webhook response that this player is able to connect? Who would be judging the player if he is able or not able to connect to the session?

Would playerID be stored on GS SDK side or separately?

Then we can build a separate system(s) for storing and tracking connections, and/or respond to events as needed as well.

[cloudrobx]

Hi Mark - Could this metric be utilized for autoscaling? I'm wondering for the relay server use case, in which multiple relay server processes would run inside a single GameServer container and we need a test for "fullness" for autoscaling up and down.

…

--Rob

On Wed, Jan 22, 2020 at 9:48 AM Alexander Apalikov ***@***.***> wrote: I have some questions regarding PlayerConnect: SDK.Alpha.PlayerConnect(playerID) This increases the SDK’s stored player count by one, and passes the playerID to the system such that it can fire the webhook with a connection event to any configured receivers. Should we wait for webhook response that this player is able to connect? Who would be judging the player if he is able or not able to connect to the session? Would playerID be stored on GS SDK side or separately? Then we can build a separate system(s) for storing and tracking connections, and/or respond to events as needed as well. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1033?email_source=notifications&email_token=AAQ3CQNVZWLHOLJX6CIFJC3Q7BMCFA5CNFSM4ITUPHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJTZ5HI#issuecomment-577216157>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3CQJ6KFG5VBITVL5CFWLQ7BMCFANCNFSM4ITUPHOA> .

-- * • **Robert Martin* * • *Chief Architect *• *Google Cloud for Games • [email protected] • m: 470-449-2926

[markmandel]

@aLekSer good questions!

Should we wait for webhook response that this player is able to connect?

Personally, I don't think so. This could potentially make things very slow in the game server binary. I feel like this should be an async operation that happens behind the scenes.

Who would be judging the player if he is able or not able to connect to the session?

This is up to the game server binary to make this decision. So it's game / transport layer specific.

Would playerID be stored on GS SDK side or separately?

In this design, we don't store the playerID at all (except transiently in memory as it gets passed to the system that sends the webhook request). playerID management is up to the game author to track and manage. This is the main impetus for the webhook, to make this easier.

While it could be useful to store playerIDs in a CRD, it seems like a performance concern for etcd. Maybe something to explore down the line if we find we have bandwidth in etcd performance.

@cloudrobx

Could this metric be utilized for autoscaling? I'm wondering for the relay
server use case, in which multiple relay server processes would run inside
a single GameServer container and we need a test for "fullness" for
autoscaling up and down.

Like this? #1034 😄 would that work for what you were thinking?

[cloudrobx]

Yeah something like that looks pretty good. For placement purposes is there an easy way to pull the most (or least) full game server by player count, other than iterating through all the game servers in the cluster?

…

On Wed, Jan 22, 2020 at 12:37 PM Mark Mandel ***@***.***> wrote: @aLekSer <https://github.com/aLekSer> good questions! Should we wait for webhook response that this player is able to connect? Personally, I don't think so. This could potentially make things very slow in the game server binary. I feel like this should be an async operation that happens behind the scenes. Who would be judging the player if he is able or not able to connect to the session? This is up to the game server binary to make this decision. So it's game / transport layer specific. Would playerID be stored on GS SDK side or separately? In this design, we don't store the playerID at all (except transiently in memory as it gets passed to the system that sends the webhook request). playerID management is up to the game author to track and manage. This is the main impetus for the webhook, to make this easier. While it could be useful to store playerIDs in a CRD, it seems like a performance concern for etcd. Maybe something to explore down the line if we find we have bandwidth in etcd performance. @cloudrobx <https://github.com/cloudrobx> Could this metric be utilized for autoscaling? I'm wondering for the relay server use case, in which multiple relay server processes would run inside a single GameServer container and we need a test for "fullness" for autoscaling up and down. Like this? #1034 <#1034> 😄 would that work for what you were thinking? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1033?email_source=notifications&email_token=AAQ3CQJF3ULGCG7PHJILVV3Q7B735A5CNFSM4ITUPHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJUOOIY#issuecomment-577300259>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3CQIRGINELI5QQFPG77DQ7B735ANCNFSM4ITUPHOA> .

-- * • **Robert Martin* * • *Chief Architect * • *Google Cloud for Games • [email protected] • m: 470-449-2926

[markmandel]

Yeah something like that looks pretty good. For placement purposes is
there an easy way to pull the most (or least) full game server by player
count, other than iterating through all the game servers in the cluster?

See: #1239 for current initial thoughts.

[cloudrobx]

For relays / high density would probably also want the ability to find the most empty or most full server as well. Since you're essentially adding another layer of bin-packing.

…

On Thu, Jan 23, 2020 at 7:59 PM Mark Mandel ***@***.***> wrote: Yeah something like that looks pretty good. For placement purposes is there an easy way to pull the most (or least) full game server by player count, other than iterating through all the game servers in the cluster? See: #1239 <#1239> for current initial thoughts. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1033?email_source=notifications&email_token=AAQ3CQPRB6P7554DM3VDJ7DQ7I4ONA5CNFSM4ITUPHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZMURI#issuecomment-577948229>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3CQOJ6WZGSWJX5AQY3NTQ7I4ONANCNFSM4ITUPHOA> .

-- * • **Robert Martin* * • *Chief Architect * • *Google Cloud for Games • [email protected] • m: 470-449-2926

[markmandel]

For relays / high density would probably also want the ability to find the
most empty or most full server as well. Since you're essentially adding
another layer of bin-packing.

This sounds like the current "Distributed" vs "Packed" strategies we already have in place. Also probably a topic to discuss more on #1239 rather than here, since this ticket isn't covering allocation - just player count tracking.

[markmandel]

Made a small tweak to SDK. Renamed GetCapacity and SetCapacity to GetPlayerCapacity and SetPlayerCapacity - just to be clearer.

[Reousa]

Hey Mark! Thanks for the heads up, would be great to be able to contribute again! :)

Gave the issue a read, I see these have already been updated to the relevant proto files, so all that's needed is the relevant language implementation, yeah?

If my understanding is correct, consider it done (hopefully in a timely manner)! 👌

[markmandel]

That's a good point - you will need to edit the gen.sh scripts such that they generate the code for the alpha.proto, and then yes -- implement the SDK over the top 👍

Thanks!

[steven-supersolid]

Working on Node.js now

[markmandel]

I'm going to suggest we close this ticket, since this has been released to alpha, and there is (mostly?) supported in the SDKs - and I think we can then track each of those as seperate tickets as needed.

Also #1677 is going to break all the things (will likely aim to tackle in the next release), so we can track the ongoing work there as well.

Sound good?

[roberthbailey]

Closing as per the last comment (1.5 years ago).