xDS: Locality leak on GameServer event handling

[markmandel]

What happened:

I'm pretty sure there's a leak in localities whenever a GameServer is added or deleted.

To watch this, I ran kubectl port-forward -n 1662074311 quilkin-proxies-8687b95f58-s89tq 9091 to view the http://localhost:9091/config to see changes.

Setup: I have a quilkin manage agones xDS control plane, and 1 quilkin run instance connected to it.

I created 3 GameServers, and allocated one of them. The /config json looked as following:

{
    "admin": {
        "address": "0.0.0.0:9091"
    },
    "clusters": {
        "default": {
            "localities": [
                {
                    "locality": null,
                    "endpoints": []
                },
                {
                    "locality": null,
                    "endpoints": [
                        {
                            "address": "34.83.160.299:7135",
                            "metadata": {
                                "quilkin.dev": {
                                    "tokens": []
                                }
                            }
                        }
                    ]
                }
            ]
        }
    },
    "filters": [],
    "management_servers": [
        {
            "address": "http://quilkin-manage-agones:80"
        }
    ],
    "proxy": null,
    "version": "v1alpha1"
}

I then allocated a second GameServer, and /config became this:

{
    "admin": {
        "address": "0.0.0.0:9091"
    },
    "clusters": {
        "default": {
            "localities": [
                {
                    "locality": null,
                    "endpoints": []
                },
                {
                    "locality": null,
                    "endpoints": [
                        {
                            "address": "34.83.160.299:7135",
                            "metadata": {
                                "quilkin.dev": {
                                    "tokens": []
                                }
                            }
                        }
                    ]
                },
                {
                    "locality": null,
                    "endpoints": [
                        {
                            "address": "34.83.160.299:7056",
                            "metadata": {
                                "quilkin.dev": {
                                    "tokens": []
                                }
                            }
                        }
                    ]
                }
            ]
        }
    },
    "filters": [],
    "management_servers": [
        {
            "address": "http://quilkin-manage-agones:80"
        }
    ],
    "proxy": null,
    "version": "v1alpha1"
}

I expected a single locality, since everything is a single cluster.

I then deleted one of the allocated GameServers, and I got this:

{
    "admin": {
        "address": "0.0.0.0:9091"
    },
    "clusters": {
        "default": {
            "localities": [
                {
                    "locality": null,
                    "endpoints": []
                },
                {
                    "locality": null,
                    "endpoints": [
                        {
                            "address": "34.83.160.299:7135",
                            "metadata": {
                                "quilkin.dev": {
                                    "tokens": []
                                }
                            }
                        }
                    ]
                },
                {
                    "locality": null,
                    "endpoints": []
                },
                {
                    "locality": null,
                    "endpoints": []
                }
            ]
        }
    },
    "filters": [],
    "management_servers": [
        {
            "address": "http://quilkin-manage-agones:80"
        }
    ],
    "proxy": null,
    "version": "v1alpha1"
}

Which looks to me like a leak of empty localities?

Also very strange

What you expected to happen:

• I expected that there would be a single locality in which endpoints where added and deleted, since all endpoints are within the same cluster.
• Localities wouldn't be left behind when a GameServer was deleted.

How to reproduce it (as minimally and precisely as possible):

• Create a Fleet with 3 GameServers
• Allocate one of them.
• Allocate another one
• Delete a GameServer

Have a look at /config on the proxy to see what the current config is.

Anything else we need to know?:

Environment:

• Quilkin version: dev
• Execution environment (binary, container, etc): container
• Operating system: Linux
• Custom filters? (Yes/No - if so, what do they do?): No
• Log(s): No errors - looks good!
• Others: N/A

[XAMPPRocky]

Interesting, definitely a bug.

[markmandel]

Not sure if related, but just had a weird occurrence where I deleted an allocated GameServer, and it got added a second time.

➜  agones-xonotic-xds git:(examples/xonotic-xds) ✗ curl -s http://localhost:9091/config | jq
{
  "admin": {
    "address": "0.0.0.0:9091"
  },
  "clusters": {
    "default": {
      "localities": [
        {
          "locality": null,
          "endpoints": []
        },
        {
          "locality": null,
          "endpoints": [
            {
              "address": "34.127.72.131:7118",
              "metadata": {
                "quilkin.dev": {
                  "tokens": []
                }
              }
            }
          ]
        },
        {
          "locality": null,
          "endpoints": [
            {
              "address": "34.127.72.131:7118",
              "metadata": {
                "quilkin.dev": {
                  "tokens": []
                }
              }
            }
          ]
        }
      ]
    }
  },
  "filters": [
    {
      "name": "quilkin.filters.capture.v1alpha1.Capture",
      "config": {
        "metadataKey": "quilkin.dev/capture",
        "suffix": {
          "size": 3,
          "remove": true
        }
      }
    },
    {
      "name": "quilkin.filters.token_router.v1alpha1.TokenRouter",
      "config": null
    }
  ],
  "management_servers": [
    {
      "address": "http://quilkin-manage-agones:80"
    }
  ],
  "proxy": null,
  "version": "v1alpha1"
}

Rather odd. Just noting it for whenever someone digs in here.