Upgrade to Kubernetes 1.17

[roberthbailey]

We aren't ready for this yet, but now that we are wrapping up the upgrade to 1.16 we have some steps we want to proactively capture for 1.17 so it's time to start writing them down.

List of items to do for upgrading to 1.17 (this was started from the list in the 1.16 issue and likley need further updates):

• Update e2e cluster to run against 1.17
  • Recreate cluster with new scripts (after updating the GKE terraform module)
  • Update kubectl in e2e-image/Dockerfile
• Update prow cluster to use 1.17 (even though we aren't using it yet, we should keep it in sync)
  • Recreate cluster with new scripts
• Update the dev tooling to create 1.17 clusters
  • GKE
  • Minikube
  • Kind
  • Update kubectl
• Update terraform submodules
  • GKE
  • Azure
  • EKS
• Update documentation for creating clusters to 1.17
  • Usage requirements
  • GKE
  • Minikube
  • EKS
  • AKS
• Update links to k8s documentation
  • examples/fleet.yaml
  • examples/fleetautoscaler.yaml
  • examples/gameserver.yaml
  • site/content/en/docs/Reference/fleet.md
  • site/content/en/docs/Reference/fleetautoscaler.md
  • site/content/en/docs/Reference/gameserver.md
  • site/content/en/docs/Advanced/limiting-resources.md
  • site/assets/templates/crd-doc-config.json
• Update client-go
• Regenerate CRD Kubernetes client libraries
• Move API references from beta to v1 (see the Kubernetes v1.16 Release Notes)
  • CRDs
  • Admission Webhooks

[markmandel]

We should target this at our 1.11.0 release - 1.17 is available across all three cloud providers.

[markmandel]

Wondering if we want to add running a long running version of #1867 to ensure there are no memory leaks in client-go when we upgrade?

This is the second time (#1871) we've been bitten by a memory leak in client-go.

[roberthbailey]

#1799 is captured as a checkbox in the list above but also in a separate issue.

[markmandel]

Current status: Moving the CRDs to v1.

[markmandel]

Going to tackle upgrading go-client - it likely would need to happen after we get some more CRD v1 stuff in, but I expect we can manage the merge, so I figured I can get the work running in parallel and handle the merging.

[markmandel]

Ran an overnight allocation load test.

TESTRUNSCOUNT=30 ./runAllocation.sh 40 100

I can see a minor increase in memory over time, but looking at the heap graph, I don't see any egregious memory usage, so I'm expecting that it is likely normal memory usage, that will plateau over time.

No CPU leaks.

Heap Graph:

Usually when we have a memory leak, it's extremely obvious because it takes up a majority of the memory space.

[markmandel]

All items have now been checked! Happy to close this issue if everyone else is!

[markmandel]

Running some additional allocations over the past few hours - and yeah, looks like it's plateau's around the 200-300 MB mark (on alloc, light blue), so what we're seeing it just a gradual increase to that point. 😅

[markmandel]

Poking at this some more because I wasn't 100% happy - and it seems like it's okay, as it keep coming down to that baseline.

(This is the past 23 hours, which includes 2 load tests)

Nothing in the heap aloc graph seems to stands out;

I do see:

32MB 14.32%                | k8s.io/client-go/tools/cache.MetaNamespaceKeyFunc /go/src/agones.dev/agones/vendor/k8s.io/client-go/tools/cache/store.go:85

Which was smaller before, but that (a) could be coincidence and (b) there's nothing in that function that actually stores anything, it's all transient String creation.

Maybe I'm just being overtly worrisome. 🤷

[markmandel]

I'm going to say let's close this issue, since RC is tomorrow. If it looks like something has gone wrong during RC week, we can reopen a new issue. Sound good?

[markmandel]

Closing, since no objections!