Upgrade to Kubernetes 1.16 #1649

roberthbailey · 2020-06-25T16:43:04Z

roberthbailey · 2020-06-25T16:57:53Z

The Kubernetes v1.16 Release Notes have some points that we should keep in mind for this upgrade:

The 1.16 release marks the graduation of CRDs to general availability (GA).
- The apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition is deprecated and will no longer be served in v1.19. Use apiextensions.k8s.io/v1 instead.
- There are some changes to the CustomResourceDefinition API type as it was promoted to apiextensions.k8s.io/v1
The 1.16 release marks the graduation of admission webhooks to general availability (GA).
- The admissionregistration.k8s.io/v1beta1 versions of MutatingWebhookConfiguration and ValidatingWebhookConfiguration are deprecated and will no longer be served in v1.19. Use admissionregistration.k8s.io/v1 instead.
Many beta APIs are no longer served: use apps/v1, networking.k8s.io/v1, policy/v1beta1, networking.k8s.io/v1beta1, scheduling.k8s.io/v1
Aggregated discovery requests can now timeout. Aggregated API servers must complete discovery calls within 5 seconds (other requests can take longer).
Server-side apply feature is now beta
- Server-side apply will now use the openapi provided in the CRD validation field to help figure out how to correctly merge objects and update ownership.
The CustomResourceDefaulting feature is promoted to beta and enabled by default. Defaults may be specified in structural schemas via the apiextensions.k8s.io/v1 API.

roberthbailey · 2020-08-13T21:44:41Z

We should switch to 1.16 after this next release is cut, as Azure is about to put 1.18 into GA (estimated date is Aug 20th) at which point they will drop official support for 1.15.

markmandel · 2020-08-25T04:35:43Z

Working up updating the dev gke and e2e clusters to 1.16 -- which is now powered by the GKE terraform module, so taking care of several tasks in one go.

thisisnotapril · 2020-08-27T22:19:26Z

Just heads-up that the final 1.16 patch is scheduled for 9/2

roberthbailey · 2020-08-27T22:54:19Z

Ugh. Always on the tail end.....

That being said, GKE's release notes for today just remove support for new clusters at 1.14 and are upgrading 1.14 -> 1.15 in the stable channel while the regular channel is being moved from 1.15 -> 1.16. So we are right in line with where GKE is at in terms of k8s version support.

aLekSer · 2020-09-03T08:00:48Z

I would add a PR for AKS and EKS soon.

aLekSer · 2020-09-03T08:11:58Z

Adding some notes from the Agones Community Meeting, mentioned by @roberthbailey :

1.16 has a few api changes to be aware of - things graduating from beta -> ga, older beta apis being dropped
We reference beta APIs in our go code: https://github.com/googleforgames/agones/search?l=Go&q=v1beta1
A good first step would be to move the resources in the Helm chart (like CRDs) from betav1 to v1
E.g. Many beta APIs are no longer served: use apps/v1, networking.k8s.io/v1, policy/v1beta1, networking.k8s.io/v1beta1, scheduling.k8s.io/v1 - we should explore and see if we have any.

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#deprecations-and-removals

The apiextensions.k8s.io/v1beta1 version of CustomResourceDefinition is deprecated and will no longer be served in v1.19. Use apiextensions.k8s.io/v1 instead.

roberthbailey · 2020-09-03T15:25:38Z

Most of this is in this issue above.

The important things to catch when upgrading to 1.16 are to make sure that we aren't relying on APIs that are no longer being served (like scheduling.k8s.io/v1beta1).

For APIs that graduated to v1 in 1.16, if we immediately adopt the v1 API then we will prevent Agones from being installed on older versions of k8s. I'd suggest that we wait to take advantage of the newly promoted APIs until 1.17 (or even 1.18) to give folks a longer window to upgrade (even though as a project we won't be testing those versions any longer).

markmandel · 2020-09-04T15:28:38Z

Working on upgrading client-go 👍

Worth noting that cache.WaitForCacheSync has changed its internal implementation to now use `err := wait.PollImmediateUntil(...)`, so there is no more implicit 100ms sleep before the sync. Therefore it didn't _actually_ wait to populate from Watch or allow events to fire - it just made room for it to occur. So had to now using assert/require.Eventually() to make sure that the system is in a state that matches what we expect before testing. Work on googleforgames#1649

* Update k8s.io/client-go to v0.16.15 * Upgrade k8s.io/apiextensions-apiserver to v0.16.15 Note: overwrote grpc to keep at current version. * Upgrade client-go and apimachinery to 0.16.5 Worth noting that cache.WaitForCacheSync has changed its internal implementation to now use `err := wait.PollImmediateUntil(...)`, so there is no more implicit 100ms sleep before the sync. Therefore it didn't _actually_ wait to populate from Watch or allow events to fire - it just made room for it to occur. So had to now using assert/require.Eventually() to make sure that the system is in a state that matches what we expect before testing. Work on #1649 Co-authored-by: Alexander Apalikov <[email protected]>

Both the build and the e2e image. Work on googleforgames#1649

markmandel · 2020-09-11T22:06:32Z

Adding to this list:

Regenerate CRD Kubernetes client libraries

Both the build and the e2e image. Work on #1649

Only a small change Work on googleforgames#1649

Only a small change Work on #1649 Co-authored-by: Robert Bailey <[email protected]>

markmandel · 2020-09-15T00:49:38Z

A reminder - RC is next week, so probably makes sense to prioritise work for this ticket over others, so we can get support for 1.16 out the door.

markmandel · 2020-09-15T00:56:47Z

Did a review - looks like it's just the EKS/AKS stuff that needs updating. @aLekSer are you on that?

And to confirm:

Move API references from beta to GA. For instance, scheduling.k8s.io/v1beta1 to scheduling.k8s.io/v1. See https://github.com/googleforgames/agones/search?l=Go&q=v1beta1

We're waiting on 1.17 to do this (@roberthbailey ) ?

roberthbailey · 2020-09-15T04:33:58Z

For things that just graduated to GA, we are waiting until 1.17 to migrate our configs (see #1799).

For things where beta was dropped in 1.16 (like scheduling.k8s.io/v1beta1) we need to move now. But I did a search and only found one instance which was updated already in #1714 so that task is complete.

aLekSer · 2020-09-15T11:12:20Z

AKS is ready, regarding EKS I have a draft PR working on the fix.

markmandel · 2020-09-15T16:10:42Z

For things where beta was dropped in 1.16 (like scheduling.k8s.io/v1beta1) we need to move now. But I did a search and only found one instance which was updated already in #1714 so that task is complete.

Sweet. Ticked the box.

aLekSer · 2020-09-16T21:41:08Z

All ticks are in place.
Just want to add that we need to add check Metrics checkbox as some names might be changed, there is a section about metrics:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#metrics-changes
I look through our metrics, definitely that's not the case right now.

roberthbailey · 2020-09-17T07:46:44Z

Thanks @aLekSer. I've created #1824 for the 1.17 upgrade so that we can start capturing any additional steps there. Please feel free to edit that issue to add any steps that you know are missing.

markmandel · 2020-09-17T15:44:19Z

Awesome work. Any objections to closing this issue then?

roberthbailey · 2020-09-17T15:58:17Z

I think we are all set.

markmandel · 2020-09-17T16:16:31Z

CLOSIING!

* Update k8s.io/client-go to v0.16.15 * Upgrade k8s.io/apiextensions-apiserver to v0.16.15 Note: overwrote grpc to keep at current version. * Upgrade client-go and apimachinery to 0.16.5 Worth noting that cache.WaitForCacheSync has changed its internal implementation to now use `err := wait.PollImmediateUntil(...)`, so there is no more implicit 100ms sleep before the sync. Therefore it didn't _actually_ wait to populate from Watch or allow events to fire - it just made room for it to occur. So had to now using assert/require.Eventually() to make sure that the system is in a state that matches what we expect before testing. Work on googleforgames#1649 Co-authored-by: Alexander Apalikov <[email protected]>

Both the build and the e2e image. Work on googleforgames#1649

Only a small change Work on googleforgames#1649 Co-authored-by: Robert Bailey <[email protected]>

roberthbailey added the kind/feature New features for Agones label Jun 25, 2020

aLekSer mentioned this issue Jul 8, 2020

k8s.io/client-go v0.18.0 #1682

Closed

markmandel pinned this issue Jul 23, 2020

roberthbailey mentioned this issue Jul 23, 2020

Use the scheduling.k8s.io/v1 API since the beta API will no longer be served by default starting with Kubernetes 1.16 #1714

Merged

markmandel mentioned this issue Aug 25, 2020

Update GKE Terraform clusters to 1.16 #1772

Merged

This was referenced Sep 3, 2020

Docs: updating Minikube kubernetes version to 1.16 #1786

Merged

Examples: update Kubernetes version #1787

Merged

This was referenced Sep 4, 2020

Docs: updated advised version of Kubernetes to use #1789

Merged

Docs and TF: Update EKS Kubernetes version to use 1.16 #1790

Merged

Docs: updating GKE and AKS Kubernetes versions #1791

Merged

markmandel mentioned this issue Sep 5, 2020

Upgrade client-go and apimachinery to 0.16.15 #1794

Merged

markmandel added a commit to markmandel/agones that referenced this issue Sep 11, 2020

Upgrade Kubectl to 1.16.15

1a10473

Both the build and the e2e image. Work on googleforgames#1649

markmandel mentioned this issue Sep 11, 2020

Upgrade Kubectl to 1.16.15 #1806

Merged

aLekSer pushed a commit that referenced this issue Sep 11, 2020

Upgrade Kubectl to 1.16.15 (#1806)

fe14aea

Both the build and the e2e image. Work on #1649

markmandel added a commit to markmandel/agones that referenced this issue Sep 11, 2020

Rerun Agones CRD client generation

b63b5f7

Only a small change Work on googleforgames#1649

markmandel mentioned this issue Sep 11, 2020

Rerun Agones CRD client generation #1807

Merged

markmandel added a commit that referenced this issue Sep 14, 2020

Rerun Agones CRD client generation (#1807)

d7dd795

Only a small change Work on #1649 Co-authored-by: Robert Bailey <[email protected]>

aLekSer added the area/operations Installation, updating, metrics etc label Sep 15, 2020

This was referenced Sep 15, 2020

Website: Update properly to Kubernetes 1.16 #1813

Merged

Website: Update API docs to Kubernetes 1.16 #1814

Merged

roberthbailey mentioned this issue Sep 17, 2020

Upgrade to Kubernetes 1.17 #1824

Closed

34 tasks

markmandel closed this as completed Sep 17, 2020

markmandel added this to the 1.9.0 milestone Sep 17, 2020

aLekSer unpinned this issue Sep 18, 2020

markmandel added the kind/breaking Breaking change label Sep 22, 2020

ilkercelikyilmaz pushed a commit to ilkercelikyilmaz/agones that referenced this issue Oct 23, 2020

Upgrade Kubectl to 1.16.15 (googleforgames#1806)

551be6c

Both the build and the e2e image. Work on googleforgames#1649

ilkercelikyilmaz pushed a commit to ilkercelikyilmaz/agones that referenced this issue Oct 23, 2020

Rerun Agones CRD client generation (googleforgames#1807)

9067c36

Only a small change Work on googleforgames#1649 Co-authored-by: Robert Bailey <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Kubernetes 1.16 #1649

Upgrade to Kubernetes 1.16 #1649

roberthbailey commented Jun 25, 2020 •

edited by aLekSer

Loading

roberthbailey commented Jun 25, 2020

roberthbailey commented Aug 13, 2020

markmandel commented Aug 25, 2020 •

edited

Loading

thisisnotapril commented Aug 27, 2020

roberthbailey commented Aug 27, 2020

aLekSer commented Sep 3, 2020

aLekSer commented Sep 3, 2020 •

edited

Loading

roberthbailey commented Sep 3, 2020

markmandel commented Sep 4, 2020

markmandel commented Sep 11, 2020 •

edited

Loading

markmandel commented Sep 15, 2020

markmandel commented Sep 15, 2020

roberthbailey commented Sep 15, 2020

aLekSer commented Sep 15, 2020 •

edited

Loading

markmandel commented Sep 15, 2020

aLekSer commented Sep 16, 2020

roberthbailey commented Sep 17, 2020

markmandel commented Sep 17, 2020

roberthbailey commented Sep 17, 2020

markmandel commented Sep 17, 2020

Upgrade to Kubernetes 1.16 #1649

Upgrade to Kubernetes 1.16 #1649

Comments

roberthbailey commented Jun 25, 2020 • edited by aLekSer Loading

roberthbailey commented Jun 25, 2020

roberthbailey commented Aug 13, 2020

markmandel commented Aug 25, 2020 • edited Loading

thisisnotapril commented Aug 27, 2020

roberthbailey commented Aug 27, 2020

aLekSer commented Sep 3, 2020

aLekSer commented Sep 3, 2020 • edited Loading

roberthbailey commented Sep 3, 2020

markmandel commented Sep 4, 2020

markmandel commented Sep 11, 2020 • edited Loading

markmandel commented Sep 15, 2020

markmandel commented Sep 15, 2020

roberthbailey commented Sep 15, 2020

aLekSer commented Sep 15, 2020 • edited Loading

markmandel commented Sep 15, 2020

aLekSer commented Sep 16, 2020

roberthbailey commented Sep 17, 2020

markmandel commented Sep 17, 2020

roberthbailey commented Sep 17, 2020

markmandel commented Sep 17, 2020

roberthbailey commented Jun 25, 2020 •

edited by aLekSer

Loading

markmandel commented Aug 25, 2020 •

edited

Loading

aLekSer commented Sep 3, 2020 •

edited

Loading

markmandel commented Sep 11, 2020 •

edited

Loading

aLekSer commented Sep 15, 2020 •

edited

Loading