Update Calico to v3.16.0 for k8s 1.16+

[hakman]

https://docs.projectcalico.org/archive/v3.16/release-notes/

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

/retest

[gjtempleton]

Do we want to move the logic from

kops/upup/models/bindata.go

Line 8618 in fc2e70e

{{ if and (eq .CloudProvider "aws") (.Networking.Calico.CrossSubnet) -}}

for installing the k8s-ec2-srcdst pod to here as this is now the supported path?

I think allowing users to enable this will require them to add permissions to worker nodes as we currently only give the required ec2:ModifyInstanceAttribute permissions to masters, at a minimum I think we need to add something to the docs, and make clear that enabling this alongside the existing logic for installing k8s-ec2-srcdst is an untested path.

[hakman]

Yes, we do, and also enable WireGuard. But best to do it in some follow-up PRs. k8s-ec2-srcdst would be removed completely. No reason to have both, I guess.

[seh]

Does calico-node talk to the AWS API for this, or is it the calico-kube-controllers pod? If it's the latter, we could optionally confine it to run only on master nodes, and grant the IAM permissions only to the instance profile used by the masters.

[Time passes...]

It's part of Felix, and therefore part of calico-node.

[hakman]

@lwr20 I am a bit lost here and really puzzled about why I cannot make 3.16.0 work. Any idea what I am doing wrong? 😄

[fasaxc]

@hakman is there a way to get the actual error that caused validation to fail?

[hakman]

@fasaxc I am not really sure where the issue comes from from. I also tried using https://docs.projectcalico.org/manifests/calico.yaml and still same issue.

You can find the logs of the master node from the test here:
https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/kops/9829/pull-kops-e2e-cni-calico/1299270983718277120/artifacts/ip-172-20-39-18.ap-northeast-2.compute.internal/

[seh]

Looking at the controller manager logs, I see lots of errors that match what I saw when I tried upgrading etcd directly from version 3.2.24 to version 3.4.3.

[hakman]

@seh I got some help from @fasaxc and tracked the issue to projectcalico/cni-plugin#942.
Looks like the pre-release calico/cni:v3.16.0-2-gf6793aa image addresses the issue and the tests can run now.

Thanks again for all your help @fasaxc :).

[hakman]

/hold to decide if should be merge with patch to wait for v3.16.1.

[hakman]

Works pretty well to use during the net few weeks until a new release is available.
/hold cancel

[olemarkus]

/lgtm