Add ability to set encryption flags for Cilium

[cten]

When using the CNI for Cilium my team would like to enable the ipsec settings for pod-to-pod and node traffic.

[moshevayner]

Hey @cten !
From looking at the cilium-agent command-line switches, did you mean these two parameters?

      --encrypt-interface string                      Transparent encryption interface
      --encrypt-node                                  Enables encrypting traffic from non-Cilium pods and host networking

I just want to make sure I have the right idea of what needs to be done here.
Thanks!

[olemarkus]

Have a look at https://docs.cilium.io/en/v1.7/gettingstarted/encryption/

See what the helm template is changing when you enable/disable encryption and try to mimic this in kops. Could be you also want to try to manage the encryption key.

[moshevayner]

Hey @olemarkus, thanks for sharing that, really helpful! 😄
So I assume the idea is to grab the changes from the helm chart as you suggested, then parameterize the template for Cilium and add the necessary fields to Cilium spec in networking.go and v1alpha2/networking.go?

[olemarkus]

Yep. That's about it.

[moshevayner]

Cool. I can handle that one!
/assign

[moshevayner]

Hey @olemarkus
So I was able to find the differences between the helm-generated templates of with and without the encryption flags, but one thing I'm not sure about is the part of generating the encryption key as described here. According to the doc, the secret containing the encryption key should be created before applying the manifests. Otherwise, I assume that the cilium pods won't get scheduled, probably with CreateContainerConfigError due to the missing secret.
How would you say we should approach that from kops perspective?

[olemarkus]

The kops way of doing tis would probably be to use the kops secret store.
Similar to how weavepassword works, you would crreate a "ciliumkey" secret or something that is deployed to a well-known path on every node. Instead of mounting a configmap, the cilium template would simply point to this file instead.

The kubectl create secret step would be fully replaced by this.

[moshevayner]

Hey again @olemarkus 😄
So following our discussion in #kops-dev Slack channel, it's pretty certain now that adding node-encryption breaks the connection between the nodes and the masters.
I've spent a good amount of time on it and found some possible issues on cilium repo which could be related (one that is a fairly good candidate is cilium/cilium#10743).
Except for that matter- everything works as expected.
Enabling encryption between cilium pods without adding the nodes themselves works as should.
At this point- do you think I'm good with moving forward to creating the PR?
I only need to wrap up the documentation update and pretty much done.
I just wanted to get your thought on the node encryption thing, since I'm slightly uncomfortable with adding support for something that doesn't work at the moment.
I'd love to get your thought on that part.
Thanks much!

[olemarkus]

I think it is fine to not support node encryption until we figure out why it is failing.

You can enable etcd-manager for cilium (https://kops.sigs.k8s.io/networking/#using-etcd-for-agent-state-sync) to see if this helps. it is unclear to me what exactly is the problem in that issue though.

There are so many cilium PRs that have gone into 1.18 though, so a PR for encryption may hold until we cut the 1.18 branch.

[moshevayner]

Hmm, yeah that's a valid point. I'll give that a try later tonight and see if it helps.
If it still doesn't work- I'll probably go as you suggested and only add the encryptionEnabled flag. That should save a few steps at least.
Thanks again!

[moshevayner]

So no matter what I tried, enabling node encryption breaks the cluster.
I'm going to drop the nodeEncryption support and only add the support to the main encryption flag which enables encryption between pods within the cluster.

[cten]

Does this require Kops 1.19? Thanks by the way for doing this!!!

[moshevayner]

@cten correct, this is included in 1.19 release.
Glad to help!! 😄