Introduce support for Projected Volumes kubernetes ServiceAccount Tokens

[JJotah]

Add support for Projected Volumes kubernetes ServiceAccount Tokens. In some cloud providers we can't change the expiration token and linkerd is adding the projected volumes. So the install-cni should watch like istio did checking the serviceaccount token

[wmorgan]

Thanks @JJotah , we will take a look.

[alpeb]

Fixes linkerd/linkerd2#12573

[alpeb]

Thanks @JJotah , this makes a lot of sense!
Can I suggest introducing a variable analog to cni_conf_sha but tracking the sha for the token file? So we avoid introducing that new if/else in sync(). This would require two additional refactors though:

• Pass the entire filepath to the function instead of using filepath="${HOST_CNI_NET}/$filename"
• Under your new inotifywait call when you call sync, you'll have to add some logic to figure what's the actual config file name being used (as per my comment below).

[alpeb]

We can't be sure at this point if the config is held on $DEFAULT_CNI_CONF_PATH, which is a .conf file, or a .conflist file.

[alpeb]

In create_cni_conf() we have SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount. Can you make that a global variable, so we can reuse it here?

[olix0r]

We probably need to be careful with backgrounding these tasks -- we want to ensure that inotifywait failures are propagated so that the program can fail if any of these calls fail.

There are other changes we probably need to make to this script first, but we'll want to ensure that we preserve PIDs and wait on them to surface errors.

[alpeb]

I did some more research on this, thinking it's been a while since k8s injects projected kube-api-access volumes, so why is linkerd-cni still working past the token expiration (1 hour by default)?
Inspecting the linkerd-cni pod we see the injected volume:

  - name: kube-api-access-729gv
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

I could also observe how the token file got indeed recycled every hour. But actually inspecting those tokens revealed they're good for one year! So even if the token file gets renewed, linkerd-cni keeps on using the first one, which will continue to be valid.
Turns out the reason is kube-api's flag --service-account-extend-token-expiration:

Turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.

This flag still shows as default true in the docs, which explains the above. But obviously we can't rely on that, thus the importance of this PR.