Kubernetes: does not reload projected service account token

[ncsibra]

Bug Report

Describe the bug
In EKS 1.21 projected service account tokens are enabled by default, so the token is renewed every hour and fluent-bit should reload it to use the latest token.
The feature is described here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md

An example volume configuration:

  volumes:
  - configMap:
      defaultMode: 420
      name: fluent-bit
    name: config
  - hostPath:
      path: /var/log
      type: ""
    name: varlog
  - hostPath:
      path: /var/lib/docker/containers
      type: ""
    name: varlibdockercontainers
  - hostPath:
      path: /etc/machine-id
      type: File
    name: etcmachineid
  - name: kube-api-access-ffbpz
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

To help with the migration, kube apiserver has some metrics to find related services which does not reload the token, described here:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md#serviceaccount-admission-controller-migration

In kube-apiserver-audit logs I see events like this, which shows that fluent-bit using an old token:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "ab36f0b8-2732-4c8f-83d7-6e517c3c056b",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/monitoring/pods/dev-elasticsearch-curator-27288720-xlk96",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:monitoring:fluent-bit",
    "uid": "c9528d37-c045-426b-94a9-43f91773bd81",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:monitoring",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/pod-name": [
        "fluent-bit-5cs6q"
      ],
      "authentication.kubernetes.io/pod-uid": [
        "f1dcd718-0399-437c-9bcb-5cbd8ae2462f"
      ]
    }
  },
  "sourceIPs": [
    "10.151.25.2"
  ],
  "userAgent": "Fluent-Bit",
  "objectRef": {
    "resource": "pods",
    "namespace": "monitoring",
    "name": "dev-elasticsearch-curator-27288720-xlk96",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2021-11-19T12:00:05.670833Z",
  "stageTimestamp": "2021-11-19T12:00:05.674382Z",
  "annotations": {
    "authentication.k8s.io/stale-token": "subject: system:serviceaccount:monitoring:fluent-bit, seconds after warning threshold: 2766881",
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"fluent-bit\" of ClusterRole \"fluent-bit\" to ServiceAccount \"fluent-bit/monitoring\""
  }
}

The relevant part is: "authentication.k8s.io/stale-token": "subject: system:serviceaccount:monitoring:fluent-bit, seconds after warning threshold: 2766881".

To Reproduce
Just use a kubernetes cluster with Bound Service Account Tokens feature enabled and wait for more than expirationSeconds time (1 hour by default) to see these events in kube apiserver audit logs.

Expected behavior
Reload the service account token periodically, preferably less than the expirationSeconds set in projected volume.

Your Environment

• Version used: 1.8.9
• Configuration:

apiVersion: v1
data:
  custom_parsers.conf: |2

    [PARSER]
      Name containerd
      Format regex
      Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
      Time_Key    time
      Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  fluent-bit.conf: |2

    [SERVICE]
      Daemon Off
      Flush 5
      HTTP_Server On
      HTTP_Listen 0.0.0.0
      HTTP_Port 2020
      Log_Level info
      Parsers_File parsers.conf
      Parsers_File custom_parsers.conf


    [INPUT]
      Name tail
      Path /var/log/containers/*.log
      Parser containerd
      Tag kube.*
      Refresh_Interval 5
      Mem_Buf_Limit 64MB
      Skip_Long_Lines On


    [FILTER]
      Name kubernetes
      Match kube.*
      Kube_Tag_Prefix kube.var.log.containers.
      Kube_URL https://kubernetes.default.svc:443
      Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
      K8S-Logging.Exclude On


    [OUTPUT]
      Name forward
      Match *
      Host fluentd
      Port 24224
      Retry_Limit False
      tls off
kind: ConfigMap
metadata:
  creationTimestamp: "2021-10-18T10:22:49Z"
  labels:
    app.kubernetes.io/instance: fluent-bit
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: fluent-bit
    app.kubernetes.io/version: 1.8.9
    helm.sh/chart: fluent-bit-0.19.5
  name: fluent-bit
  namespace: monitoring

• Environment name and version (e.g. Kubernetes? What version?): EKS 1.21
• Operating System and version: Amazon Linux 2 (Ami name: amazon-eks-node-1.21-v20211004)
• Filters and plugins: See the configuration above.

[novegit]

I created a PR to fix it #4487.

Meanwhile you can mitigate it, using the Kube_Token_Command option, because this reloads the token every 600 seconds:

Kube_Token_Command  /usr/local/bin/cat /var/run/secrets/kubernetes.io/serviceaccount/token

you may have to check the full path for cat, in the default docker image, its /usr/local/bin/cat

[KouriR]

you may have to check the full path for cat, in the default docker image, its /usr/local/bin/cat

This is only present in the -debug tagged images.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.