createOrReplace Service Account leaks secret tokens

[leonz]

Hi,

We are facing an issue that our code is regularly calling createOrReplace on a service account to make sure that it stays up to date, but it seems that on each call we are generating a new token without clearing out the old:

elastic-operator-token-stx62                kubernetes.io/service-account-token   3      97d
elastic-operator-token-svl9g                kubernetes.io/service-account-token   3      201d
elastic-operator-token-swv6z                kubernetes.io/service-account-token   3      120d
elastic-operator-token-sztz2                kubernetes.io/service-account-token   3      70d
elastic-operator-token-t47x6                kubernetes.io/service-account-token   3      247d
elastic-operator-token-t4ccc                kubernetes.io/service-account-token   3      75d
elastic-operator-token-t4mfl                kubernetes.io/service-account-token   3      124d
elastic-operator-token-t584b                kubernetes.io/service-account-token   3      108d
elastic-operator-token-t6pg5                kubernetes.io/service-account-token   3      252d
...

Is this expected behavior? I would expect that createOrReplace also deletes this service account token when deleting the service account? Or is there an alternative function call that we should use?

[shawkins]

There is no delete happening under the covers from the client perspective. It's just doing a POST, then a PUT if that fails. The controller for service accounts should have the responsibility for cleanup on a PUT - did you see if there's already a kubernetes issue for that?

Could you do your update as a patch instead? From what I can see that will not automatically trigger the generation of a new secret.

[leonz]

We can certainly patch instead of createOrReplace within our code, but we end up with a somewhat un-elegant solution where we need to perform different actions for different types of resources. For example, with the load function:

KubernetesClient client = new DefaultKubernetesClient();
// yamlResourceStream has a bunch of different K8s resources
List<HasMetadata> resources = client.load(yamlResourceStream).get();

List<HasMetadata> resourcesToCreate = new ArrayList<>();
for (HasMetadata resource : resources) {
    if (resource instanceof ServiceAccount && resourceExists(resource)) {
        client.serviceAccounts().patch(resource);
    } else {
        resourcesToCreate.add(resource);
    }
}

client.resourceList(resourcesToCreate).createOrReplace();

I'm curious how/if others have approached or thought about this.

---

Investigating a bit into Kubernetes docs - it does seem to imply that the job of the TokenController is to delete these secrets. But all of these secrets are being mapped (by name and uid) to the same serviceaccount which is receiving updates from Kubernetes, so they are not getting deleted.

# all secrets have this, mapped to the same service account
metadata:
  annotations:
    kubernetes.io/service-account.name: elastic-operator
    kubernetes.io/service-account.uid: 76068489-e1b4-4955-952b-b3eca1049e62

# service account is getting updated:
    manager: kube-controller-manager
    operation: Update

So my question would be - why are we creating a new secret on each update to the service account?

[leonz]

I've filed kubernetes/kubernetes#106131 with some more information that I've uncovered. I suspect that this issue will be closed as "working as expected", but either way this is a weird behavior that discourages me from wanting to use createOrReplace altogether.

[shawkins]

So my question would be - why are we creating a new secret on each update to the service account?

The yaml you are using to create the service account does not have the secrets specified correct? This is a similar issue to #3122 - the kubernetes logic acts upon the PUT independently of the existing item. My guess is that to workaround you need to use a patch, or that some logic - either in your application, or through an enhancement here - would need to check for the existence of the service account and update the object with the existing secrets if none were specified. Related to using a patch instead is that you may be expecting createOrReplace to behave like kubectl apply, but it does not #3420 - kubectl's logic, which will produce patches, rather than puts, does not trigger the creation of a new secret.

[stale]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!