How versioned secrets can help with auto-restart message pumps

[stijnmoreels]

We have released something called 'versioned secrets' in our Arcus Security library, which makes sure that multiple versions of a secret are taken into account when requesting a secret. This can come in handy when you want to rotate keys and don't want downtime (as you can rely on the second key).
The background jobs repo has a job that listens for 'secret new version created' events and uses this to restart the message pump, but, with this new versioned secrets, that may not be necessary anymore if we support versioned secrets in our message pumps.

Comments? @pim-simons @fgheysels @gverstraete @meersschautarnaud

[pim-simons]

I haven't used this background job so no "real" experience here, but the way you describe it seems like supporting versioned secrets is the way to go.

[gverstraete]

Do we really need a "restart"? - What do you actually mean with that? Can't we just update the config? - Or do you mean re-initialing the connection?

[stijnmoreels]

'Restart' here means indeed re-initiating the connection with Azure Service Bus. Internally, this means that the background job will request via the message pump a new secret containing the 'new' connection string in the secret store.

[stijnmoreels]

The 'restart' functionality here helps to internally re-initialize jobs instead of restarting the entire application to request a new secret.

[fgheysels]

I was going to type something, but now I'm thinking: How does versioned secrets resolve this issue ?
When a secret is rolled, you'll create a new version of the secret. The component that uses that secret will not be able to connect to the resource since the secret that it has, is expired / rolled. So, that component needs to retrieve a new version of the secret. Normally , the component will be able to connect to the resource with the most recent secret that is found in KeyVault.
So, actually, I don't see how versioned secrets resolves this problem ? It is solved without making use of versioned secrets ?

[stijnmoreels]

It is solved without making use of versioned secrets ?

It is currently solved in two ways:

1. configurable amount of authentication failures before restart (in the message pump itself) -> includes downtime
2. background job that listens on 'secret new version created' -> requires additional resources (serivce bus topic, and registration on system event grid topic of key vault events)

Versioned secrets could help here if a primary and secondary key is used. While rolling the primary, the secondary key could be used. If these keys are both alternatively set as a 'new secret version', then the message pump can fully rely on the secrets without external help of events.

[meersschautarnaud]

@stijnmoreels in this case you would add both the primary and secondary key to the KeyVault secret?
If so this would solve the issue I was thinking about. If you would use the versioned secrets there would probably be a delay for the new secret version to come in depending on the polling on the secret?

[stijnmoreels]

Yeah, you could indeed do something yourself with both secrets as well. Then versioned secrets would maybe only have added-value in specific circumstances.

[fgheysels]

The background job is imo more usable then the versioned secrets feature.

Suppose you have a client-secret that is rolled. The new secret is added in KeyVault (the secret is updated with a new version). KeyVault raises an event, and the background job is triggered. We can make sure that our components that use this specific secret, are notified and do the necessary things to use the new version of the secret, making sure that our component continues to work seamlessly.

With the versioned secrets feature, on the other hand, our component has the ability to retrieve x versions of the secret. Typically, we'll use the first secret to connect to a resource, and if that doesn't work, we'll use the version before that etc...
But I'm wondering: in what case would using the most recent version not work, but using the previous version will ?

[gverstraete]

This is assuming that you have third parties that need time to use the new secret.

[fgheysels]

This is a bit explained here: if you use both primary and secondary keys as multiple versions, you could rotate one and the secret store would move over to the other.

Imo, when you have primary and secondary keys, you should have 2 secrets.

[fgheysels]

This is assuming that you have third parties that need time to use the new secret.

Might be me, but I don't see it :)
It's the 'protected resource' that defines which secret must be used to connect to it. It's not the consumer that can decide on that. If the consumer uses the old version, the resource will say: 'hey, that key is not correct'.

[gverstraete]

As discussed during this week, it is the same secret because it has the same meaning. The rotation between primary & secondary is just latest version & version - 1. The main goal I had was just to have the SAS token validation on WebApi's to be able to validate the last two secrets, so that third party consumers have the time to migrate to the new secret.

[gverstraete]

So, to add on to this, this is from a consumer perspective. From a producer perspective I do agree with you @fgheysels . But let's not question versioned secrets on itself :) They do have value!