Enable a "forward rolling" PKI strategy that does not break AIA or CRLs

[abriening]

Is your feature request related to a problem? Please describe.

As documented in other issues the pki rotation isn't as clear or clean as it could be.

Related to #2581

Describe the solution you'd like

Here are the steps to create a "forward rolling" PKI strategy that does not break AIA or CRLs. Assuming new endpoints are added to enable this process.

Given the endpoints "pki/cert/{serial}/raw(/pem)" are implemented (similar to "/pki/ca(/pem)" endpoints).

Given you have a root or intermidate at "pki-int-1" mount.

Create Intermediate CA at "pki" mount. Sign with "pki-int-1". Set signed intermediate on "pki". The "pki" cert should now exist in both "pki-int-1" and "pki".

Create/sign another mount at "pki1", this is just for bootstrapping CRLs.

Set "issuing_certificates" to "pki-int-1/cert/{serial}/raw" with the serial of the intermediate cert.

Set "crl_distribution_points" to "/pki/crl" & "/pki1/crl".

Any certs created on "pki" mount will have AIA info pointing at the exact cert in "pki-int-1" and CRL pointing to both "/pki/crl" & "/pki1/crl".

Process to Roll Forward"pki"

Add a new "pki-next" create and sign against "pki-int-1" same as "pki".

Set "issuing_certificates" to "pki-int-1/cert/{serial-for-new-pki-next}/raw"

Set "crl_distribution_points" to "/pki/crl" & "/pki1/crl", same as bootstrapping.

Use /sys/remount to move "pki1" to "pki2" (or drop "pki1"), "pki" to "pki1", "pki-next" to "pki". This should be close to zero downtime if scripted.

CRLs work because they are always looking in both locations (previous pki and new pki). CRLs for certs on the old "pki1" should have all rolled/off expired by now anyway (assuming we're rotating at a reasonable interval).

AIA works because they're set to at exact paths under "pki-int-1".

Process to Key-Roll "pki-int-1"

Create/sign the new root or intermediate under "pki-int-2". Use "pki-int-2" for future "pki" "issuing_certificates".

Instead of using the "rotate between two pki mounts" suggested here #2581 (comment), this allows consumers to always use "pki" mount. Using the "rotate between two pki mounts" method means all consumers need the extra knowledge of which pki mount is the active pki mount. It's easy to store that in consul but adds that complexity to all consumers. This way we keep the complexity on the pki configuration/rolling strategy and the consumers can remain "stupid".

Describe alternatives you've considered

The alternatives are to use the "rotate between two pki mounts" method or don't use rely on AIA/CRL.

[HridoyRoy]

Hi @abriening , this is a very interesting solution. I'll raise it with the team and report back on the ticket. Thanks!

[jefferai]

CRLs are signed, so a very easy way to do this is just use a different HTTP distribution point for your CRLs and CA certs.

[abriening]

@jefferai do you mean just use api-gateway/reverse-proxy/load-balancer in front front of the CRL/CA using the consul pointer to the active pki mount? We did discuss that, simple but still requires an extra piece of infrastructure to run (or piggy-back on existing system). Check the MR, it's not much code to support the raw endpoints on any cert.

I also am tending towards shortening cert expiry so I care less about AIA/CRL in general and keep things as simple as possible.

[jefferai]

No, I meant set the URLs that are embedded in the certs to point to a separate location that is accessible via unauthenticated HTTP. Then you can point clients there, but change what populates that location as you wish.

That said...we have always suggested keeping lifetimes short enough that you don't need a CRL. Vault can handle it so long as you have the entropy available (some users generate a certificate on the fly per incoming connection) and there are a lot of articles out there from security experts that go into the issues around using CRLs such as the soft-fail problem (which also exists with OCSP). Definitely a 👍 if you're heading down the path towards short expiry.

[abriening]

For nomad deployed jobs, we're already there; seamless pki integration. We're trying to figure out the legacy situation, where we have servers that may not get reprovisioned/configured very often. The approach above, enabled by the MR, reduces the footprint to just vault instead of vault+crl server.

But now I'm thinking the better approach is reduce expiry on all systems, including legacy. Reducing the likelihood of needing a cert revoked. Vault agent (although on windows, but seems like nssm works there) running on those servers might mean we can keep the expiry shorter than reprovisioning cycle.

Thanks for the discussion and quick responses 👍

[hghaf099]

@abriening Thank you for filing this ticket. Based on the comments, it seems that there is not much more discussion left on the issue. I am going to close this issue for now. But, please feel free to re-open this issue or a new one if you would like to discuss further on the matter.