consul connect ca w/ vault provider stuck, can't rotate to new pki mounts or back to consul ca provider

[ericbrumfield]

I've had a consul cluster operating off and on for about a year now. Overall the original consul version we used was 1.5.3. A normal procedure we'd do periodically was to update the consul ca config using set-config to rotate the pki mounts that connect ca used with the vault provider. Today this failed and the only recent change I'm aware of is that I upgraded consul to 1.8.2 about a month ago to begin using ingress gateways and other new features we require. The upgrade seemed to go fine at the time, no problems, client agents connect and everything seemed good.

I now seem to be stuck, when I attempted to use set-config with a config file that used a new RootPKIPath and IntermediatePKIPath, it created the new root pki mount in vault but I think it failed during cross signing or something. The reason I attempted to do this rotation is that the root CA cert that consul has been using expired about 4 days ago.

Some background info I came across that it had expired while I was trying to test connectivity between ingress gateway and an application's sidecar proxy, eventually after digging for hours I was able to get envoy trace logs in the sidecar and saw cert expired related errors, then was able to find from vault side that the ca vault pki mount root cert expired. I feel like this is another bug or feature request with consul UI as everything there was green and appeared fine setup wise even though the certs in play were expired that the sidecar was using.

My current issue for this report is that I can not get consul connect ca to rotate to new pki mounts via consul connect ca set-config approach, or get it to move from the vault provider back to the consul provider. Both of these attempts gives me this error at cli and I feel completely stuck. I fear that the consul cluster is hosed and that I may have to setup new server nodes from scratch again.

Error setting CA configuration: Unexpected response code: 500 (rpc error making call: error having Vault cross-sign cert: Error making API request.

URL: PUT https://internal-vault-ha-509231764.us-gov-west-1.elb.amazonaws.com:8200/v1/pki_consul_connect_root_8/root/sign-self-issued
Code: 500. Errors:

* 1 error occurred:
        * error signing self-issued certificate: x509: requested SignatureAlgorithm does not match private key type

)

I've also tried setting ForceWithoutCrossSigning: true , and that hasn't changed anything either. I'd really appreciate some guidance if there is an approach or something I can try to get out of this situation without the cluster being borked and requiring a new setup.

Using Vault version 1.1.1 and consul 1.8.2

[ericbrumfield]

Just an update, due to time factors for the project I'm working on, I've re-built our consul cluster from scratch because of this. We will be proceeding with using consul's built in CA instead of the consul connect ca vault integration. The consul-vault integration has been pain over the past year and has resulted in 2 bricked consul clusters for me. We don't feel that we can run a production setup with consul-vault integration as the connect CA provider.

On a good note, after I rebuilt our cluster server nodes, and setup everything again, I was able to finally curl through our consul ingress gateway => app sidecar proxy => app api.

For this specific situation where the vault issued certs consul connect ca applied to the agents for mTLS being expired, when you try to test traffic through the ingress gateway you'll be greeted with a 503 response. The consul admin UI will show green for everything (ingress service, app service, upstream etc.), which is misleading that the ingress/upstreams etc. are all good to go, while the sidecar will drop during the tls handshake and report that in the envoy logs on the sidecar proxy.

[ashwinkupatkar]

I am facing similar issue with consul version 1.9.3

I am unable to rotate ca . It runs smooth for the fresh installation but during rotating gives me error as above

error generating CA certificate: x509: requested SignatureAlgorithm does not match private key type

I checked and my key is of ec type and it works perfectly fine for the fresh cluster deployment with consul connect but fails during ca rotation.

[ashwinkupatkar]

I believe it was working fine in conversion 1.8.x

[markan]

Tracing the logic around these error messages this appears to be a problem with a mismatch between the key types; one key is EC and the other RSA. Vault cannot cross sign keys if they mismatch; we're tracking this as: hashicorp/vault#7709.

I'm still somewhat puzzled how this bit you in your circumstance; perhaps when doing 'set-config' a mismatched key slipped in.

[markan]

This should be fixed with hashicorp/vault#12514, closing. Feel free to reopen if it persists with the latest vault.