Unable to rotate Consul CA after 1st successful rotation

[ashwinkupatkar]

Hello,

I enabled connect in my consul cluster so it had the default consul ca provider.

I rotated it with my custom CA configs, forming a chain to the root CA system. All went smooth.

The second time I try to rotate it. It does not work. It gives below error
rpc error making call: error generating CA certificate: x509: requested SignatureAlgorithm does not match private key type

This hinders CA rotation as expiration time nears. I verified from my end that the 2nd time I create a new csr and its private key, its an "ec" private key.

Why does this happen? Can someone provide an insight as to what is happening here ?

Version of consul : 1.9.5

Signer is Vault (Intermediate , key type : RSA )

Consul CA (key type : EC ) is getting its CSR signed from this vault

[ashwinkupatkar]

@ishustava not sure whom to reach out so tagged you to this issue as i can recollect your name.

Can you please let me know what is the issue here ?

Thanks

[mister2d]

I would also like the know Consul's designed behavior when using the built-in CA.

1. Will the intermediate CA get automatically rotated and distributed to clients?
2. Can the intermediate CA be rotated manually without regenerating a new root certificate?
3. If not, what is the point of the root CA having such a long TTL (generally x2 of the intermediate) if that time value can't be used?

Thanks!

[ashwinkupatkar]

Hi @dnephin, if you could help debug this issue please .... as i am unable to rotate ca after first successful rotation.

[kyhavlov]

@ashwinkupatkar Can you give more details about what you're doing to get the error?

• Is the CA currently using the built-in (Consul) provider and you're trying to reconfigure it to use Vault?
• Was it already on the Vault provider here and you're trying to update its certificate?
• Is this in a primary or a secondary datacenter?

@mister2d The built-in CA doesn't create an intermediate CA cert in the primary datacenter (intermediates are used by secondary DCs). Whatever is provided to the built-in CA in the primary datacenter is used as the root of trust for the cluster and is used to sign client certs in the primary DC. Each secondary datacenter gets an intermediate CA by sending a CSR to the primary to get back an intermediate signed by the root CA there, and the secondary uses that intermediate CA to sign client certs for its datacenter. In the secondaries, these intermediate certs are periodically rotated and watched by clients (envoy proxies) in order to update when a rotation happens.

These intermediate certs can be manually rotated by reconfiguring the CA provider for a given secondary datacenter with a new signing key, or a new CA provider (like going from consul -> vault provider) - both of those changes would regenerate the intermediate by going through the CSR process and getting a new cert signed by the primary.

Hope that helps - happy to answer more questions about this.

[ashwinkupatkar]

Hi @kyhavlov , Thanks for reaching out.

I am trying to do the following:

1. I have a vault cluster serving as a pki intermediate CA , say pki_int1 [RSA CA]
2. I enabled another intermediate end point on the same vault cluster, say pki_int2 [EC CA]
3. From this vault, I created CSR and EC key and exported the key out.
4. I signed this generated csr with the pki_int1 CA
5. I imported this signed CA (pki_int2) into consul CA using the update ca configuration api call as mentioned here.
6. The new CA gets applied to the primary and secondary consul cluster and all the communication just works fine.
7. Now as part of my testing, I try to rotate the consul CA, assume the expiration time is nearing. I see the error as below:

rpc error making call: error generating CA certificate: x509: requested SignatureAlgorithm does not match private key type

Why is this happening? I do not see this error while rotating built-in Consul CA with my custom CA. It is only after rotating the custom CA with new custom CA I face the above issue.

I hope that clears your understanding.

I do not wish to integrate with Vault to manage Connect CA at the moment. I want to use vault just for the signing purpose and keep less dependency on vault.

Please let me know.

Thankyou!

[ashwinkupatkar]

Hi @kyhavlov, did you get any clue ? Thanks

[ashwinkupatkar]

Hi @kyhavlov can you help me with above issue ?

[ashwinkupatkar]

Hi @kyhavlov, I was able to rotate the CA by introducing a subordinate EC CA to RSA CA. From this EC CA, I could create a Consul CA and updated it to Consul. CA Rotation works fine.

But after rotating the CA, communication between application breaks in primary cluster and it takes 1 hour 50 mins to recognize this change. It keeps on complaining with the following error in envoy logs:

Failed to load trusted CA certificate from <inline>

This is issue is automatically resolved by itself in 1 hour 50 mins and communication between the applications begins to work.

In the secondary cluster the new CA replication is almost instant. But the new communications break and gives the same error as above. The only solution to restore the communication in the secondary cluster is to rebuild secondary cluster from scratch.

What can be the issue?

Following versions are used:

Consul : 1.9.5
Consul helm : 0.30.0
consul-k8s: 0.24.0
envoy: 1.16.2

[ashwinkupatkar]

Hi @lkysow, can you please provide me guidance on the above issue ? Thanks

[dhiaayachi]

Hi @ashwinkupatkar,
sorry for the delay. I did some analysis based on the information you provided and I think that the first issue you are experiencing is related to the same issue experienced in #9268 which is caused by a Vault issue hashicorp/vault#7709. This happen when Consul, using a Vault provider, try to cross sign a certificate with an RSA algorithm type with a ca that have an EC algorithm type.

We are actively working to get the fix to hashicorp/vault#7709 merged in vault in the mean time using the same key type cross provider should work around this problem, this can be set using PrivateKeyType parameter as follow:

  "Provider": "consul",
  "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RotationPeriod": "2160h",
    "PrivateKey": "private_ec_key_json",
    "PrivateKeyType": "ec",
    "PrivateKeyBits": 256
  }
}

or for RSA:

  "Provider": "consul",
  "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RotationPeriod": "2160h",
    "PrivateKey": "private_rsa_key_json",
    "PrivateKeyType": "rsa",
    "PrivateKeyBits": 2048
  }
}

[ashwinkupatkar]

Hi @dhiaayachi, I still have doubts as to how the above vault issue fix the CA rotation executed on consul.

I believe consul connect CA issue is similar to vault.

[jkirschner-hashicorp]

@kisunji : Is this intended to be closed by #17846?

[kisunji]

Oh no, I must have closed this by accident. Re-opening.