Secondary CA Cert is generated twice during initialization

[mkeeler]

Overview of the Issue

During initialize of a secondary datacenters CA, it will do two rounds of CSR generation and RPCs to get it signed by the primary instead of 1.

The first is done here:

consul/agent/consul/leader_connect.go

Line 204 in 6817676

if err := s.initializeSecondaryCA(provider, roots); err != nil {

Then the secondaryCARootWatch routine is started and another one takes place here:

consul/agent/consul/leader_connect.go

Line 610 in 6817676

if err := s.initializeSecondaryCA(provider, roots); err != nil {

The root cause of this is that we are making no attempt to prevent certificate regeneration when the roots haven't changed. The secondaryCARootWatch routine starts initially with performing a non-blocking query which will return immediately with the same roots as the same RPC performed in initializeCA.

This should be benign enough but maybe we want to fix it 🤷‍♂

Reproduction Steps

Steps to reproduce this issue, eg:

1. Setup a primary datacenter with connect enabled.
2. Start a secondary datacenter with connect enabled.

[banks]

I think the issue is a little more subtle than described. I described it previously here after debugging for about a day: #6771

The root cause of this is that we are making no attempt to prevent certificate regeneration when the roots haven't changed.

Actually we do... The real issue is that our code uses the init routine in two places, in one place the "state" passed is the local state, in the replication routine it's the primary's state yet our "has it changed" check is done assuming the state is always the local state which means in replication case we are comparing the primary's keyID against the secondary's which will always fail and so cause another CA init round.