services/horizon: update config to use history archive urls from our entire quorum set

[mollykarcher]

What problem does your feature solve?

See parent epic: #5163

In puppet, we currently set history_archive_urls to a single validator on our Horizon instances.

What would you like to see?

We should set history_archive_urls to be a list/collection of history urls from all of the validators in the quorum set which is defined in the same config.

Horizon is set up to handle pooling of archives and random access across said pool of archives. However, given that this has always set up to be 1 in our infrastructure, we should take the time to ensure this is working as expected once done. In particular:

• Random access of pooled archives works as expected
• Potentially consider moving this to round-robin to equalize load if necessary historyarchive: Harden the ArchivePool to be robust against errors.  #5167
• Failure to access certain history archive urls (timeouts / existence / corruption / etc) is retried properly. "Unhealthy" archives are dropped/ignored, service is resilient to these types of failures historyarchive: Harden the ArchivePool to be robust against errors.  #5167
• [Optional] Consider dynamically generating history_archive_urls from the quorum set config services/horizon/cmd: spike, options on how to obtain archive server list #5142

[sreuland]

@mollykarcher , to confirm, do we want to include all other external non-sdf validator servers from qset in horizon history_archive_urls , or just limit this to include the 3 SDF validator history servers?

[mollykarcher]

@sreuland yes, we want to include all validators that are configured in our qset

[sreuland]

further discussion, the scope on this is refined a bit:

• reconfig archive urls for just the three hosted archive servers from the three respective validators -
• defer to historyarchive: Harden the ArchivePool to be robust against errors.  #5167 for the other hardening aspects mentioned in description
• this ticket scope will focus on changing staging and in-active clusters first, monitor to confirm changes are fine
• do a second puppet pr to change the active cluster

[sreuland]

moving to in review as puppet pr/3642 is up for review