services/horizon/internal/ingest: Prevent redundant and concurrent state verification runs

[tamirms]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Close #4833

As a (most likely unintentional) side effect of #4204 , it is possible for multiple ingesting nodes to perform state verification concurrently. This commit introduces a state verification lock which ensures that only one ingesting node can perform state verification at a time.

This commit also adds two new configuration variables:

• --ingest-state-verification-frequency which specifies the frequency in checkpoints for how often state verification is run
• --ingest-state-verification-timeout which specifies a timeout on how long state verification can run

Also, state verification is now running on the read replica if there is one configured.

Why

We have noticed that State verification is taking more than 30 minutes to run. During this time state verification has an open repeatable read transaction. Maintaining multiple of these transactions for such an extended period of time may have a negative effect on ingestion performance. Not to mention that multiple state verification runs for the same checkpoint ledger is wasteful because it's performing the same check multiple times.

Also, if state verification is having a negative impact on ingestion performance it is important that we have configuration knobs to limit how frequently it is run. e.g. instead of running state verification on every checkpoint we can instead run it on every 288th checkpoint (approximately once per day).

Given that state verification is a read only operation we can also move it to run on the read replica which will minimize the impact of the rw database used by ingestion.

Known limitations

We need to verify that running state verification on read replicas will not significantly impact the performance of request serving horizon instances which also use the read replica db.

[sreuland]

this is a significant feature addition, seems like qualifies for an issue/ticket to provide visibility of the effort on Stability & Performance board, maybe we can retro-point it also then, just so the effort gets included in sprint velocity?

[sreuland]

nice work!

[tamirms]

After some tests in staging I discovered that state verification does not work on the read replica because it keeps failing with the following error:

cancelling statement due to conflict with recovery

So, I have removed that feature from this PR. Apart from that everything looked good