services/horizon: Add middleware for checking read-replica lag

[bartekn]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

• Adds a new RO_DATABASE_URL that allows setting a connection to a read-replica DB.
• When the flag is set:
  • A new middleware checking if the read-replica is behind a primary DB will be enabled. If such check fails three times, the stale_history error will be returned. The middleware will sleep 20ms and 40ms on the first and second failure.
  • All DB queries will be sent to read-replica (except the one that checks the primary sequence number in the middleware above).
• Adds a new horizon_http_replica_lag_errors_count metric to be able to track number of error responses due to replica lag.

Why

Horizon should provide clients with strongly consistent responses when using read-replicas. Specifically, for consecutive HTTP requests: A and B, response for B must always return data at ledger sequence higher or equal to the response for A.

Known limitations

In case of a higher replica lag, the middleware may increase HTTP error rate.

[tamirms]

would it make sense to have metrics on the distribution of requests which satisfy this condition on attempt 1, 2, 3, and 4?