services/horizon/internal/db2/history: Mitigate deadlock when running reingest range

[tamirms]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Fixes #2370

• Remove database functions which were previously used by legacy ingestion but now are unused
• Sort assets before inserting rows into history_assets
• Sort accounts before inserting rows into history_accounts
• Ingest each ledger in reingest range in separate transaction

Why

Running horizon db reingest range multiple times in parallel sometimes causes a deadlock in Postgres where multiple connections inserting into history_accounts are blocked trying to acquire a ShareLock.

ShareLocks are row locks that are taken on each row as it is written by the transaction. ShareLock deadlocks occurs when two separate transactions try to write the same rows in opposite order.

That is why assets and accounts are sorted before insertion.

Another change in this PR which should mitigate against deadlocks is keeping transactions short lived. Previously, we reingested the entire range in just one transaction. If you have multiple reingesting processes which are running in parallel over very long ranges that increases the chances there will be a deadlock.

Known limitations

[N/A]

[bartekn]

All looks good to me. I think a single thing is worth changing. By removing createTx param in ingestRange and moving the code responsible for begin/commit/rollback here we solve two potential issues:

• Incorrect usage of ingestRange - if a DB transaction is not started in a caller function we should never insert data without a DB transaction because in case of errors the ledger data will be inserted partially,
• It's hard to track DB transaction flow - currently DB transaction can be started in two different places and both functions need to make sure they know about one another, it was similar in FSM before where some states expected to be in a DB tx and some didn't.

[abuiles]

👍