support/db: Add timeout on horizon db ingestion statements

[tamirms]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Add timeout on horizon db select and exec operations.

Why

Currently there is no timeout on any of the horizon DB operations. That means a postgres query which hangs indefinitely could block ingestion from progressing.

Known limitations

[N/A]

[bartekn]

I wonder if we still need it since we have #2560 now? AFAIR, adding timeout was suppose to fix issues when ingestion is starved by HTTP server but it shouldn't be a problem any longer?

[ire-and-curses]

I wonder if we still need it since we have #2560 now? AFAIR, adding timeout was suppose to fix issues when ingestion is starved by HTTP server but it shouldn't be a problem any longer?

Is there any downside to a timeout? Having the possibility of these hanging forever seems worse?

[tamirms]

I'm ok with closing the PR if that is your preference @bartekn

[bartekn]

Is there any downside to a timeout? Having the possibility of these hanging forever seems worse?

It's not a problem if it's doing nothing (ex. waiting for a free connection from a pool as before #2560) but if the query actually executes but a little longer I think it's actually better to wait. With timeout it's possible that ingestion will fall into an infinite loop (slow query, timeout, slow query, timeout, ...).

I'm ok with closing the PR if that is your preference @bartekn

I think we should leave the timeout code in db.Session, just remove it from ingest. And maybe let's add other stellar/go members to review because it's a shared package.

[tamirms]

Going to close this PR since there is no immediate use case anymore