Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest/ledgerbackend: Implement db backed ledger store for the captive core backend #3203

Merged
merged 4 commits into from
Nov 13, 2020

Conversation

tamirms
Copy link
Contributor

@tamirms tamirms commented Nov 10, 2020

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with name of package that is most changed in the PR, ex.
    services/friendbot, or all or doc if the changes are broad or impact many
    packages.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated any docs (developer docs, .md
    files, etc... affected by this change). Take a look in the docs folder for a given service,
    like this one.

Release planning

  • I've updated the relevant CHANGELOG (here for Horizon) if
    needed with deprecations, added features, breaking changes, and DB schema changes.
  • I've decided if this PR requires a new major/minor version according to
    semver, or if it's mainly a patch change. The PR is targeted at the next
    release branch if it's not a patch change.

What

When running stellar-core run --start-at-ledger <sequence> you need to provide an additional --start-at-hash parameter. We were only able to use --start-at-ledger values which are checkpoint sequences because we can obtain the ledger hash for any checkpoint from the history archives.

However, we do have ledger hashes for the ledgers Horizon has previously ingested. This PR uses the horizon database of ingested ledgers as an additional source for obtaining ledger hashes.

Why

The history archives cannot be fully trusted because there's always the possibility that someone can compromise the history archives either by compromising the S3 bucket which stores the archives or compromising one of the layers above the s3 bucket. When we provide a corrupt ledger hash to stellar core as --start-at-hash parameter, stellar core will download the ledger chain and validate all the hashes from consensus back to the start hash. If the start hash is not valid, stellar-core will exit with an error before streaming any ledgers. This means that ingestion will be blocked until the history archives are fixed.

Since stellar core will always verify the ledger chain before emitting any ledgers, we can assume that any ledger hashes ingested into the horizon database are correct. So, in the scenario where the history archives are corrupt, we can avoid blocking ingestion by using ledger hashes found in the Horizon database.

Close #3172

@cla-bot cla-bot bot added the cla: yes label Nov 10, 2020
@tamirms tamirms requested a review from a team November 10, 2020 12:59
ingest/ledgerbackend/captive_core_backend.go Outdated Show resolved Hide resolved
ingest/ledgerbackend/ledger_store.go Outdated Show resolved Hide resolved
ingest/ledgerbackend/ledger_store.go Outdated Show resolved Hide resolved
ingest/ledgerbackend/ledger_store.go Outdated Show resolved Hide resolved
ingest/ledgerbackend/ledger_store.go Outdated Show resolved Hide resolved
ingest/ledgerbackend/captive_core_backend.go Outdated Show resolved Hide resolved
@tamirms
Copy link
Contributor Author

tamirms commented Nov 13, 2020

@bartekn can you take another look?

Copy link
Contributor

@bartekn bartekn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tamirms tamirms merged commit 21e076a into stellar:master Nov 13, 2020
@tamirms tamirms deleted the ledger-store branch November 13, 2020 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove trust on history archives in captive core
2 participants