Skip to content

Commit

Permalink
Merge #1380: Simplified EsploraExt API
Browse files Browse the repository at this point in the history 
96a9aa6 feat(chain): refactor `merge_chains` (志宇)
2f22987 chore(chain): fix comment (志宇)
daf588f feat(chain): optimize `merge_chains` (志宇)
77d3595 feat(chain)!: rm `local_chain::Update` (志宇)
1269b06 test(chain): fix incorrect test case (志宇)
72fe65b feat(esplora)!: simplify chain update logic (志宇)
eded1a7 feat(chain): introduce `CheckPoint::insert` (志宇)
519cd75 test(esplora): move esplora tests into src files (志宇)
a6e613e test(esplora): add `test_finalize_chain_update` (志宇)
494d253 feat(testenv): add `genesis_hash` method (志宇)
886d72e chore(chain)!: rm `missing_heights` and `missing_heights_from` methods (志宇)
bd62aa0 feat(esplora)!: remove `EsploraExt::update_local_chain` (志宇)
1e99793 feat(testenv): add `make_checkpoint_tip` (志宇)

Pull request description:

  Fixes #1354

  ### Description

  Built on top of both #1369 and #1373, we simplify the `EsploraExt` API by removing the `update_local_chain` method and having `full_scan` and `sync` update the local chain in the same call. The `full_scan` and `sync` methods now takes in an additional input (`local_tip`) which provides us with the view of the `LocalChain` before the update. These methods now return structs `FullScanUpdate` and `SyncUpdate`.

  The examples are updated to use this new API. `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` are no longer needed, therefore they are removed.

  Additionally, we used this opportunity to simplify the logic which updates `LocalChain`. We got rid of the `local_chain::Update` struct (which contained the update `CheckPoint` tip and a `bool` which signaled whether we want to introduce blocks below point of agreement). It turns out we can use something like `CheckPoint::insert` so the chain source can craft an update based on the old tip. This way, we can make better use of `merge_chains`' optimization that compares the `Arc` pointers of the local and update chain (before we were crafting the update chain NOT based on top of the previous local chain). With this, we no longer need the `Update::introduce_older_block` field since the logic will naturally break when we reach a matching `Arc` pointer.

  ### Notes to the reviewers

  * Obtaining the `LocalChain`'s update now happens within `EsploraExt::full_scan` and `EsploraExt::sync`. Creating the `LocalChain` update is now split into two methods (`fetch_latest_blocks` and `chain_update`) that are called before and after fetching transactions and anchors.
  * We need to duplicate code for `bdk_esplora`. One for blocking and one for async.

  ### Changelog notice

  * Changed `EsploraExt` API so that sync only requires one round of fetching data. The `local_chain_update` method is removed and the `local_tip` parameter is added to the `full_scan` and `sync` methods.
  * Removed `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` methods.
  * Introduced `CheckPoint::insert` which allows convenient checkpoint-insertion. This is intended for use by chain-sources when crafting an update.
  * Refactored `merge_chains` to also return the resultant `CheckPoint` tip.
  * Optimized the update `LocalChain` logic - use the update `CheckPoint` as the new `CheckPoint` tip when possible.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [x] I've added tests for the new feature
  * [x] I've added docs for the new feature

ACKs for top commit:
  LLFourn:
    ACK 96a9aa6

Tree-SHA512: 3d4f2eab08a1fe94eb578c594126e99679f72e231680b2edd4bfb018ba1d998ca123b07acb2d19c644d5887fc36b8e42badba91cd09853df421ded04de45bf69
  • Loading branch information
@evanlinjin
evanlinjin committed Apr 22, 2024
2 parents 9800f8d + 96a9aa6 commit 8e73998
Show file tree
Hide file tree
Showing 19 changed files with 1,606 additions and 1,117 deletions.
2 changes: 1 addition & 1 deletion crates/bdk/src/wallet/mod.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ pub struct Update {
/// Update for the wallet's internal [`LocalChain`].
///
/// [`LocalChain`]: local_chain::LocalChain
pub chain: Option<local_chain::Update>,
pub chain: Option<CheckPoint>,
}

/// The changes made to a wallet by applying an [`Update`].
Expand Down
25 changes: 6 additions & 19 deletions crates/bitcoind_rpc/tests/test_emitter.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ use bdk_bitcoind_rpc::Emitter;
use bdk_chain::{
bitcoin::{Address, Amount, Txid},
keychain::Balance,
local_chain::{self, CheckPoint, LocalChain},
local_chain::{CheckPoint, LocalChain},
Append, BlockId, IndexedTxGraph, SpkTxOutIndex,
};
use bdk_testenv::TestEnv;
Expand Down Expand Up @@ -47,10 +47,7 @@ pub fn test_sync_local_chain() -> anyhow::Result<()> {
);

assert_eq!(
local_chain.apply_update(local_chain::Update {
tip: emission.checkpoint,
introduce_older_blocks: false,
})?,
local_chain.apply_update(emission.checkpoint,)?,
BTreeMap::from([(height, Some(hash))]),
"chain update changeset is unexpected",
);
Expand Down Expand Up @@ -95,10 +92,7 @@ pub fn test_sync_local_chain() -> anyhow::Result<()> {
);

assert_eq!(
local_chain.apply_update(local_chain::Update {
tip: emission.checkpoint,
introduce_older_blocks: false,
})?,
local_chain.apply_update(emission.checkpoint,)?,
if exp_height == exp_hashes.len() - reorged_blocks.len() {
core::iter::once((height, Some(hash)))
.chain((height + 1..exp_hashes.len() as u32).map(|h| (h, None)))
Expand Down Expand Up @@ -168,10 +162,7 @@ fn test_into_tx_graph() -> anyhow::Result<()> {

while let Some(emission) = emitter.next_block()? {
let height = emission.block_height();
let _ = chain.apply_update(local_chain::Update {
tip: emission.checkpoint,
introduce_older_blocks: false,
})?;
let _ = chain.apply_update(emission.checkpoint)?;
let indexed_additions = indexed_tx_graph.apply_block_relevant(&emission.block, height);
assert!(indexed_additions.is_empty());
}
Expand Down Expand Up @@ -232,10 +223,7 @@ fn test_into_tx_graph() -> anyhow::Result<()> {
{
let emission = emitter.next_block()?.expect("must get mined block");
let height = emission.block_height();
let _ = chain.apply_update(local_chain::Update {
tip: emission.checkpoint,
introduce_older_blocks: false,
})?;
let _ = chain.apply_update(emission.checkpoint)?;
let indexed_additions = indexed_tx_graph.apply_block_relevant(&emission.block, height);
assert!(indexed_additions.graph.txs.is_empty());
assert!(indexed_additions.graph.txouts.is_empty());
Expand Down Expand Up @@ -294,8 +282,7 @@ fn process_block(
block: Block,
block_height: u32,
) -> anyhow::Result<()> {
recv_chain
.apply_update(CheckPoint::from_header(&block.header, block_height).into_update(false))?;
recv_chain.apply_update(CheckPoint::from_header(&block.header, block_height))?;
let _ = recv_graph.apply_block(block, block_height);
Ok(())
}
Expand Down
225 changes: 120 additions & 105 deletions crates/chain/src/local_chain.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,16 +96,6 @@ impl CheckPoint {
.expect("must construct checkpoint")
}

/// Convenience method to convert the [`CheckPoint`] into an [`Update`].
///
/// For more information, refer to [`Update`].
pub fn into_update(self, introduce_older_blocks: bool) -> Update {
Update {
tip: self,
introduce_older_blocks,
}
}

/// Puts another checkpoint onto the linked list representing the blockchain.
///
/// Returns an `Err(self)` if the block you are pushing on is not at a greater height that the one you
Expand Down Expand Up @@ -187,6 +177,82 @@ impl CheckPoint {
core::ops::Bound::Unbounded => true,
})
}

/// Inserts `block_id` at its height within the chain.
///
/// The effect of `insert` depends on whether a height already exists. If it doesn't the
/// `block_id` we inserted and all pre-existing blocks higher than it will be re-inserted after
/// it. If the height already existed and has a conflicting block hash then it will be purged
/// along with all block followin it. The returned chain will have a tip of the `block_id`
/// passed in. Of course, if the `block_id` was already present then this just returns `self`.
#[must_use]
pub fn insert(self, block_id: BlockId) -> Self {
assert_ne!(block_id.height, 0, "cannot insert the genesis block");

let mut cp = self.clone();
let mut tail = vec![];
let base = loop {
if cp.height() == block_id.height {
if cp.hash() == block_id.hash {
return self;
}
// if we have a conflict we just return the inserted block because the tail is by
// implication invalid.
tail = vec![];
break cp.prev().expect("can't be called on genesis block");
}

if cp.height() < block_id.height {
break cp;
}

tail.push(cp.block_id());
cp = cp.prev().expect("will break before genesis block");
};

base.extend(core::iter::once(block_id).chain(tail.into_iter().rev()))
.expect("tail is in order")
}

/// Apply `changeset` to the checkpoint.
fn apply_changeset(mut self, changeset: &ChangeSet) -> Result<CheckPoint, MissingGenesisError> {
if let Some(start_height) = changeset.keys().next().cloned() {
// changes after point of agreement
let mut extension = BTreeMap::default();
// point of agreement
let mut base: Option<CheckPoint> = None;

for cp in self.iter() {
if cp.height() >= start_height {
extension.insert(cp.height(), cp.hash());
} else {
base = Some(cp);
break;
}
}

for (&height, &hash) in changeset {
match hash {
Some(hash) => {
extension.insert(height, hash);
}
None => {
extension.remove(&height);
}
};
}

let new_tip = match base {
Some(base) => base
.extend(extension.into_iter().map(BlockId::from))
.expect("extension is strictly greater than base"),
None => LocalChain::from_blocks(extension)?.tip(),
};
self = new_tip;
}

Ok(self)
}
}

/// Iterates over checkpoints backwards.
Expand Down Expand Up @@ -215,31 +281,6 @@ impl IntoIterator for CheckPoint {
}
}

/// Used to update [`LocalChain`].
///
/// This is used as input for [`LocalChain::apply_update`]. It contains the update's chain `tip` and
/// a flag `introduce_older_blocks` which signals whether this update intends to introduce missing
/// blocks to the original chain.
///
/// Block-by-block syncing mechanisms would typically create updates that builds upon the previous
/// tip. In this case, `introduce_older_blocks` would be `false`.
///
/// Script-pubkey based syncing mechanisms may not introduce transactions in a chronological order
/// so some updates require introducing older blocks (to anchor older transactions). For
/// script-pubkey based syncing, `introduce_older_blocks` would typically be `true`.
#[derive(Debug, Clone, PartialEq)]
pub struct Update {
/// The update chain's new tip.
pub tip: CheckPoint,

/// Whether the update allows for introducing older blocks.
///
/// Refer to [struct-level documentation] for more.
///
/// [struct-level documentation]: Update
pub introduce_older_blocks: bool,
}

/// This is a local implementation of [`ChainOracle`].
#[derive(Debug, Clone, PartialEq)]
pub struct LocalChain {
Expand Down Expand Up @@ -347,36 +388,22 @@ impl LocalChain {

/// Applies the given `update` to the chain.
///
/// The method returns [`ChangeSet`] on success. This represents the applied changes to `self`.
/// The method returns [`ChangeSet`] on success. This represents the changes applied to `self`.
///
/// There must be no ambiguity about which of the existing chain's blocks are still valid and
/// which are now invalid. That is, the new chain must implicitly connect to a definite block in
/// the existing chain and invalidate the block after it (if it exists) by including a block at
/// the same height but with a different hash to explicitly exclude it as a connection point.
///
/// Additionally, an empty chain can be updated with any chain, and a chain with a single block
/// can have it's block invalidated by an update chain with a block at the same height but
/// different hash.
///
/// # Errors
///
/// An error will occur if the update does not correctly connect with `self`.
///
/// Refer to [`Update`] for more about the update struct.
///
/// [module-level documentation]: crate::local_chain
pub fn apply_update(&mut self, update: Update) -> Result<ChangeSet, CannotConnectError> {
let changeset = merge_chains(
self.tip.clone(),
update.tip.clone(),
update.introduce_older_blocks,
)?;
// `._check_index_is_consistent_with_tip` and `._check_changeset_is_applied` is called in
// `.apply_changeset`
self.apply_changeset(&changeset)
.map_err(|_| CannotConnectError {
try_include_height: 0,
})?;
pub fn apply_update(&mut self, update: CheckPoint) -> Result<ChangeSet, CannotConnectError> {
let (new_tip, changeset) = merge_chains(self.tip.clone(), update)?;
self.tip = new_tip;
self._check_changeset_is_applied(&changeset);
Ok(changeset)
}

Expand Down Expand Up @@ -428,11 +455,8 @@ impl LocalChain {
conn => Some(conn),
};

let update = Update {
tip: CheckPoint::from_block_ids([conn, prev, Some(this)].into_iter().flatten())
.expect("block ids must be in order"),
introduce_older_blocks: false,
};
let update = CheckPoint::from_block_ids([conn, prev, Some(this)].into_iter().flatten())
.expect("block ids must be in order");

self.apply_update(update)
.map_err(ApplyHeaderError::CannotConnect)
Expand Down Expand Up @@ -471,43 +495,10 @@ impl LocalChain {

/// Apply the given `changeset`.
pub fn apply_changeset(&mut self, changeset: &ChangeSet) -> Result<(), MissingGenesisError> {
if let Some(start_height) = changeset.keys().next().cloned() {
// changes after point of agreement
let mut extension = BTreeMap::default();
// point of agreement
let mut base: Option<CheckPoint> = None;

for cp in self.iter_checkpoints() {
if cp.height() >= start_height {
extension.insert(cp.height(), cp.hash());
} else {
base = Some(cp);
break;
}
}

for (&height, &hash) in changeset {
match hash {
Some(hash) => {
extension.insert(height, hash);
}
None => {
extension.remove(&height);
}
};
}

let new_tip = match base {
Some(base) => base
.extend(extension.into_iter().map(BlockId::from))
.expect("extension is strictly greater than base"),
None => LocalChain::from_blocks(extension)?.tip(),
};
self.tip = new_tip;

debug_assert!(self._check_changeset_is_applied(changeset));
}

let old_tip = self.tip.clone();
let new_tip = old_tip.apply_changeset(changeset)?;
self.tip = new_tip;
debug_assert!(self._check_changeset_is_applied(changeset));
Ok(())
}

Expand Down Expand Up @@ -730,14 +721,17 @@ impl core::fmt::Display for ApplyHeaderError {
#[cfg(feature = "std")]
impl std::error::Error for ApplyHeaderError {}

/// Applies `update_tip` onto `original_tip`.
///
/// On success, a tuple is returned `(changeset, can_replace)`. If `can_replace` is true, then the
/// `update_tip` can replace the `original_tip`.
fn merge_chains(
original_tip: CheckPoint,
update_tip: CheckPoint,
introduce_older_blocks: bool,
) -> Result<ChangeSet, CannotConnectError> {
) -> Result<(CheckPoint, ChangeSet), CannotConnectError> {
let mut changeset = ChangeSet::default();
let mut orig = original_tip.into_iter();
let mut update = update_tip.into_iter();
let mut orig = original_tip.iter();
let mut update = update_tip.iter();
let mut curr_orig = None;
let mut curr_update = None;
let mut prev_orig: Option<CheckPoint> = None;
Expand All @@ -746,6 +740,12 @@ fn merge_chains(
let mut prev_orig_was_invalidated = false;
let mut potentially_invalidated_heights = vec![];

// If we can, we want to return the update tip as the new tip because this allows checkpoints
// in multiple locations to keep the same `Arc` pointers when they are being updated from each
// other using this function. We can do this as long as long as the update contains every
// block's height of the original chain.
let mut is_update_height_superset_of_original = true;

// To find the difference between the new chain and the original we iterate over both of them
// from the tip backwards in tandem. We always dealing with the highest one from either chain
// first and move to the next highest. The crucial logic is applied when they have blocks at the
Expand All @@ -771,6 +771,8 @@ fn merge_chains(
prev_orig_was_invalidated = false;
prev_orig = curr_orig.take();

is_update_height_superset_of_original = false;

// OPTIMIZATION: we have run out of update blocks so we don't need to continue
// iterating because there's no possibility of adding anything to changeset.
if u.is_none() {
Expand All @@ -793,12 +795,20 @@ fn merge_chains(
}
point_of_agreement_found = true;
prev_orig_was_invalidated = false;
// OPTIMIZATION 1 -- If we know that older blocks cannot be introduced without
// invalidation, we can break after finding the point of agreement.
// OPTIMIZATION 2 -- if we have the same underlying pointer at this point, we
// can guarantee that no older blocks are introduced.
if !introduce_older_blocks || Arc::as_ptr(&o.0) == Arc::as_ptr(&u.0) {
return Ok(changeset);
if Arc::as_ptr(&o.0) == Arc::as_ptr(&u.0) {
if is_update_height_superset_of_original {
return Ok((update_tip, changeset));
} else {
let new_tip =
original_tip.apply_changeset(&changeset).map_err(|_| {
CannotConnectError {
try_include_height: 0,
}
})?;
return Ok((new_tip, changeset));
}
}
} else {
// We have an invalidation height so we set the height to the updated hash and
Expand Down Expand Up @@ -832,5 +842,10 @@ fn merge_chains(
}
}

Ok(changeset)
let new_tip = original_tip
.apply_changeset(&changeset)
.map_err(|_| CannotConnectError {
try_include_height: 0,
})?;
Ok((new_tip, changeset))
}
Loading

0 comments on commit 8e73998

Please sign in to comment.