Merge #926: Introduce redesigned bdk_chain structures

b799a57 [bdk_chain_redesign] Add tests for `IndexedTxGraph` with `LocalChain` (rajarshimaitra)
8cd0328 [bdk_chain_redesign] Implement `OwnedIndexer` for indexers (rajarshimaitra)
911af34 [bdk_chain_redesign] Fix calculation bugs. (rajarshimaitra)
e536307 [bdk_chain_redesign] Fix `tx_graph::Additions::append` logic (志宇)
f101dde [bdk_chain_redesign] Fix `tx_graph::Additions::append` logic (志宇)
1b15264 [bdk_chain_redesign] Change `insert_relevant_txs` method (志宇)
ecc74ce [bdk_chain_redesign] Docs for `is_mature` and `is_confirmed_and_spendable` (志宇)
ac336aa [bdk_chain_redesign] Make `insert_relevant_txs` topologically-agnostic (志宇)
165b874 [bdk_chain_redesign] Add test for `insert_relevant_txs` (志宇)
f3e7b67 [bdk_chain_redesign] Various tweaks and fixes (志宇)
03c1283 [bdk_chain_redesign] Revert changes to `SparseChain` (志宇)
34a7bf5 [bdk_chain_redesign] Rm unnecessary code and premature optimisation (志宇)
6c49570 [bdk_chain_redesign] Rm `HashSet` from `TxGraph::relevant_heights` (志宇)
1003fe2 [bdk_chain_redesign] Test `LocalChain` (志宇)
7175a82 [bdk_chain_redesign] Add tests for `TxGraph::relevant_heights` (志宇)
8e36a2e [bdk_chain_redesign] Remove incomplete logic (志宇)
81436fc [bdk_chain_redesign] Fix `Anchor` definition + docs (志宇)
001efdd Include tests for new updates of TxGraph (rajarshimaitra)
10ab77c [bdk_chain_redesign] MOVE `TxIndex` into `indexed_chain_graph.rs` (志宇)
7d92337 [bdk_chain_redesign] Remove `IndexedTxGraph::last_height` (志宇)
a7fbe0a [bdk_chain_redesign] Documentation improvements (志宇)
ee1060f [bdk_chain_redesign] Simplify `LocalChain` (志宇)
611d2e3 [bdk_chain_redesign] Consistent `ChainOracle` (志宇)
bff80ec [bdk_chain_redesign] Improve `BlockAnchor` docs (志宇)
24cd8c5 [bdk_chain_redesign] More tweaks and renamings (志宇)
ddd5e95 [bdk_chain_redesign] Modify signature of `TxIndex` (志宇)
da4cef0 [bdk_chain_redesign] Introduce `Append` trait for additions (志宇)
89cfa4d [bdk_chain_redesign] Better names, comments and generic bounds (志宇)
6e59dce [bdk_chain_redesign] `chain_oracle::Cache` (志宇)
a7eaebb [bdk_chain_redesign] Add serde support for `IndexedAdditions` (志宇)
c09cd2a [bdk_chain_redesign] Added methods to `LocalChain` (志宇)
7810059 [bdk_chain_redesign] `TxGraph` tweaks (志宇)
a63ffe9 [bdk_chain_redesign] Simplify `TxIndex` (志宇)
a1172de [bdk_chain_redesign] Revert some API changes (志宇)
8c90617 [bdk_chain_redesign] Make default anchor for `TxGraph` as `()` (志宇)
468701a [bdk_chain_redesign] Initial work on `LocalChain`. (志宇)
34d0277 [bdk_chain_redesign] Rm anchor type param for structs that don't use it (志宇)
3440a05 [bdk_chain_redesign] Add docs (志宇)
236c50f [bdk_chain_redesign] `IndexedTxGraph` keeps track of the last synced height (志宇)
e902c10 [bdk_chain_redesign] Fix `apply_additions` logic for `IndexedTxGraph`. (志宇)
313965d [bdk_chain_redesign] `mut_index` should be `index_mut` (志宇)
db7883d [bdk_chain_redesign] Add balance methods to `IndexedTxGraph` (志宇)
d0a2aa8 [bdk_chain_redesign] Add `apply_additions` to `IndexedTxGraph` (志宇)
6cbb18d [bdk_chain_redesign] MOVE: `IndexedTxGraph` into submodule (志宇)
784cd34 [bdk_chain_redesign] List chain data methods can be try/non-try (志宇)
43b648f [bdk_chain_redesign] Add `..in_chain` methods (志宇)
61a8606 [bdk_chain_redesign] Introduce `ChainOracle` and `TxIndex` traits (志宇)
5ae5fe3 [bdk_chain_redesign] Introduce `BlockAnchor` trait (志宇)

Pull request description:

  ### Description

  This is part of #895

  The initial `bdk_chain` structures allowed updating to be done without blocking IO (alongside many other benefits). However, the requirement to have transactions "perfectly positioned" in our `SparseChain` increased complexity of the syncing API. Updates needed to be meticulously crafted to properly connect with the original `SparseChain`. Additionally, it forced us to keep a local copy of the "best chain" data (which may not always be needed depending on the chain source).

  The redesigned structures, as introduced by this PR, fixes these shortcomings.

  1. Instead of `SparseChain`, we introduce the ability to `Anchor` a transaction to multiple blocks that may or may not be in the same chain history. We expand `TxGraph` to records these anchors (while still maintaining the *monotone* nature of `TxGraph`). When updating our new `TxGraph`, we don't need to complicated *are-we-connected* checks that we need for `SparseChain`.
  2. The chain source, in combination with our new`TxGraph` is all that we need to determine the "chain position" of a transaction. The chain source only needs to implement a single trait `ChainOracle`. This typically does not need to be updated (as it is remote), although we can have a special `ChainOracle` implementation that is stored locally. This only needs block height and hash information, reducing the scope of *non-monotine* structures that need to be updated.

  **What is done:**

  * `TxGraph` includes anchors (ids of blocks that the tx is seen in) and last-seem unix timestamp (for determining which non-confirmed tx we should consider as part of "best chain" if there are conflicts). This structure continues to be fully "monotone"; we can introduce data in any manner and not risk resulting in an inconsistent state.
  * `LocalChain` includes the "checkpoint" logic of `SparseChain` but removes the `txid` data. `LocalChain` implements the `ChainOracle` trait. Any blockchain-source can also implement the `ChainOracle` trait.
  * `IndexedTxGraph` is introduced and contains two fields; an internal `TxGraph` struct and a `TxIndex` implementation. These two fields will be updated atomically and can replace the functionality of `keychain::Tracker`.

  **What is in-progress:**

  * ~@evanlinjin: The `TxIndex` trait should not have `is_{}_relevant` methods as we cannot guarantee this across all transaction indexes. We should introduce extension traits for these (#926 (comment)
  * ~@evanlinjin: `BlockAnchor` should be defined as "if this block is in chain, then this tx must be in chain". Therefore, the anchor does not provide us with the exact confirmation height of the tx. We need to introduce an extension trait `ExactConfirmationHeightAnchor` for certain operations (#926 (comment)

  **What will be done external to this PR:**

  * @rajarshimaitra: Persisting `indexed_tx_graph::Additions` (#937).
  * @notmandatory: Update examples to use redesigned structures.
  * Update `bdk::Wallet` to use the redesigned structures.

  ### Changelog notice

  * Initial implementation of the `bdk_chain` redesign (as mentioned in #895). Currently, only the `bdk_chain` core structures are implemented. Persistence and updates to the current examples and `bdk::Wallet` will be done in separate PRs.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [x] I've added tests for the new feature
  * [x] I've added docs for the new feature

ACKs for top commit:
  rajarshimaitra:
    ACK b799a57
  danielabrozzoni:
    Partial ACK b799a57 - good job! I haven't looked at the tests or at the methods implementations in depth, but I have reviewed the architecture and it looks good. I have a few non-blocking questions.

Tree-SHA512: 8c386354cbd02f0701b5134991b65d206a54d19a2e78ab204e6ff1fa78a18f16299051bc0bf4ff4d2f5a0adab9b15658fa53cd0de2ca16969f4bf6a485225082

crates/chain/src/chain_data.rs

@@ -2,9 +2,29 @@ use bitcoin::{hashes::Hash, BlockHash, OutPoint, TxOut, Txid};
 
 use crate::{
     sparse_chain::{self, ChainPosition},
-    COINBASE_MATURITY,
+    Anchor, COINBASE_MATURITY,
 };
 
+/// Represents an observation of some chain data.
+///
+/// The generic `A` should be a [`Anchor`] implementation.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, core::hash::Hash)]
+pub enum ObservedAs<A> {
+    /// The chain data is seen as confirmed, and in anchored by `A`.
+    Confirmed(A),
+    /// The chain data is seen in mempool at this given timestamp.
+    Unconfirmed(u64),
+}
+
+impl<A: Clone> ObservedAs<&A> {
+    pub fn cloned(self) -> ObservedAs<A> {
+        match self {
+            ObservedAs::Confirmed(a) => ObservedAs::Confirmed(a.clone()),
+            ObservedAs::Unconfirmed(last_seen) => ObservedAs::Unconfirmed(last_seen),
+        }
+    }
+}
+
 /// Represents the height at which a transaction is confirmed.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
 #[cfg_attr(
@@ -118,7 +138,7 @@ impl ConfirmationTime {
 }
 
 /// A reference to a block in the canonical chain.
-#[derive(Debug, Clone, PartialEq, Eq, Copy, PartialOrd, Ord)]
+#[derive(Debug, Clone, PartialEq, Eq, Copy, PartialOrd, Ord, core::hash::Hash)]
 #[cfg_attr(
     feature = "serde",
     derive(serde::Deserialize, serde::Serialize),
@@ -140,6 +160,12 @@ impl Default for BlockId {
     }
 }
 
+impl Anchor for BlockId {
+    fn anchor_block(&self) -> BlockId {
+        *self
+    }
+}
+
 impl From<(u32, BlockHash)> for BlockId {
     fn from((height, hash): (u32, BlockHash)) -> Self {
         Self { height, hash }
@@ -162,21 +188,21 @@ impl From<(&u32, &BlockHash)> for BlockId {
 }
 
 /// A `TxOut` with as much data as we can retrieve about it
-#[derive(Debug, Clone, PartialEq)]
-pub struct FullTxOut<I> {
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
+pub struct FullTxOut<P> {
     /// The location of the `TxOut`.
     pub outpoint: OutPoint,
     /// The `TxOut`.
     pub txout: TxOut,
     /// The position of the transaction in `outpoint` in the overall chain.
-    pub chain_position: I,
+    pub chain_position: P,
     /// The txid and chain position of the transaction (if any) that has spent this output.
-    pub spent_by: Option<(I, Txid)>,
+    pub spent_by: Option<(P, Txid)>,
     /// Whether this output is on a coinbase transaction.
     pub is_on_coinbase: bool,
 }
 
-impl<I: ChainPosition> FullTxOut<I> {
+impl<P: ChainPosition> FullTxOut<P> {
     /// Whether the utxo is/was/will be spendable at `height`.
     ///
     /// It is spendable if it is not an immature coinbase output and no spending tx has been
@@ -215,4 +241,69 @@ impl<I: ChainPosition> FullTxOut<I> {
     }
 }
 
-// TODO: make test
+impl<A: Anchor> FullTxOut<ObservedAs<A>> {
+    /// Whether the `txout` is considered mature.
+    ///
+    /// This is the alternative version of [`is_mature`] which depends on `chain_position` being a
+    /// [`ObservedAs<A>`] where `A` implements [`Anchor`].
+    ///
+    /// Depending on the implementation of [`confirmation_height_upper_bound`] in [`Anchor`], this
+    /// method may return false-negatives. In other words, interpretted confirmation count may be
+    /// less than the actual value.
+    ///
+    /// [`is_mature`]: Self::is_mature
+    /// [`confirmation_height_upper_bound`]: Anchor::confirmation_height_upper_bound
+    pub fn is_mature(&self, tip: u32) -> bool {
+        if self.is_on_coinbase {
+            let tx_height = match &self.chain_position {
+                ObservedAs::Confirmed(anchor) => anchor.confirmation_height_upper_bound(),
+                ObservedAs::Unconfirmed(_) => {
+                    debug_assert!(false, "coinbase tx can never be unconfirmed");
+                    return false;
+                }
+            };
+            let age = tip.saturating_sub(tx_height);
+            if age + 1 < COINBASE_MATURITY {
+                return false;
+            }
+        }
+
+        true
+    }
+
+    /// Whether the utxo is/was/will be spendable with chain `tip`.
+    ///
+    /// This method does not take into account the locktime.
+    ///
+    /// This is the alternative version of [`is_spendable_at`] which depends on `chain_position`
+    /// being a [`ObservedAs<A>`] where `A` implements [`Anchor`].
+    ///
+    /// Depending on the implementation of [`confirmation_height_upper_bound`] in [`Anchor`], this
+    /// method may return false-negatives. In other words, interpretted confirmation count may be
+    /// less than the actual value.
+    ///
+    /// [`is_spendable_at`]: Self::is_spendable_at
+    /// [`confirmation_height_upper_bound`]: Anchor::confirmation_height_upper_bound
+    pub fn is_confirmed_and_spendable(&self, tip: u32) -> bool {
+        if !self.is_mature(tip) {
+            return false;
+        }
+
+        let confirmation_height = match &self.chain_position {
+            ObservedAs::Confirmed(anchor) => anchor.confirmation_height_upper_bound(),
+            ObservedAs::Unconfirmed(_) => return false,
+        };
+        if confirmation_height > tip {
+            return false;
+        }
+
+        // if the spending tx is confirmed within tip height, the txout is no longer spendable
+        if let Some((ObservedAs::Confirmed(spending_anchor), _)) = &self.spent_by {
+            if spending_anchor.anchor_block().height <= tip {
+                return false;
+            }
+        }
+
+        true
+    }
+}

crates/chain/src/chain_graph.rs

@@ -3,7 +3,7 @@ use crate::{
     collections::HashSet,
     sparse_chain::{self, ChainPosition, SparseChain},
     tx_graph::{self, TxGraph},
-    BlockId, ForEachTxOut, FullTxOut, TxHeight,
+    Append, BlockId, ForEachTxOut, FullTxOut, TxHeight,
 };
 use alloc::{string::ToString, vec::Vec};
 use bitcoin::{OutPoint, Transaction, TxOut, Txid};

crates/chain/src/chain_oracle.rs

@@ -0,0 +1,21 @@
+use crate::BlockId;
+
+/// Represents a service that tracks the blockchain.
+///
+/// The main method is [`is_block_in_chain`] which determines whether a given block of [`BlockId`]
+/// is an ancestor of another "static block".
+///
+/// [`is_block_in_chain`]: Self::is_block_in_chain
+pub trait ChainOracle {
+    /// Error type.
+    type Error: core::fmt::Debug;
+
+    /// Determines whether `block` of [`BlockId`] exists as an ancestor of `static_block`.
+    ///
+    /// If `None` is returned, it means the implementation cannot determine whether `block` exists.
+    fn is_block_in_chain(
+        &self,
+        block: BlockId,
+        static_block: BlockId,
+    ) -> Result<Option<bool>, Self::Error>;
+}