Sprout note commitment trees (#3051)

* Implement incremental note commitment Merkle tree for Sprout * Add tests for Sprout note commitment tree * Remove the `Arbitrary` attribute * Reverse the vector of empty roots * Add more tests * Refactor rustdoc Co-authored-by: Deirdre Connolly <[email protected]> * Refactor rustdoc Co-authored-by: Deirdre Connolly <[email protected]> * rustdoc * Rustdoc * rustdoc links * Oops, need the trait in scope to use it * Avoid accessing the wrapped hash directly Co-authored-by: Deirdre Connolly <[email protected]> * rustfmt * Add typing * Avoid accessing the wrapped hash directly * Implement incremental note commitment Merkle tree for Sprout * Add tests for Sprout note commitment tree * Remove the `Arbitrary` attribute * Reverse the vector of empty roots * Add more tests * Refactor rustdoc Co-authored-by: Deirdre Connolly <[email protected]> * Refactor rustdoc Co-authored-by: Deirdre Connolly <[email protected]> * rustdoc * Rustdoc * rustdoc links * Oops, need the trait in scope to use it * Avoid accessing the wrapped hash directly Co-authored-by: Deirdre Connolly <[email protected]> * rustfmt * Add typing * Avoid accessing the wrapped hash directly * Add Overwinter final roots (test vectors) * Test sprout note commitments trees on Overwinter blocks * Add new test vectors * Finish the tests for the note commitment trees * Make the wrapped hash in `Root` private Co-authored-by: Deirdre Connolly <[email protected]> Co-authored-by: Deirdre Connolly <[email protected]>
ZcashFoundation · Nov 18, 2021 · 8963007 · 8963007
1 parent ad81718
commit 8963007
Show file tree

Hide file tree

Showing 11 changed files with 600 additions and 189 deletions.
diff --git a/README.md b/README.md
@@ -53,16 +53,16 @@ in terms of speed and resistance to denial of service attacks, for example.
 
 These are some of the advantages or benefits of Zebra:
 
-- Better performance: since it was implemented from scratch in an async, parallelized way, Zebra 
+- Better performance: since it was implemented from scratch in an async, parallelized way, Zebra
   is currently faster than `zcashd`.
 - Better security: since it is developed in a memory-safe language (Rust), Zebra
-  is less likely to be affected by memory-safety and correctness security bugs that 
+  is less likely to be affected by memory-safety and correctness security bugs that
   could compromise the environment where it is run.
 - Better governance: with a new node deployment, there will be more developers
   who can implement different features for the Zcash network.
 - Dev accessibility: supports more developers, which gives new developers
   options for contributing to Zcash protocol development.
-- Runtime safety: with an independeny implementation, the detection of consensus bugs 
+- Runtime safety: with an independent implementation, the detection of consensus bugs
   can happen quicker, reducing the risk of consensus splits.
 - Spec safety: with several node implementations, it is much easier to notice
   bugs and ambiguity in protocol specification.
@@ -78,7 +78,7 @@ Every few weeks, we release a new Zebra beta [release](https://github.com/ZcashF
 Zebra's network stack is interoperable with `zcashd`,
 and Zebra implements all the features required to reach Zcash network consensus.
 
-The goals of the beta release series are for Zebra to act as a fully validating Zcash node for 
+The goals of the beta release series are for Zebra to act as a fully validating Zcash node for
 all applicable consensus rules as of NU5 activation.
 
 Currently, Zebra does not validate the following Zcash consensus rules:

diff --git a/zebra-chain/src/sprout/tests.rs b/zebra-chain/src/sprout/tests.rs
@@ -1 +1,5 @@
+//! Sprout tests.
+
 mod preallocate;
+mod test_vectors;
+mod tree;
diff --git a/zebra-chain/src/sprout/tests/test_vectors.rs b/zebra-chain/src/sprout/tests/test_vectors.rs
@@ -0,0 +1,84 @@
+// From https://github.com/zcash/zcash/blob/master/src/zcash/IncrementalMerkleTree.cpp#L439
+pub const HEX_EMPTY_ROOTS: [&str; 30] = [
+    "0000000000000000000000000000000000000000000000000000000000000000",
+    "da5698be17b9b46962335799779fbeca8ce5d491c0d26243bafef9ea1837a9d8",
+    "dc766fab492ccf3d1e49d4f374b5235fa56506aac2224d39f943fcd49202974c",
+    "3f0a406181105968fdaee30679e3273c66b72bf9a7f5debbf3b5a0a26e359f92",
+    "26b0052694fc42fdff93e6fb5a71d38c3dd7dc5b6ad710eb048c660233137fab",
+    "0109ecc0722659ff83450b8f7b8846e67b2859f33c30d9b7acd5bf39cae54e31",
+    "3f909b8ce3d7ffd8a5b30908f605a03b0db85169558ddc1da7bbbcc9b09fd325",
+    "40460fa6bc692a06f47521a6725a547c028a6a240d8409f165e63cb54da2d23f",
+    "8c085674249b43da1b9a31a0e820e81e75f342807b03b6b9e64983217bc2b38e",
+    "a083450c1ba2a3a7be76fad9d13bc37be4bf83bd3e59fc375a36ba62dc620298",
+    "1ddddabc2caa2de9eff9e18c8c5a39406d7936e889bc16cfabb144f5c0022682",
+    "c22d8f0b5e4056e5f318ba22091cc07db5694fbeb5e87ef0d7e2c57ca352359e",
+    "89a434ae1febd7687eceea21d07f20a2512449d08ce2eee55871cdb9d46c1233",
+    "7333dbffbd11f09247a2b33a013ec4c4342029d851e22ba485d4461851370c15",
+    "5dad844ab9466b70f745137195ca221b48f346abd145fb5efc23a8b4ba508022",
+    "507e0dae81cbfbe457fd370ef1ca4201c2b6401083ddab440e4a038dc1e358c4",
+    "bdcdb3293188c9807d808267018684cfece07ac35a42c00f2c79b4003825305d",
+    "bab5800972a16c2c22530c66066d0a5867e987bed21a6d5a450b683cf1cfd709",
+    "11aa0b4ad29b13b057a31619d6500d636cd735cdd07d811ea265ec4bcbbbd058",
+    "5145b1b055c2df02b95675e3797b91de1b846d25003c0a803d08900728f2cd6a",
+    "0323f2850bf3444f4b4c5c09a6057ec7169190f45acb9e46984ab3dfcec4f06a",
+    "671546e26b1da1af754531e26d8a6a51073a57ddd72dc472efb43fcb257cffff",
+    "bb23a9bba56de57cb284b0d2b01c642cf79c9a5563f0067a21292412145bd78a",
+    "f30cc836b9f71b4e7ee3c72b1fd253268af9a27e9d7291a23d02821b21ddfd16",
+    "58a2753dade103cecbcda50b5ebfce31e12d41d5841dcc95620f7b3d50a1b9a1",
+    "925e6d474a5d8d3004f29da0dd78d30ae3824ce79dfe4934bb29ec3afaf3d521",
+    "08f279618616bcdd4eadc9c7a9062691a59b43b07e2c1e237f17bd189cd6a8fe",
+    "c92b32db42f42e2bf0a59df9055be5c669d3242df45357659b75ae2c27a76f50",
+    "c0db2a74998c50eb7ba6534f6d410efc27c4bb88acb0222c7906ea28a327b511",
+    "d7c612c817793191a1e68652121876d6b3bde40f4fa52bc314145ce6e5cdd259",
+];
+// From https://github.com/zcash/zcash/blob/master/src/test/data/merkle_commitments.json
+// Byte-reversed from those ones because the original test vectors are
+// loaded using uint256S()
+pub const COMMITMENTS: [&str; 16] = [
+    "62fdad9bfbf17c38ea626a9c9b8af8a748e6b4367c8494caf0ca592999e8b6ba",
+    "68eb35bc5e1ddb80a761718e63a1ecf4d4977ae22cc19fa732b85515b2a4c943",
+    "836045484077cf6390184ea7cd48b460e2d0f22b2293b69633bb152314a692fb",
+    "92498a8295ea36d593eaee7cb8b55be3a3e37b8185d3807693184054cd574ae4",
+    "ff7c360374a6508ae0904c782127ff5dce90918f3ee81cf92ef1b69afb8bf443",
+    "68c4d0f69d1f18b756c2ee875c14f1c6cd38682e715ded14bf7e3c1c5610e9fc",
+    "8b16cd3ec44875e4856e30344c0b4a68a6f929a68be5117b225b80926301e7b1",
+    "50c0b43061c39191c3ec529734328b7f9cafeb6fd162cc49a4495442d9499a2d",
+    "70ffdd5fa0f3aea18bd4700f1ac2e2e03cf5d4b7b857e8dd93b862a8319b9653",
+    "d81ef64a0063573d80cd32222d8d04debbe807345ad7af2e9edf0f44bdfaf817",
+    "8b92a4ec694271fe1b16cc0ea8a433bf19e78eb5ca733cc137f38e5ecb05789b",
+    "04e963ab731e4aaaaaf931c3c039ea8c9d7904163936e19a8929434da9adeba3",
+    "be3f6c181f162824191ecf1f78cae3ffb0ddfda671bb93277ce6ebc9201a0912",
+    "1880967fc8226380a849c63532bba67990f7d0a10e9c90b848f58d634957c6e9",
+    "c465bb2893cba233351094f259396301c23d73a6cf6f92bc63428a43f0dd8f8e",
+    "84c834e7cb38d6f08d82f5cf4839b8920185174b11c7af771fd38dd02b206a20",
+];
+
+// Calculated by the above implementation for MERKLE_DEPTH = 29 by the
+// same code confirmed to produce the test vectors from
+// https://github.com/zcash/zcash/blob/master/src/test/data/merkle_roots.json
+// when MERKLE_DEPTH = 4.
+pub const ROOTS: [&str; 16] = [
+    "b8e10b6c157be92c43a733e2c9bddb963a2fb9ea80ebcb307acdcc5fc89f1656",
+    "83a7754b8240699dd1b63bf70cf70db28ffeb74ef87ce2f4dd32c28ae5009f4f",
+    "c45297124f50dcd3f78eed017afd1e30764cd74cdf0a57751978270fd0721359",
+    "b61f588fcba9cea79e94376adae1c49583f716d2f20367141f1369a235b95c98",
+    "a3165c1708f0cc028014b9bf925a81c30091091ca587624de853260cd151b524",
+    "6bb8c538c550abdd26baa2a7510a4ae50a03dc00e52818b9db3e4ffaa29c1f41",
+    "e04e4731085ba95e3fa7c8f3d5eb9a56af63363403b783bc68802629c3fe505b",
+    "c3714ab74d8e3984e8b58a2b4806934d20f6e67d7246cf8f5b2762305294a0ea",
+    "63657edeead4bc45610b6d5eb80714a0622aad5788119b7d9961453e3aacda21",
+    "e31b80819221718440c5351525dbb902d60ed16b74865a2528510959a1960077",
+    "872f13df2e12f5503c39100602930b0f91ea360e5905a9f5ceb45d459efc36b2",
+    "bdd7105febb3590832e946aa590d07377d1366cf5e7267507efa399dd0febdbc",
+    "0f45f4adcb846a8bb56833ca0cae96f2fb8747958daa191a46d0f9d93268260a",
+    "41c6e456e2192ab74f72cb27c444a2734ca8ade5a4788c1bc2546118dda01778",
+    "8261355fd9bafc52a08d738fed29a859fbe15f2e74a5353954b150be200d0e16",
+    "90665cb8a43001f0655169952399590cd17f99165587c1dd842eb674fb9f0afe",
+];
+
+/// Empty (unused) Sprout note commitment tree leaf node.
+///
+/// Uncommitted^Sprout = [0]^(l^Sprout_Merkle).
+///
+/// <https://zips.z.cash/protocol/protocol.pdf#constants>
+pub const EMPTY_LEAF: [u8; 32] = [0; 32];
diff --git a/zebra-chain/src/sprout/tests/tree.rs b/zebra-chain/src/sprout/tests/tree.rs
@@ -0,0 +1,185 @@
+//! Tests for Sprout Note Commitment Trees.
+
+use std::sync::Arc;
+
+use color_eyre::eyre;
+use eyre::Result;
+
+use hex::FromHex;
+use zebra_test::vectors;
+
+use crate::block::Block;
+use crate::parameters::{Network, NetworkUpgrade};
+use crate::serialization::ZcashDeserializeInto;
+use crate::sprout::commitment::NoteCommitment;
+use crate::sprout::tests::test_vectors;
+use crate::sprout::tree;
+
+/// Tests if empty roots are generated correctly.
+#[test]
+fn empty_roots() {
+    zebra_test::init();
+
+    for i in 0..tree::EMPTY_ROOTS.len() {
+        assert_eq!(
+            hex::encode(tree::EMPTY_ROOTS[i]),
+            // The test vector is in reversed order.
+            test_vectors::HEX_EMPTY_ROOTS[tree::MERKLE_DEPTH - i]
+        );
+    }
+}
+
+/// Tests if we have the right unused (empty) leaves.
+#[test]
+fn empty_leaf() {
+    assert_eq!(
+        tree::NoteCommitmentTree::uncommitted(),
+        test_vectors::EMPTY_LEAF
+    );
+}
+
+/// Tests if we can build the tree correctly.
+#[test]
+fn incremental_roots() {
+    zebra_test::init();
+
+    let mut leaves = vec![];
+
+    let mut incremental_tree = tree::NoteCommitmentTree::default();
+
+    for (i, cm) in test_vectors::COMMITMENTS.iter().enumerate() {
+        let bytes = <[u8; 32]>::from_hex(cm).unwrap();
+        let cm = NoteCommitment::from(bytes);
+
+        // Test if we can append a new note commitment to the tree.
+        let _ = incremental_tree.append(cm);
+        let incremental_root_hash = incremental_tree.hash();
+        assert_eq!(hex::encode(incremental_root_hash), test_vectors::ROOTS[i]);
+
+        // The root hashes should match.
+        let incremental_root = incremental_tree.root();
+        assert_eq!(<[u8; 32]>::from(incremental_root), incremental_root_hash);
+
+        // Test if the note commitments are counted correctly.
+        assert_eq!(incremental_tree.count(), (i + 1) as u64);
+
+        // Test if we can build the tree from a vector of note commitments
+        // instead of appending only one note commitment to the tree.
+        leaves.push(cm);
+        let ad_hoc_tree = tree::NoteCommitmentTree::from(leaves.clone());
+        let ad_hoc_root_hash = ad_hoc_tree.hash();
+        assert_eq!(hex::encode(ad_hoc_root_hash), test_vectors::ROOTS[i]);
+
+        // The root hashes should match.
+        let ad_hoc_root = ad_hoc_tree.root();
+        assert_eq!(<[u8; 32]>::from(ad_hoc_root), ad_hoc_root_hash);
+
+        // Test if the note commitments are counted correctly.
+        assert_eq!(ad_hoc_tree.count(), (i + 1) as u64);
+    }
+}
+
+#[test]
+fn incremental_roots_with_blocks() -> Result<()> {
+    incremental_roots_with_blocks_for_network(Network::Mainnet)?;
+    incremental_roots_with_blocks_for_network(Network::Testnet)?;
+
+    Ok(())
+}
+
+fn incremental_roots_with_blocks_for_network(network: Network) -> Result<()> {
+    // The mainnet block height at which the first JoinSplit occurred.
+    const MAINNET_FIRST_JOINSPLIT_HEIGHT: u32 = 396;
+
+    // The testnet block height at which the first JoinSplit occurred.
+    const TESTNET_FIRST_JOINSPLIT_HEIGHT: u32 = 2259;
+
+    // Load the test data.
+    let (blocks, sprout_roots, next_height) = match network {
+        Network::Mainnet => (
+            &*vectors::MAINNET_BLOCKS,
+            &*vectors::MAINNET_FINAL_SPROUT_ROOTS,
+            MAINNET_FIRST_JOINSPLIT_HEIGHT,
+        ),
+        Network::Testnet => (
+            &*vectors::TESTNET_BLOCKS,
+            &*vectors::TESTNET_FINAL_SPROUT_ROOTS,
+            TESTNET_FIRST_JOINSPLIT_HEIGHT,
+        ),
+    };
+
+    // Load the Genesis height.
+    let genesis_height = NetworkUpgrade::Genesis
+        .activation_height(network)
+        .unwrap()
+        .0;
+
+    // Load the Genesis block.
+    let genesis_block = Arc::new(
+        blocks
+            .get(&genesis_height)
+            .expect("test vector exists")
+            .zcash_deserialize_into::<Block>()
+            .expect("block is structurally valid"),
+    );
+
+    // Build an empty note commitment tree.
+    let mut note_commitment_tree = tree::NoteCommitmentTree::default();
+
+    // Add note commitments from Genesis to the tree.
+    for transaction in genesis_block.transactions.iter() {
+        for sprout_note_commitment in transaction.sprout_note_commitments() {
+            note_commitment_tree
+                .append(*sprout_note_commitment)
+                .expect("we should not be able to fill up the tree");
+        }
+    }
+
+    // Load the Genesis note commitment tree root.
+    let genesis_anchor = tree::Root::from(
+        **sprout_roots
+            .get(&genesis_height)
+            .expect("test vector exists"),
+    );
+
+    // Check if the root of the note commitment tree of Genesis is correct.
+    assert_eq!(genesis_anchor, note_commitment_tree.root());
+
+    // Load the first block after Genesis that contains a JoinSplit transaction
+    // so that we can add new note commitments to the tree.
+    let next_block = Arc::new(
+        blocks
+            .get(&(genesis_height + next_height))
+            .expect("test vector exists")
+            .zcash_deserialize_into::<Block>()
+            .expect("block is structurally valid"),
+    );
+
+    // Load the note commitment tree root of `next_block`.
+    let next_block_anchor = tree::Root::from(
+        **sprout_roots
+            .get(&(genesis_height + next_height))
+            .expect("test vector exists"),
+    );
+
+    // Add the note commitments from `next_block` to the tree.
+    let mut appended_count = 0;
+    for transaction in next_block.transactions.iter() {
+        for sprout_note_commitment in transaction.sprout_note_commitments() {
+            note_commitment_tree
+                .append(*sprout_note_commitment)
+                .expect("test vector is correct");
+            appended_count += 1;
+        }
+    }
+
+    // We also want to make sure that sprout_note_commitments() is returning
+    // the commitments in the right order. But this will only be actually tested
+    // if there are more than one note commitment in a block.
+    assert!(appended_count > 1);
+
+    // Check if the root of `next_block` is correct.
+    assert_eq!(next_block_anchor, note_commitment_tree.root());
+
+    Ok(())
+}