From 18940423eff88b826cfbd232b5340a95e8f21eaa Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 1 Sep 2020 15:49:17 -0400 Subject: [PATCH 01/18] Start of treestate RFC --- book/src/dev/rfcs/0005-treestate.md | 56 +++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 book/src/dev/rfcs/0005-treestate.md diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md new file mode 100644 index 00000000000..22fecfbae0f --- /dev/null +++ b/book/src/dev/rfcs/0005-treestate.md @@ -0,0 +1,56 @@ +# Treestate + +- Feature Name: treestate +- Start Date: 2020-08-31 +- Design PR: +- Zebra Issue: [ZcashFoundation/zebra#958](https://github.com/ZcashFoundation/zebra/issues/958) + +# Summary +[summary]: #summary + +To validate blocks involving shielded transactions, we have to check the +computed treestate from the included transactions against the block header +metadata. This document describes how we compute and manage that data, assuming +a finalized state service as described in the [State Updates RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.md). + + +# Motivation +[motivation]: #motivation + +Block validation requires checking that the treestate of the block (consisting +of the note commitment tree and nullifier set) is consistent with the metadata +we have in the block header (the root of the note commitment tree). + + +# Definitions +[definitions]: #definitions + + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + + +# Drawbacks +[drawbacks]: #drawbacks + + + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + + +# Prior art +[prior-art]: #prior-art + + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + + +# Future possibilities +[future-possibilities]: #future-possibilities + From cd211651fd77b47386654b014d2a241c5f6de674 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 1 Sep 2020 16:11:14 -0400 Subject: [PATCH 02/18] Update PR number --- book/src/dev/rfcs/0005-treestate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index 22fecfbae0f..028ee9e1b45 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -2,7 +2,7 @@ - Feature Name: treestate - Start Date: 2020-08-31 -- Design PR: +- Design PR: [ZcashFoundation/zebra#983](https://github.com/ZcashFoundation/zebra/issues/983) - Zebra Issue: [ZcashFoundation/zebra#958](https://github.com/ZcashFoundation/zebra/issues/958) # Summary From 6dbb5867a300eaec49683e1553bc0f00e71c2d63 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 1 Sep 2020 19:03:26 -0400 Subject: [PATCH 03/18] Add lots of definitions --- book/src/dev/rfcs/0005-treestate.md | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index 028ee9e1b45..24bc70a5ec5 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -25,10 +25,52 @@ we have in the block header (the root of the note commitment tree). # Definitions [definitions]: #definitions +Many terms used here are defined in the [Zcash Protocol Specification](https://zips.z.cash/protocol/protocol.pdf) + +**notes**: Represents that a value is spendable by the recipient who holds the +spending key corresponding to a given shielded payment address. + +**nullifiers**: Revealed by Spend descriptions when its associated note is spent. + +**nullifier set**: The set of unique nullifiers revealed by any transactions +within a block. Nullifiers are enforced to be unique within a valid block chain +by commiting to previous treestates in Spend descriptions, in order to prevent +double-spends. + +**note commitments**: Pedersen commitment to the values consisting a note. One +should not be able to construct a note from its commitment. + +**note position**: The index of a note commitment at the leafmost layer. + +**note commitment tree**: An incremental Merkle tree of fixed depth used to +store note commitments that JoinSplit transfers or Spend transfers produce. It +is used to express the existence of value and the capability to spend it. It is +not the job of this tree to protect against double-spending, as it is +append-only: that's what the nullifier set is for. + +**root**: Layer 0 of a note commitment tree associated with each treestate. + +**anchor**: A Merkle tree root of a note commitment tree. It uniquely identies a +note commitment tree state given the assumed security properties of the Merkle +tree’s hash function. Since the nullier set is always updated together with the +note commitment tree, this also identies a particular state of the associated +nullier set. + +**joinsplits**: + +**spend descriptions**: + +**output descriptions**: + + # Guide-level explanation [guide-level-explanation]: #guide-level-explanation +As the transactions within a block are parsed, Sapling shielded transactions +including Spend descriptions and Output descriptions describe the spending and +creation of Zcash notes. A Spend description includes an anchor, + # Reference-level explanation [reference-level-explanation]: #reference-level-explanation From 04a9b8e55d8a0c8c9a254cca8685dfaa42db1eb4 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Wed, 2 Sep 2020 23:41:23 -0400 Subject: [PATCH 04/18] Filling out guide, with q's about managing joinsplit anchors --- book/src/dev/rfcs/0005-treestate.md | 69 +++++++++++++++++++++-------- 1 file changed, 51 insertions(+), 18 deletions(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index 24bc70a5ec5..be173419eb5 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -10,8 +10,9 @@ To validate blocks involving shielded transactions, we have to check the computed treestate from the included transactions against the block header -metadata. This document describes how we compute and manage that data, assuming -a finalized state service as described in the [State Updates RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.md). +metadata (for Sapling) or previously finalized state (for Sprout). This document +describes how we compute and manage that data, assuming a finalized state +service as described in the [State Updates RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.md). # Motivation @@ -19,7 +20,8 @@ a finalized state service as described in the [State Updates RFC](https://zebra. Block validation requires checking that the treestate of the block (consisting of the note commitment tree and nullifier set) is consistent with the metadata -we have in the block header (the root of the note commitment tree). +we have in the block header (the root of the note commitment tree) or previously +finalized state (for Sprout). # Definitions @@ -27,8 +29,9 @@ we have in the block header (the root of the note commitment tree). Many terms used here are defined in the [Zcash Protocol Specification](https://zips.z.cash/protocol/protocol.pdf) -**notes**: Represents that a value is spendable by the recipient who holds the -spending key corresponding to a given shielded payment address. +**notes**: Represents a value bound to a shielded payment address (public key) +which is spendable by the recipient who holds the spending key corresponding to +a given shielded payment address. **nullifiers**: Revealed by Spend descriptions when its associated note is spent. @@ -40,36 +43,66 @@ double-spends. **note commitments**: Pedersen commitment to the values consisting a note. One should not be able to construct a note from its commitment. -**note position**: The index of a note commitment at the leafmost layer. - **note commitment tree**: An incremental Merkle tree of fixed depth used to store note commitments that JoinSplit transfers or Spend transfers produce. It is used to express the existence of value and the capability to spend it. It is not the job of this tree to protect against double-spending, as it is append-only: that's what the nullifier set is for. -**root**: Layer 0 of a note commitment tree associated with each treestate. - -**anchor**: A Merkle tree root of a note commitment tree. It uniquely identies a -note commitment tree state given the assumed security properties of the Merkle -tree’s hash function. Since the nullier set is always updated together with the -note commitment tree, this also identies a particular state of the associated -nullier set. +**note position**: The index of a note commitment at the leafmost layer, +counting leftmost to rightmost. The [position in the tree is determined by the +order of transactions in the block](https://zips.z.cash/protocol/canopy.pdf#transactions). -**joinsplits**: +**root**: The layer 0 node of a merkle tree. -**spend descriptions**: +**anchor**: A Merkle tree root of a note commitment tree. It uniquely identifies +a note commitment tree state given the assumed security properties of the Merkle +tree’s hash function. Since the nullifier set is always updated together with +the note commitment tree, this also identifies a particular state of the +associated nullier set. -**output descriptions**: +**spend descriptions**: A shielded Sapling transfer that spends a note. Includes +an anchor of some previous block's note commitment tree. +**output descriptions**: A shielded Sapling transfer that creates a +note. Includes the u-coordinate of the note commitment itself. +**joinsplit**: A shielded transfer that can spend Sprout notes and transparent +value, and create new Sprout notes and transparent value, in one Groth16 proof +statement. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation +As blocks are validated, the note commitments revealed by all the transcations +within that block are used to construct note commitment trees, with the note +commitments aligned in their note positions in the bottom layer of the Sprout or +Sapling note tree from the left-most leaf to the right-most in transaction order +in the block. So the Sprout note commitments revealed by the first +JoinSplit in a block would take note position 0 in the Sprout note +commitment tree, for example. Once all the transactions in a block are parsed +and the notes for each tree collected in their appropriate positions, the root +of each tree is computed. While the trees are being built, the respective block +nullifier sets are updated in memory as note nullifiers are revealed. If the +rest of the block is validated according to consensus rules, that root is +committed to its own datastructure via our state service (Sprout anchors, +Sapling anchors). Sapling block validation includes comparing the specified +FinalSaplingRoot in its block header to the root of the Sapling note commitment +tree that we have just computed to make sure they match. + As the transactions within a block are parsed, Sapling shielded transactions including Spend descriptions and Output descriptions describe the spending and -creation of Zcash notes. A Spend description includes an anchor, +creation of Zcash Sapling notes, and JoinSplit on Groth16 descriptions to +transfer/spend/create Sprout notes and transparent value. JoinSplit and Spend +descriptions includes an anchor, which references a previous note commitment +tree root: for Spend's, this is a previous block's anchor as defined in their +block header, for JoinSplit's, it may be a previous block's anchor or the root +produced by a previous JoinSplit transfer in its transaction. For Spend's, this +is convenient because we can query out state service for previously finalized +Sapling block anchors, and validate Spend description independently. For +JoinSplit's, if it's not a previously finalized block anchor, it must be the +treestate anchor of previous JoinSplit in this transaction, and we have to wait +for that one to be parsed and its root computed to check that ours is valid. # Reference-level explanation From 656502f4d012e01ecea781486a92d6e0ed805367 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Mon, 7 Sep 2020 01:24:44 -0400 Subject: [PATCH 05/18] Elaborate on differences in handling Sprout/JoinSplit and Sapling note commitment tree roots/anchor lookup --- book/src/dev/rfcs/0005-treestate.md | 79 ++++++++++++++++++++++++++--- 1 file changed, 71 insertions(+), 8 deletions(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index be173419eb5..938d2d0b874 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -71,6 +71,7 @@ note. Includes the u-coordinate of the note commitment itself. value, and create new Sprout notes and transparent value, in one Groth16 proof statement. + # Guide-level explanation [guide-level-explanation]: #guide-level-explanation @@ -92,17 +93,79 @@ tree that we have just computed to make sure they match. As the transactions within a block are parsed, Sapling shielded transactions including Spend descriptions and Output descriptions describe the spending and -creation of Zcash Sapling notes, and JoinSplit on Groth16 descriptions to +creation of Zcash Sapling notes, and JoinSplit-on-Groth16 descriptions to transfer/spend/create Sprout notes and transparent value. JoinSplit and Spend -descriptions includes an anchor, which references a previous note commitment -tree root: for Spend's, this is a previous block's anchor as defined in their -block header, for JoinSplit's, it may be a previous block's anchor or the root -produced by a previous JoinSplit transfer in its transaction. For Spend's, this -is convenient because we can query out state service for previously finalized -Sapling block anchors, and validate Spend description independently. For +descriptions specify an anchor, which references a previous note commitment tree +root: for Spend's, this is a previous block's anchor as defined in their block +header, for JoinSplit's, it may be a previous block's anchor or the root +produced by a strictly previous JoinSplit description in its transaction. For +Spend's, this is convenient because we can query our state service for +previously finalized Sapling block anchors, and if they are found, then that +[consensus check](https://zips.z.cash/protocol/canopy.pdf#spendsandoutputs) has +been satisfied and the Spend description can be validated independently. For JoinSplit's, if it's not a previously finalized block anchor, it must be the treestate anchor of previous JoinSplit in this transaction, and we have to wait -for that one to be parsed and its root computed to check that ours is valid. +for that one to be parsed and its root computed to check that ours is +valid. Luckily, it can only be a previous JoinSplit in this transaction, and are +[usually the immediately previous one](zcashd), so the set of candidate anchors +is smaller for earlier JoinSplit's in a transcation, but larger for the later +ones. For these JoinSplit's, they can be validated independently of their +anchor's finalization status as long as the final check of the anchor is done, +when available, such as at the Transaction level after all the JoinSplit's have +finished validating everything that can be validated without the context of +their anchor's finalization state. + +So for each transaction, for both Spend descriptions and JoinSplit's, we can +pre-emptively try to do our consensus check by looking up the anchors in our +finalized that first. For Spend's, we then trigger the remaining validation and +when that finishes we are full done with those. For JoinSplit's, the anchor +state check may pass early if it's a previous block Sprout note commitment tree +root, but it may fail because it's an earlier JoinSplit's root instead, so once +the JoinSplit validates independently of the anchor, we wait for all candidate +previous JoinSplit's in that transaction finish validating before doing the +anchor consensus check again, but against the output treestate roots of earlier +JoinSplit's. + +Both Sprout and Sapling note commitment trees must be computed for the whole +block to validate, but Sprout, we need to compute interstitial treestates in +between JoinSplit's in order to do the final consensus check for each/all +JoinSplit's, not just for the whole block, like in Sapling. + +For Sapling, at the block layer, we can iterate over all the transactions in +order and if they have Spend's and/or Output's, we update our Nullifer set for +the block as nullifiers are revealed in Spend descriptions, and update our note +commitment tree as note commitments are revealed in Output descriptions, adding +them as leaves in positions according to their order as they appear transaction +to transction, output to output, in the block. This can be done independent of +the transaction validations. When the Sapling transactions are all validated, +the note commitment tree root should be computed: this is the anchor for this +block. For Sapling and Blossom blocks, we need to check that this root matches +the `RootHash` bytes in this block's header, as the `FinalSaplingRoot`. Once all +other consensus and validation checks are done, this will be saved down to our +finalized state to our `sapling_anchors` set, making it available for lookup by +other Sapling descriptions in future transactions. + +For Sprout, we must compute/update interstitial note commitment trees between +JoinSplit's that may reference an earlier one's root as its anchor. If we do +this at the transaction layer, we can iterate throught all the JoinSplit's and +compute the Sprout note commitment tree and nullifier set similar to how we do +the Sapling ones as described above, but at each state change (ie, +per-JoinSplit) we note the root and cache it for lookup later. As the +JoinSplit's are validated without context, we check for its specified anchor +amongst the interstitial roots we've already calculated (according to the spec, +these interstitial roots don't have to be finalized or the result of an +independently validated JoinSplit, they just must refer to any prior JoinSplit +root in the same transaction). So we only have to wait for our previous root to +be computed via any of our candidates, which in the worst case is waiting for +all of them to be computed for the last JoinSplit. If our JoinSplit's defined +root pops out, that JoinSplit passes that check. + +To finalize the block, the Sprout and Sapling treestates are the ones resulting +from the last transaction in the block, and determines the Sprout and Sapling +anchors that will be associated with this block as we commit it to our finalized +state. The Sprout and Sapling nullifiers revealed in the block will be merged +with the exising ones in our finalized state (ie, it should strictly grow over +time). # Reference-level explanation From f04f2546ce843cea79925ad2433be93ecb5ef495 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 8 Sep 2020 14:03:16 -0400 Subject: [PATCH 06/18] Quote Spends for better rendering Co-authored-by: teor --- book/src/dev/rfcs/0005-treestate.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index 938d2d0b874..6b560e7f3b1 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -96,7 +96,7 @@ including Spend descriptions and Output descriptions describe the spending and creation of Zcash Sapling notes, and JoinSplit-on-Groth16 descriptions to transfer/spend/create Sprout notes and transparent value. JoinSplit and Spend descriptions specify an anchor, which references a previous note commitment tree -root: for Spend's, this is a previous block's anchor as defined in their block +root: for `Spend`s, this is a previous block's anchor as defined in their block header, for JoinSplit's, it may be a previous block's anchor or the root produced by a strictly previous JoinSplit description in its transaction. For Spend's, this is convenient because we can query our state service for @@ -191,4 +191,3 @@ time). # Future possibilities [future-possibilities]: #future-possibilities - From a9a12d7ad84ee25bf1c334c50786c186d3a0f6ec Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 8 Sep 2020 14:03:47 -0400 Subject: [PATCH 07/18] Words Co-authored-by: teor --- book/src/dev/rfcs/0005-treestate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index 6b560e7f3b1..a588c6986e7 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -106,7 +106,7 @@ been satisfied and the Spend description can be validated independently. For JoinSplit's, if it's not a previously finalized block anchor, it must be the treestate anchor of previous JoinSplit in this transaction, and we have to wait for that one to be parsed and its root computed to check that ours is -valid. Luckily, it can only be a previous JoinSplit in this transaction, and are +valid. Luckily, it can only be a previous JoinSplit in this transaction, and is [usually the immediately previous one](zcashd), so the set of candidate anchors is smaller for earlier JoinSplit's in a transcation, but larger for the later ones. For these JoinSplit's, they can be validated independently of their From 4a79b5280bb012a84e0a9cdbf0aab7935836aed1 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 8 Sep 2020 14:04:01 -0400 Subject: [PATCH 08/18] Better words Co-authored-by: teor --- book/src/dev/rfcs/0005-treestate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index a588c6986e7..a815a244bb5 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -117,7 +117,7 @@ their anchor's finalization state. So for each transaction, for both Spend descriptions and JoinSplit's, we can pre-emptively try to do our consensus check by looking up the anchors in our -finalized that first. For Spend's, we then trigger the remaining validation and +finalized set first. For Spend's, we then trigger the remaining validation and when that finishes we are full done with those. For JoinSplit's, the anchor state check may pass early if it's a previous block Sprout note commitment tree root, but it may fail because it's an earlier JoinSplit's root instead, so once From b9bc72dee5cb95682da58bd0c1309cbc46570c02 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 8 Sep 2020 14:04:35 -0400 Subject: [PATCH 09/18] Even betterer words Co-authored-by: Daira Hopwood --- book/src/dev/rfcs/0005-treestate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index a815a244bb5..7e917663b45 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -147,7 +147,7 @@ other Sapling descriptions in future transactions. For Sprout, we must compute/update interstitial note commitment trees between JoinSplit's that may reference an earlier one's root as its anchor. If we do -this at the transaction layer, we can iterate throught all the JoinSplit's and +this at the transaction layer, we can iterate through all the JoinSplit's and compute the Sprout note commitment tree and nullifier set similar to how we do the Sapling ones as described above, but at each state change (ie, per-JoinSplit) we note the root and cache it for lookup later. As the From b60e08cf110f3c87d237434a64ed1002fb924b7f Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 8 Sep 2020 17:54:08 -0400 Subject: [PATCH 10/18] Quote a lot of types for better rendering --- book/src/dev/rfcs/0005-treestate.md | 138 ++++++++++++++-------------- 1 file changed, 69 insertions(+), 69 deletions(-) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/0005-treestate.md index 7e917663b45..00ae502c0a6 100644 --- a/book/src/dev/rfcs/0005-treestate.md +++ b/book/src/dev/rfcs/0005-treestate.md @@ -33,132 +33,132 @@ Many terms used here are defined in the [Zcash Protocol Specification](https://z which is spendable by the recipient who holds the spending key corresponding to a given shielded payment address. -**nullifiers**: Revealed by Spend descriptions when its associated note is spent. +**nullifiers**: Revealed by `Spend` descriptions when its associated `Note` is spent. -**nullifier set**: The set of unique nullifiers revealed by any transactions -within a block. Nullifiers are enforced to be unique within a valid block chain -by commiting to previous treestates in Spend descriptions, in order to prevent +**nullifier set**: The set of unique `Nullifier`s revealed by any `Transaction`s +within a `Block`. `Nullifier`s are enforced to be unique within a valid block chain +by commiting to previous treestates in `Spend` descriptions, in order to prevent double-spends. -**note commitments**: Pedersen commitment to the values consisting a note. One -should not be able to construct a note from its commitment. +**note commitments**: Pedersen commitment to the values consisting a `Note`. One +should not be able to construct a `Note` from its commitment. **note commitment tree**: An incremental Merkle tree of fixed depth used to -store note commitments that JoinSplit transfers or Spend transfers produce. It +store `NoteCommitment`s that `JoinSplit` transfers or `Spend` transfers produce. It is used to express the existence of value and the capability to spend it. It is not the job of this tree to protect against double-spending, as it is -append-only: that's what the nullifier set is for. +append-only: that's what the `Nullifier` set is for. -**note position**: The index of a note commitment at the leafmost layer, +**note position**: The index of a `NoteCommitment` at the leafmost layer, counting leftmost to rightmost. The [position in the tree is determined by the order of transactions in the block](https://zips.z.cash/protocol/canopy.pdf#transactions). -**root**: The layer 0 node of a merkle tree. +**root**: The layer 0 node of a Merkle tree. -**anchor**: A Merkle tree root of a note commitment tree. It uniquely identifies -a note commitment tree state given the assumed security properties of the Merkle -tree’s hash function. Since the nullifier set is always updated together with -the note commitment tree, this also identifies a particular state of the -associated nullier set. +**anchor**: A Merkle tree root of a `NoteCommitment` tree. It uniquely +identifies a `NoteCommitment` tree state given the assumed security properties +of the Merkle tree’s hash function. Since the `Nullifier` set is always updated +together with the `NoteCommitment` tree, this also identifies a particular state +of the associated `Nullifier` set. -**spend descriptions**: A shielded Sapling transfer that spends a note. Includes -an anchor of some previous block's note commitment tree. +**spend descriptions**: A shielded Sapling transfer that spends a `Note`. Includes +an anchor of some previous `Block`'s `NoteCommitment` tree. **output descriptions**: A shielded Sapling transfer that creates a -note. Includes the u-coordinate of the note commitment itself. +`Note`. Includes the u-coordinate of the `NoteCommitment` itself. -**joinsplit**: A shielded transfer that can spend Sprout notes and transparent -value, and create new Sprout notes and transparent value, in one Groth16 proof +**joinsplit**: A shielded transfer that can spend Sprout `Note`s and transparent +value, and create new Sprout `Note`s and transparent value, in one Groth16 proof statement. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -As blocks are validated, the note commitments revealed by all the transcations -within that block are used to construct note commitment trees, with the note -commitments aligned in their note positions in the bottom layer of the Sprout or -Sapling note tree from the left-most leaf to the right-most in transaction order -in the block. So the Sprout note commitments revealed by the first -JoinSplit in a block would take note position 0 in the Sprout note -commitment tree, for example. Once all the transactions in a block are parsed -and the notes for each tree collected in their appropriate positions, the root -of each tree is computed. While the trees are being built, the respective block -nullifier sets are updated in memory as note nullifiers are revealed. If the -rest of the block is validated according to consensus rules, that root is +As `Block`s are validated, the `NoteCommitment`s revealed by all the transactions +within that block are used to construct `NoteCommitmentTree`s, with the +`NoteCommitment`s aligned in their note positions in the bottom layer of the +Sprout or Sapling tree from the left-most leaf to the right-most in +`Transaction` order in the `Block`. So the Sprout `NoteCommitment`s revealed by +the first `JoinSplit` in a block would take note position 0 in the Sprout +note commitment tree, for example. Once all the transactions in a block are +parsed and the notes for each tree collected in their appropriate positions, the +root of each tree is computed. While the trees are being built, the respective +block nullifier sets are updated in memory as note nullifiers are revealed. If +the rest of the block is validated according to consensus rules, that root is committed to its own datastructure via our state service (Sprout anchors, Sapling anchors). Sapling block validation includes comparing the specified -FinalSaplingRoot in its block header to the root of the Sapling note commitment +FinalSaplingRoot in its block header to the root of the Sapling `NoteCommitment` tree that we have just computed to make sure they match. As the transactions within a block are parsed, Sapling shielded transactions -including Spend descriptions and Output descriptions describe the spending and +including `Spend` descriptions and `Output` descriptions describe the spending and creation of Zcash Sapling notes, and JoinSplit-on-Groth16 descriptions to -transfer/spend/create Sprout notes and transparent value. JoinSplit and Spend -descriptions specify an anchor, which references a previous note commitment tree +transfer/spend/create Sprout notes and transparent value. `JoinSplit` and `Spend` +descriptions specify an anchor, which references a previous `NoteCommitment` tree root: for `Spend`s, this is a previous block's anchor as defined in their block -header, for JoinSplit's, it may be a previous block's anchor or the root -produced by a strictly previous JoinSplit description in its transaction. For -Spend's, this is convenient because we can query our state service for +header, for `JoinSplit`s, it may be a previous block's anchor or the root +produced by a strictly previous `JoinSplit` description in its transaction. For +`Spend`s, this is convenient because we can query our state service for previously finalized Sapling block anchors, and if they are found, then that [consensus check](https://zips.z.cash/protocol/canopy.pdf#spendsandoutputs) has -been satisfied and the Spend description can be validated independently. For -JoinSplit's, if it's not a previously finalized block anchor, it must be the -treestate anchor of previous JoinSplit in this transaction, and we have to wait +been satisfied and the `Spend` description can be validated independently. For +`JoinSplit`s, if it's not a previously finalized block anchor, it must be the +treestate anchor of previous `JoinSplit` in this transaction, and we have to wait for that one to be parsed and its root computed to check that ours is -valid. Luckily, it can only be a previous JoinSplit in this transaction, and is +valid. Luckily, it can only be a previous `JoinSplit` in this transaction, and is [usually the immediately previous one](zcashd), so the set of candidate anchors -is smaller for earlier JoinSplit's in a transcation, but larger for the later -ones. For these JoinSplit's, they can be validated independently of their +is smaller for earlier `JoinSplit`s in a transcation, but larger for the later +ones. For these `JoinSplit`s, they can be validated independently of their anchor's finalization status as long as the final check of the anchor is done, -when available, such as at the Transaction level after all the JoinSplit's have +when available, such as at the Transaction level after all the `JoinSplit`s have finished validating everything that can be validated without the context of their anchor's finalization state. -So for each transaction, for both Spend descriptions and JoinSplit's, we can +So for each transaction, for both `Spend` descriptions and `JoinSplit`s, we can pre-emptively try to do our consensus check by looking up the anchors in our -finalized set first. For Spend's, we then trigger the remaining validation and -when that finishes we are full done with those. For JoinSplit's, the anchor -state check may pass early if it's a previous block Sprout note commitment tree -root, but it may fail because it's an earlier JoinSplit's root instead, so once -the JoinSplit validates independently of the anchor, we wait for all candidate -previous JoinSplit's in that transaction finish validating before doing the +finalized set first. For `Spend`s, we then trigger the remaining validation and +when that finishes we are full done with those. For `JoinSplit`s, the anchor +state check may pass early if it's a previous block Sprout `NoteCommitment` tree +root, but it may fail because it's an earlier `JoinSplit`s root instead, so once +the `JoinSplit` validates independently of the anchor, we wait for all candidate +previous `JoinSplit`s in that transaction finish validating before doing the anchor consensus check again, but against the output treestate roots of earlier -JoinSplit's. +`JoinSplit`s. -Both Sprout and Sapling note commitment trees must be computed for the whole +Both Sprout and Sapling `NoteCommitment` trees must be computed for the whole block to validate, but Sprout, we need to compute interstitial treestates in -between JoinSplit's in order to do the final consensus check for each/all -JoinSplit's, not just for the whole block, like in Sapling. +between `JoinSplit`s in order to do the final consensus check for each/all +`JoinSplit`s, not just for the whole block, like in Sapling. For Sapling, at the block layer, we can iterate over all the transactions in -order and if they have Spend's and/or Output's, we update our Nullifer set for -the block as nullifiers are revealed in Spend descriptions, and update our note -commitment tree as note commitments are revealed in Output descriptions, adding +order and if they have `Spend`s and/or `Output`s, we update our Nullifer set for +the block as nullifiers are revealed in `Spend` descriptions, and update our note +commitment tree as `NoteCommitment`s are revealed in `Output` descriptions, adding them as leaves in positions according to their order as they appear transaction to transction, output to output, in the block. This can be done independent of the transaction validations. When the Sapling transactions are all validated, -the note commitment tree root should be computed: this is the anchor for this +the `NoteCommitmentTree` root should be computed: this is the anchor for this block. For Sapling and Blossom blocks, we need to check that this root matches the `RootHash` bytes in this block's header, as the `FinalSaplingRoot`. Once all other consensus and validation checks are done, this will be saved down to our finalized state to our `sapling_anchors` set, making it available for lookup by other Sapling descriptions in future transactions. -For Sprout, we must compute/update interstitial note commitment trees between -JoinSplit's that may reference an earlier one's root as its anchor. If we do -this at the transaction layer, we can iterate through all the JoinSplit's and -compute the Sprout note commitment tree and nullifier set similar to how we do +For Sprout, we must compute/update interstitial `NoteCommitmentTree`s between +`JoinSplit`s that may reference an earlier one's root as its anchor. If we do +this at the transaction layer, we can iterate through all the `JoinSplit`s and +compute the Sprout `NoteCommitmentTree` and nullifier set similar to how we do the Sapling ones as described above, but at each state change (ie, -per-JoinSplit) we note the root and cache it for lookup later. As the -JoinSplit's are validated without context, we check for its specified anchor +per-`JoinSplit`) we note the root and cache it for lookup later. As the +`JoinSplit`s are validated without context, we check for its specified anchor amongst the interstitial roots we've already calculated (according to the spec, these interstitial roots don't have to be finalized or the result of an -independently validated JoinSplit, they just must refer to any prior JoinSplit +independently validated `JoinSplit`, they just must refer to any prior `JoinSplit` root in the same transaction). So we only have to wait for our previous root to be computed via any of our candidates, which in the worst case is waiting for -all of them to be computed for the last JoinSplit. If our JoinSplit's defined -root pops out, that JoinSplit passes that check. +all of them to be computed for the last `JoinSplit`. If our `JoinSplit`s defined +root pops out, that `JoinSplit` passes that check. To finalize the block, the Sprout and Sapling treestates are the ones resulting from the last transaction in the block, and determines the Sprout and Sapling From 51759eecee4b023ada86cbb61657a8b6543e2463 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Tue, 6 Apr 2021 18:28:49 -0400 Subject: [PATCH 11/18] Move treestate design RFC to drafts for now --- book/src/dev/rfcs/{ => drafts}/0005-treestate.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename book/src/dev/rfcs/{ => drafts}/0005-treestate.md (100%) diff --git a/book/src/dev/rfcs/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md similarity index 100% rename from book/src/dev/rfcs/0005-treestate.md rename to book/src/dev/rfcs/drafts/0005-treestate.md From 67c7f49972be486cbb1ce0a6d2f46319ba103ac1 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:16:23 -0400 Subject: [PATCH 12/18] Update book/src/dev/rfcs/drafts/0005-treestate.md --- book/src/dev/rfcs/drafts/0005-treestate.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index 00ae502c0a6..c8bdbd638b6 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -167,6 +167,12 @@ state. The Sprout and Sapling nullifiers revealed in the block will be merged with the exising ones in our finalized state (ie, it should strictly grow over time). +IMPORTANT: we need to save the incremental merkle tree / serialized nodes for: + +Sapling tip block +~all Sprout blocks +chains we're tracking in memory within the reorg limit, for both +We can't just compute a fresh tree with just the note commitments within a block, we are adding them to the tree referenced by the anchor, but we cannot update that tree with just the anchor, we need the 'frontier' nodes and leaves of the incremental merkle tree. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation From 880e85e23c64ec669808067a3e034837626b0c26 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:18:27 -0400 Subject: [PATCH 13/18] Split up sentence --- book/src/dev/rfcs/drafts/0005-treestate.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index c8bdbd638b6..5afb383625a 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -127,9 +127,9 @@ anchor consensus check again, but against the output treestate roots of earlier `JoinSplit`s. Both Sprout and Sapling `NoteCommitment` trees must be computed for the whole -block to validate, but Sprout, we need to compute interstitial treestates in +block to validate. For Sprout, we need to compute interstitial treestates in between `JoinSplit`s in order to do the final consensus check for each/all -`JoinSplit`s, not just for the whole block, like in Sapling. +`JoinSplit`s, not just for the whole block, as in Sapling. For Sapling, at the block layer, we can iterate over all the transactions in order and if they have `Spend`s and/or `Output`s, we update our Nullifer set for From 997efea18fb32124b3e05dca2f5670cd32f6af6b Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:20:41 -0400 Subject: [PATCH 14/18] Add anchor/roothash todo --- book/src/dev/rfcs/drafts/0005-treestate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index 5afb383625a..0706d396cd8 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -144,7 +144,7 @@ the `RootHash` bytes in this block's header, as the `FinalSaplingRoot`. Once all other consensus and validation checks are done, this will be saved down to our finalized state to our `sapling_anchors` set, making it available for lookup by other Sapling descriptions in future transactions. - +TODO: explain Heartwood, Canopy, NU5 rule variants around anchors. For Sprout, we must compute/update interstitial `NoteCommitmentTree`s between `JoinSplit`s that may reference an earlier one's root as its anchor. If we do this at the transaction layer, we can iterate through all the `JoinSplit`s and From e9e5afa78b776e22cd4e7abc186d97626a27f53c Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:22:08 -0400 Subject: [PATCH 15/18] Add definitions management TODO --- book/src/dev/rfcs/drafts/0005-treestate.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index 0706d396cd8..0cb7cb634d4 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -27,6 +27,8 @@ finalized state (for Sprout). # Definitions [definitions]: #definitions +TODO: split up these definitions into common, Sprout, Sapling, and possibly Orchard sections + Many terms used here are defined in the [Zcash Protocol Specification](https://zips.z.cash/protocol/protocol.pdf) **notes**: Represents a value bound to a shielded payment address (public key) From 5e39bc328593e5f20338010ed607a2afdafa4820 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:23:33 -0400 Subject: [PATCH 16/18] Add TODO in guide-level explanation section --- book/src/dev/rfcs/drafts/0005-treestate.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index 0cb7cb634d4..5ea49b3176a 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -77,6 +77,8 @@ statement. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation +TODO: split into common, Sprout, Sapling, and probably Orchard sections + As `Block`s are validated, the `NoteCommitment`s revealed by all the transactions within that block are used to construct `NoteCommitmentTree`s, with the `NoteCommitment`s aligned in their note positions in the bottom layer of the From 919491d48197fc3a19a879eefcfba45e0815877f Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:26:09 -0400 Subject: [PATCH 17/18] Update summary for Orchard --- book/src/dev/rfcs/drafts/0005-treestate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index 5ea49b3176a..54b73cc432c 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -10,7 +10,7 @@ To validate blocks involving shielded transactions, we have to check the computed treestate from the included transactions against the block header -metadata (for Sapling) or previously finalized state (for Sprout). This document +metadata (for Sapling and Orchard) or previously finalized state (for Sprout). This document describes how we compute and manage that data, assuming a finalized state service as described in the [State Updates RFC](https://zebra.zfnd.org/dev/rfcs/0005-state-updates.md). From 937de0d97ff0da10a90e12bdd6cbe2adc42f9781 Mon Sep 17 00:00:00 2001 From: Deirdre Connolly Date: Thu, 15 Apr 2021 13:32:45 -0400 Subject: [PATCH 18/18] Add Action description definition --- book/src/dev/rfcs/drafts/0005-treestate.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/book/src/dev/rfcs/drafts/0005-treestate.md b/book/src/dev/rfcs/drafts/0005-treestate.md index 54b73cc432c..1fcf44e0c07 100644 --- a/book/src/dev/rfcs/drafts/0005-treestate.md +++ b/book/src/dev/rfcs/drafts/0005-treestate.md @@ -69,6 +69,12 @@ an anchor of some previous `Block`'s `NoteCommitment` tree. **output descriptions**: A shielded Sapling transfer that creates a `Note`. Includes the u-coordinate of the `NoteCommitment` itself. +**action descriptions**: A shielded Orchard transfer that spends and/or creates a `Note`. +Does not include an anchor, because that is encoded once in the `anchorOrchard` +field of a V5 `Transaction`. + + + **joinsplit**: A shielded transfer that can spend Sprout `Note`s and transparent value, and create new Sprout `Note`s and transparent value, in one Groth16 proof statement.