From d014e2567f0dd5188996d548c4270237eb77dcb2 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Thu, 9 Apr 2020 13:32:14 -0700 Subject: [PATCH 01/26] [WIP] Runtime storage Closes #5 --- current/runtime/storage.md | 78 +++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 22 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 89e87c3..3c4aa4b 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -4,30 +4,38 @@ lang: en title: Runtime Storage --- -Runtime storage allows you to store data in your blockchain which can be accessed from your runtime -logic and persists between blocks. +Runtime storage allows you to store data in your blockchain that is persisted between blocks and can be accessed from within your runtime logic. Storage should be one of the most critical concerns of a blockchain runtime developer. This statement is somewhat self-evident, since one of the primary objectives of a blockchain is to provide decentralized consensus about the state of the underlying storage. Furthermore, well designed storage systems reduce the load on nodes in the network, which will lower the overhead for participants in your blockchain. Substrate exposes a set of layered, modular storage APIs that allow runtime developers to make the storage decisions that suit them best. However, the fundamental principal of blockchain runtime storage is to minimize its use. This document is intended to provide information and best practices about Substrate's runtime storage interfaces. Please refer to [the advanced storage documentation](../advanced/storage) for more information about how these interfaces are implemented. ## Storage Items -Your runtime module has access to Substrate storage APIs which allows you to easily store common -storage items: +[The `storage` module in the Substrate Support pallet](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) gives runtime developers access to Substrate's flexible storage APIs: -- [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - - A single value. -- [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - - A key-value hash map. -- [Storage Linked Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageLinkedMap.html) - - Similar to a storage map, but allows enumeration of the stored elements. -- [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - - An implementation of a map with two keys. +* [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - A single value +* [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - A key-value hash map +* [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - An implementation of a map with two keys that + provides the important ability to efficiently remove all entries that have a common first key +* Iterable Storage Maps - Implementations of [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html) and [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html) whose keys and values can be iterated over Any value which can be encoded by the [Parity SCALE codec](../advanced/codec) is supported by these storage APIs. -### Storage Declaration +The type of storage item you use should depend on the logical way in which the value will be used by your runtime. -You can use the `decl_storage!` macro to easily create new runtime storage items. Here is an example -of what it looks like to declare each type of storage item: +### Storage Value + +This type of storage item should be used for values that are viewed as a single unit by the runtime, whether that is a single primitive value, a single `struct` or a single collection of related items. Although wrapping related items in a shared `struct` is an excellent way to reduce the number of storage reads (an important consideration), at some point the size of the object will begin to incur costs that may outweigh the optimization in storage reads. Storage values can be used to store lists of items, but runtime developers should take care with respect to the size of these lists. Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list in your runtime may result in exceeding the block production time. + +### Storage Maps + +Substrate maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to give blockchain engineers increased control over the way in which these data structures are stored, Substrate allows developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. + +### Iterable Storage Maps + +The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native lists, they are significantly more costly than lists to iterate over with respect to time. Depending on the hashing algorithm that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. + +## Declaring Storage Items + +You can use the `decl_storage!` macro to easily create new runtime storage items. Here is an example of what it looks like to declare each type of storage item: ```rust decl_storage! { @@ -40,13 +48,13 @@ decl_storage! { } ``` -## Default Value +Notice that the last item, the `double_map` specifies the "hasher" (the hash algorithm) that should be used to generate one of its key. Keep reading for more information about the different hashing algorithms and when to use them. + +### Default Values -Substrate allows you to define the default value which is returned when the storage item is not set. -This value is **not** actually stored in the runtime storage, but runtime logic will see this value -during execution. +Substrate allows you to specify a default value that is returned when a storage item's value is not set. The default value does **not** actually occupy runtime storage, but runtime logic will see this value during execution. -Here is an example for setting a default value for all items in a map: +Here is an example of specifying the default value for all items in a map: ```rust decl_storage! { @@ -56,7 +64,33 @@ decl_storage! { } ``` -## Verify First, Write Last +## Hashing Algorithms + +As mentioned above, a novel feature of Substrate's storage maps is that they allow developers to specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. In this document you will find information about the pros and cons of the different approaches to hashing; read the advanced storage document if you would like a better understanding of _why_ the different algorithms may behave differently. + +### Cryptographic Hashing Algorithms + +Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of outputs even if the inputs were the numbers 1 through 10. It is critical to use cryptographic hashing algorithms when users are able to influence the keys of a storage map. Failure to do so creates an attack vector that makes it easy for malicious actors to degrade the performance of your blockchain network. An example of a map that should use a cryptographic hash algorithm to generate its keys is a map used to track account balances. In this case, it is important to use a cryptographic hashing algorithm so that an attacker cannot bombard your system with many small transfers to sequential account numbers; without a cryptographic hash algorithm this would create an imbalanced storage structure that would suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic counterparts, which is why Substrate allows developers to select when they are used. + +### Transparent Hashing Algorithms + +A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a key's original unhashed value and verify it (by re-hashing it) if they'd like. It is generally recommended to use transparent hashing algorithms for your runtime's storage maps. In fact, it is necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. + +### Common Substrate Hashers + +This table lists some common hashers used in Substrate and describes which ones are cryptographic and which ones are transparent: + +| Hasher | Cryptographic | Transparent | +| ----------------------------------------------------------------------------------------------------- | ------------- | ----------- | +| [Blake2 128](https://substrate.dev/rustdocs/master/frame_support/struct.Blake2_128.html) | X | | +| [TwoX 128](https://substrate.dev/rustdocs/master/frame_support/struct.Twox128.html) | | | +| [Blake2 128 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Blake2_128Concat.html) | X | X | +| [TwoX 64 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Twox64Concat.html) | | X | +| [Identity](https://substrate.dev/rustdocs/master/frame_support/struct.Identity.html) | | | + +The Identity hasher exposes a hashing algorithm that has an output equal to its input (the identity function).This type of hasher should only be used when the starting key is already a cryptographic hash. + +## Child Storage Tries TODO @@ -64,7 +98,7 @@ TODO TODO -## Child Storage Tries +## Verify First, Write Last TODO From ab8aed196cd352406ca2a72f5921ec838f2ad716 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Fri, 10 Apr 2020 07:48:16 -0700 Subject: [PATCH 02/26] Add section on calculating storage keys and using RPC to query storage --- current/runtime/storage.md | 61 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 3 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 3c4aa4b..1ab6fc0 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -66,15 +66,15 @@ decl_storage! { ## Hashing Algorithms -As mentioned above, a novel feature of Substrate's storage maps is that they allow developers to specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. In this document you will find information about the pros and cons of the different approaches to hashing; read the advanced storage document if you would like a better understanding of _why_ the different algorithms may behave differently. +As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. In this document you will find information about the pros and cons of the different approaches to hashing; read [the advanced storage document](../advanced/storage) if you would like a better understanding of _why_ the different algorithms may behave differently. ### Cryptographic Hashing Algorithms -Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of outputs even if the inputs were the numbers 1 through 10. It is critical to use cryptographic hashing algorithms when users are able to influence the keys of a storage map. Failure to do so creates an attack vector that makes it easy for malicious actors to degrade the performance of your blockchain network. An example of a map that should use a cryptographic hash algorithm to generate its keys is a map used to track account balances. In this case, it is important to use a cryptographic hashing algorithm so that an attacker cannot bombard your system with many small transfers to sequential account numbers; without a cryptographic hash algorithm this would create an imbalanced storage structure that would suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic counterparts, which is why Substrate allows developers to select when they are used. +Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of outputs even if the inputs were the numbers 1 through 10. It is critical to use cryptographic hashing algorithms when users are able to influence the keys of a Storage Map. Failure to do so creates an attack vector that makes it easy for malicious actors to degrade the performance of your blockchain network. An example of a map that should use a cryptographic hash algorithm to generate its keys is a map used to track account balances. In this case, it is important to use a cryptographic hashing algorithm so that an attacker cannot bombard your system with many small transfers to sequential account numbers; without a cryptographic hash algorithm this would create an imbalanced storage structure that would suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic counterparts, which is why Substrate allows developers to select when they are used. ### Transparent Hashing Algorithms -A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a key's original unhashed value and verify it (by re-hashing it) if they'd like. It is generally recommended to use transparent hashing algorithms for your runtime's storage maps. In fact, it is necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. +A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a key's original unhashed value and verify it (by re-hashing it) if they'd like. It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. ### Common Substrate Hashers @@ -90,6 +90,61 @@ This table lists some common hashers used in Substrate and describes which ones The Identity hasher exposes a hashing algorithm that has an output equal to its input (the identity function).This type of hasher should only be used when the starting key is already a cryptographic hash. +## Querying Storage + +Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query your blockchain's runtime storage. One thing that will be important to understand before proceeding is that the Substrate runtime storage APIs are layers of abstraction that have been built upon a simple key-value database. When you use the Substrate RPC to query runtime storage, you only need to provide a single piece of information: the storage key. The rest of this section will go over the basics of how to calculate storage keys for the different types of storage items. If you'd like to learn more about how Substrate uses a key-value database to implement the different kinds of storage items, refer to [the advanced storage documentation](../advanced/storage). + +### Storage Value Keys + +To calculate the key for a simple Storage Value, take the TwoX 128 hash of the name of the module that contains the Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage Value item named [`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): + +``` +twox_128("Sudo") = "0x5c0d1176a568c1f92944340dbfed9e9c" +twox_128("Key) = "0x530ebca703c85910e7164cb7d1c9e47b" +twox_128("Sudo") + twox_128("Key") = "0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b" +``` + +If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo module's `Key` Storage Value could be represented as: + +``` +state_getStorage("0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" +``` + +In this case, the value that is returned (`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is Alice's [SCALE](../advanced/codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). + +You may have noticed that the non-cryptographic TwoX 128 hash algorithm is used to generate Storage Value keys. This is because it is not necessary to pay the performance costs associated with a cryptographic hash function since the input to the hash function (the names of the module and storage item) are determined by you, the runtime developer, and not by potentially malicious users of your blockchain. + +### Storage Map Keys + +Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double Maps), append the hash of the first map key followed by the hash of the second map key to the map's storage key. Remember, Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you will need to make sure to use the correct hashing algorithm (the one that was declared in [the `decl_storage` macro](#Declaring-Storage-Items)) when determining the hashed keys for the elements in a map. + +Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using [the transparent Blake2 128 Concat hashing algorithm](#Transparent-Hashing-Algorithms): + +``` +twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" +twox_128("FreeBalance") = "0x6482b9ade7bc6657aaca787ba1add3b4" +scale_encode("5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" + +blake2_128_concat("0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0xde1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" + +state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0x0000a0dec5adc9353600000000000000" +``` + +The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the [SCALE](../advanced/codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat is [a transparent hashing algorithm as describe above](#Transparent-Hashing-Algorithms). Although the above example may make this characteristic seem superfluous, its utility becomes more apparent when the goal is to iterate over the keys in a map (as opposed to retrieving the value associated with a single key). The ability to iterate over the keys in a map is a common requirement in order to allow _people_ to use the map in a way that seems natural (such as UIs): first, a user is presented with a list of elements in the map, then, that user can select the element that they are interested in and query the map for more details about that particular element. Here is another example that uses the same example Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing algorithm in a module named "Balances") that will demonstrate using the Substrate RPC to query a Storage Map for its list of keys via the `state_getKeys` RPC endpoint: + +``` +twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" +twox_128("FreeBalance") = "0x6482b9ade7bc6657aaca787ba1add3b4" + +state_getKeys("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4") = [ + "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d", + "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b432a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f", + ... +] +``` + +Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be directly used as input for the RPC's `state_getStorage` endpoint. In fact, the first element in the example list above is equal to the input used for the `state_getStorage` query in the previous example (the one used to find the balance for `Alice`). Because the map that these keys belong to uses a transparent hashing algorithm to generate its keys, it is possible to determine the account associated with the second element in the list. Notice that each element in the list is a hexadecimal value that begins with the same 64 characters; this is because each list element represents a key in the same map, and that map is identified by concatenating two TwoX 128 hashes, each of which are 128-bits or 32 hexadecimal characters. After discarding this portion of the second element in the list, you are left with `0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. You saw in the previous example that this represents the Blake2 128 Concat hash of some [SCALE](../advanced/codec)-encoded account ID. As described above, the Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing algorithm. In this example, after you remove the first 32 hexadecimal characters that represent a Blake2 128 hash (i.e. `0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value `0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](../advanced/codec)-encoded account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account ID for the familiar `Alice_Stash` account. + ## Child Storage Tries TODO From 556b6db48a78a6acb27085ddf1f431355f98ff69 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 13 Apr 2020 08:12:46 -0700 Subject: [PATCH 03/26] Use 120-character line length --- current/runtime/storage.md | 168 +++++++++++++++++++++++++++++-------- 1 file changed, 134 insertions(+), 34 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 1ab6fc0..f2e0446 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -4,17 +4,30 @@ lang: en title: Runtime Storage --- -Runtime storage allows you to store data in your blockchain that is persisted between blocks and can be accessed from within your runtime logic. Storage should be one of the most critical concerns of a blockchain runtime developer. This statement is somewhat self-evident, since one of the primary objectives of a blockchain is to provide decentralized consensus about the state of the underlying storage. Furthermore, well designed storage systems reduce the load on nodes in the network, which will lower the overhead for participants in your blockchain. Substrate exposes a set of layered, modular storage APIs that allow runtime developers to make the storage decisions that suit them best. However, the fundamental principal of blockchain runtime storage is to minimize its use. This document is intended to provide information and best practices about Substrate's runtime storage interfaces. Please refer to [the advanced storage documentation](../advanced/storage) for more information about how these interfaces are implemented. +Runtime storage allows you to store data in your blockchain that is persisted between blocks and can be accessed from +within your runtime logic. Storage should be one of the most critical concerns of a blockchain runtime developer. This +statement is somewhat self-evident, since one of the primary objectives of a blockchain is to provide decentralized +consensus about the state of the underlying storage. Furthermore, well designed storage systems reduce the load on +nodes in the network, which will lower the overhead for participants in your blockchain. Substrate exposes a set of +layered, modular storage APIs that allow runtime developers to make the storage decisions that suit them best. However, +the fundamental principal of blockchain runtime storage is to minimize its use. This document is intended to provide +information and best practices about Substrate's runtime storage interfaces. Please refer to +[the advanced storage documentation](../advanced/storage) for more information about how these interfaces are implemented. ## Storage Items -[The `storage` module in the Substrate Support pallet](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) gives runtime developers access to Substrate's flexible storage APIs: +[The `storage` module in the Substrate Support pallet](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) +gives runtime developers access to Substrate's flexible storage APIs: * [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - A single value * [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - A key-value hash map -* [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - An implementation of a map with two keys that - provides the important ability to efficiently remove all entries that have a common first key -* Iterable Storage Maps - Implementations of [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html) and [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html) whose keys and values can be iterated over +* [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - An + implementation of a map with two keys that provides the important ability to efficiently remove all entries that have + a common first key +* Iterable Storage Maps - Implementations of + [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html) and + [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html) + whose keys and values can be iterated over Any value which can be encoded by the [Parity SCALE codec](../advanced/codec) is supported by these storage APIs. @@ -23,19 +36,33 @@ The type of storage item you use should depend on the logical way in which the v ### Storage Value -This type of storage item should be used for values that are viewed as a single unit by the runtime, whether that is a single primitive value, a single `struct` or a single collection of related items. Although wrapping related items in a shared `struct` is an excellent way to reduce the number of storage reads (an important consideration), at some point the size of the object will begin to incur costs that may outweigh the optimization in storage reads. Storage values can be used to store lists of items, but runtime developers should take care with respect to the size of these lists. Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list in your runtime may result in exceeding the block production time. +This type of storage item should be used for values that are viewed as a single unit by the runtime, whether that is a +single primitive value, a single `struct` or a single collection of related items. Although wrapping related items in +a shared `struct` is an excellent way to reduce the number of storage reads (an important consideration), at some point +the size of the object will begin to incur costs that may outweigh the optimization in storage reads. Storage values +can be used to store lists of items, but runtime developers should take care with respect to the size of these lists. +Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list in your runtime may +result in exceeding the block production time. ### Storage Maps -Substrate maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to give blockchain engineers increased control over the way in which these data structures are stored, Substrate allows developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. +Substrate maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to +give blockchain engineers increased control over the way in which these data structures are stored, Substrate allows +developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing +sets of items whose elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. ### Iterable Storage Maps -The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native lists, they are significantly more costly than lists to iterate over with respect to time. Depending on the hashing algorithm that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. +The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of +data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in +their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native +lists, they are significantly more costly than lists to iterate over with respect to time. Depending on the hashing +algorithm that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. ## Declaring Storage Items -You can use the `decl_storage!` macro to easily create new runtime storage items. Here is an example of what it looks like to declare each type of storage item: +You can use the `decl_storage!` macro to easily create new runtime storage items. Here is an example of what it looks +like to declare each type of storage item: ```rust decl_storage! { @@ -48,11 +75,13 @@ decl_storage! { } ``` -Notice that the last item, the `double_map` specifies the "hasher" (the hash algorithm) that should be used to generate one of its key. Keep reading for more information about the different hashing algorithms and when to use them. +Notice that the last item, the `double_map` specifies the "hasher" (the hash algorithm) that should be used to generate +one of its key. Keep reading for more information about the different hashing algorithms and when to use them. ### Default Values -Substrate allows you to specify a default value that is returned when a storage item's value is not set. The default value does **not** actually occupy runtime storage, but runtime logic will see this value during execution. +Substrate allows you to specify a default value that is returned when a storage item's value is not set. The default +value does **not** actually occupy runtime storage, but runtime logic will see this value during execution. Here is an example of specifying the default value for all items in a map: @@ -66,19 +95,39 @@ decl_storage! { ## Hashing Algorithms -As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. In this document you will find information about the pros and cons of the different approaches to hashing; read [the advanced storage document](../advanced/storage) if you would like a better understanding of _why_ the different algorithms may behave differently. +As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing +algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is +referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described +in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. In this +document you will find information about the pros and cons of the different approaches to hashing; read +[the advanced storage document](../advanced/storage) if you would like a better understanding of _why_ the different +algorithms may behave differently. ### Cryptographic Hashing Algorithms -Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of outputs even if the inputs were the numbers 1 through 10. It is critical to use cryptographic hashing algorithms when users are able to influence the keys of a Storage Map. Failure to do so creates an attack vector that makes it easy for malicious actors to degrade the performance of your blockchain network. An example of a map that should use a cryptographic hash algorithm to generate its keys is a map used to track account balances. In this case, it is important to use a cryptographic hashing algorithm so that an attacker cannot bombard your system with many small transfers to sequential account numbers; without a cryptographic hash algorithm this would create an imbalanced storage structure that would suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic counterparts, which is why Substrate allows developers to select when they are used. +Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing +algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of +outputs even if the inputs were the numbers 1 through 10. It is critical to use cryptographic hashing algorithms when +users are able to influence the keys of a Storage Map. Failure to do so creates an attack vector that makes it easy for +malicious actors to degrade the performance of your blockchain network. An example of a map that should use a cryptographic +hash algorithm to generate its keys is a map used to track account balances. In this case, it is important to use a +cryptographic hashing algorithm so that an attacker cannot bombard your system with many small transfers to sequential +account numbers; without a cryptographic hash algorithm this would create an imbalanced storage structure that would +suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic +counterparts, which is why Substrate allows developers to select when they are used. ### Transparent Hashing Algorithms -A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a key's original unhashed value and verify it (by re-hashing it) if they'd like. It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. +A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a +given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. +This makes it trivial for users to retrieve a key's original unhashed value and verify it (by re-hashing it) if they'd +like. It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is +necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. ### Common Substrate Hashers -This table lists some common hashers used in Substrate and describes which ones are cryptographic and which ones are transparent: +This table lists some common hashers used in Substrate and describes which ones are cryptographic and which ones are +transparent: | Hasher | Cryptographic | Transparent | | ----------------------------------------------------------------------------------------------------- | ------------- | ----------- | @@ -88,15 +137,25 @@ This table lists some common hashers used in Substrate and describes which ones | [TwoX 64 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Twox64Concat.html) | | X | | [Identity](https://substrate.dev/rustdocs/master/frame_support/struct.Identity.html) | | | -The Identity hasher exposes a hashing algorithm that has an output equal to its input (the identity function).This type of hasher should only be used when the starting key is already a cryptographic hash. +The Identity hasher exposes a hashing algorithm that has an output equal to its input (the identity function).This type +of hasher should only be used when the starting key is already a cryptographic hash. ## Querying Storage -Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query your blockchain's runtime storage. One thing that will be important to understand before proceeding is that the Substrate runtime storage APIs are layers of abstraction that have been built upon a simple key-value database. When you use the Substrate RPC to query runtime storage, you only need to provide a single piece of information: the storage key. The rest of this section will go over the basics of how to calculate storage keys for the different types of storage items. If you'd like to learn more about how Substrate uses a key-value database to implement the different kinds of storage items, refer to [the advanced storage documentation](../advanced/storage). +Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query your +blockchain's runtime storage. One thing that will be important to understand before proceeding is that the Substrate +runtime storage APIs are layers of abstraction that have been built upon a simple key-value database. When you use the +Substrate RPC to query runtime storage, you only need to provide a single piece of information: the storage key. The rest +of this section will go over the basics of how to calculate storage keys for the different types of storage items. If +you'd like to learn more about how Substrate uses a key-value database to implement the different kinds of storage items, +refer to [the advanced storage documentation](../advanced/storage). ### Storage Value Keys -To calculate the key for a simple Storage Value, take the TwoX 128 hash of the name of the module that contains the Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage Value item named [`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): +To calculate the key for a simple Storage Value, take the TwoX 128 hash of the name of the module that contains the +Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the +[Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage Value item named +[`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): ``` twox_128("Sudo") = "0x5c0d1176a568c1f92944340dbfed9e9c" @@ -104,21 +163,34 @@ twox_128("Key) = "0x530ebca703c85910e7164cb7d1c9e47b" twox_128("Sudo") + twox_128("Key") = "0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b" ``` -If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo module's `Key` Storage Value could be represented as: +If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo module's `Key` Storage +Value could be represented as: ``` state_getStorage("0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" ``` -In this case, the value that is returned (`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is Alice's [SCALE](../advanced/codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). +In this case, the value that is returned (`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is +Alice's [SCALE](../advanced/codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). -You may have noticed that the non-cryptographic TwoX 128 hash algorithm is used to generate Storage Value keys. This is because it is not necessary to pay the performance costs associated with a cryptographic hash function since the input to the hash function (the names of the module and storage item) are determined by you, the runtime developer, and not by potentially malicious users of your blockchain. +You may have noticed that the non-cryptographic TwoX 128 hash algorithm is used to generate Storage Value keys. This is +because it is not necessary to pay the performance costs associated with a cryptographic hash function since the input +to the hash function (the names of the module and storage item) are determined by you, the runtime developer, and not by +potentially malicious users of your blockchain. ### Storage Map Keys -Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double Maps), append the hash of the first map key followed by the hash of the second map key to the map's storage key. Remember, Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you will need to make sure to use the correct hashing algorithm (the one that was declared in [the `decl_storage` macro](#Declaring-Storage-Items)) when determining the hashed keys for the elements in a map. +Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains +the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply +append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double +Maps), append the hash of the first map key followed by the hash of the second map key to the map's storage key. Remember, +Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you will need to make sure +to use the correct hashing algorithm (the one that was declared in [the `decl_storage` macro](#Declaring-Storage-Items)) +when determining the hashed keys for the elements in a map. -Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using [the transparent Blake2 128 Concat hashing algorithm](#Transparent-Hashing-Algorithms): +Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the +balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using +[the transparent Blake2 128 Concat hashing algorithm](#Transparent-Hashing-Algorithms): ``` twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" @@ -130,7 +202,19 @@ blake2_128_concat("0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56 state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0x0000a0dec5adc9353600000000000000" ``` -The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the [SCALE](../advanced/codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat is [a transparent hashing algorithm as describe above](#Transparent-Hashing-Algorithms). Although the above example may make this characteristic seem superfluous, its utility becomes more apparent when the goal is to iterate over the keys in a map (as opposed to retrieving the value associated with a single key). The ability to iterate over the keys in a map is a common requirement in order to allow _people_ to use the map in a way that seems natural (such as UIs): first, a user is presented with a list of elements in the map, then, that user can select the element that they are interested in and query the map for more details about that particular element. Here is another example that uses the same example Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing algorithm in a module named "Balances") that will demonstrate using the Substrate RPC to query a Storage Map for its list of keys via the `state_getKeys` RPC endpoint: +The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the +[SCALE](../advanced/codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice +that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` +function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat +is [a transparent hashing algorithm as describe above](#Transparent-Hashing-Algorithms). Although the above example may +make this characteristic seem superfluous, its utility becomes more apparent when the goal is to iterate over the keys +in a map (as opposed to retrieving the value associated with a single key). The ability to iterate over the keys in a +map is a common requirement in order to allow _people_ to use the map in a way that seems natural (such as UIs): first, +a user is presented with a list of elements in the map, then, that user can select the element that they are interested +in and query the map for more details about that particular element. Here is another example that uses the same example +Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing algorithm in a module named "Balances") +that will demonstrate using the Substrate RPC to query a Storage Map for its list of keys via the `state_getKeys` RPC +endpoint: ``` twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" @@ -143,7 +227,24 @@ state_getKeys("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b ] ``` -Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be directly used as input for the RPC's `state_getStorage` endpoint. In fact, the first element in the example list above is equal to the input used for the `state_getStorage` query in the previous example (the one used to find the balance for `Alice`). Because the map that these keys belong to uses a transparent hashing algorithm to generate its keys, it is possible to determine the account associated with the second element in the list. Notice that each element in the list is a hexadecimal value that begins with the same 64 characters; this is because each list element represents a key in the same map, and that map is identified by concatenating two TwoX 128 hashes, each of which are 128-bits or 32 hexadecimal characters. After discarding this portion of the second element in the list, you are left with `0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. You saw in the previous example that this represents the Blake2 128 Concat hash of some [SCALE](../advanced/codec)-encoded account ID. As described above, the Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing algorithm. In this example, after you remove the first 32 hexadecimal characters that represent a Blake2 128 hash (i.e. `0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value `0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](../advanced/codec)-encoded account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account ID for the familiar `Alice_Stash` account. +Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be directly used as input +for the RPC's `state_getStorage` endpoint. In fact, the first element in the example list above is equal to the input +used for the `state_getStorage` query in the previous example (the one used to find the balance for `Alice`). Because +the map that these keys belong to uses a transparent hashing algorithm to generate its keys, it is possible to determine +the account associated with the second element in the list. Notice that each element in the list is a hexadecimal value +that begins with the same 64 characters; this is because each list element represents a key in the same map, and that +map is identified by concatenating two TwoX 128 hashes, each of which are 128-bits or 32 hexadecimal characters. After +discarding this portion of the second element in the list, you are left with +`0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. You saw in the +previous example that this represents the Blake2 128 Concat hash of some [SCALE](../advanced/codec)-encoded account ID. +As described above, the Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's +input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat +hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing +algorithm. In this example, after you remove the first 32 hexadecimal characters that represent a Blake2 128 hash (i.e. +`0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value +`0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](../advanced/codec)-encoded +account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account +ID for the familiar `Alice_Stash` account. ## Child Storage Tries @@ -169,14 +270,13 @@ TODO ### References -- Visit the reference docs for the - [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) - more details possible storage declarations. +* Visit the reference docs for the + [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) more details + possible storage declarations. -- Visit the reference docs for +* Visit the reference docs for [StorageValue](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html), - [StorageMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html), - [StorageLinkedMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageLinkedMap.html), - and - [StorageDoubleMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - to learn more about their API. + [StorageMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html), + [StorageLinkedMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageLinkedMap.html), and + [StorageDoubleMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) to learn + more about their APIs. From b99117d48134f7c98608d28087b8cc99878dcd5a Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 13 Apr 2020 09:42:06 -0700 Subject: [PATCH 04/26] Document methods for Storage Items --- current/runtime/storage.md | 89 +++++++++++++++++++++++++++++++++++++- 1 file changed, 88 insertions(+), 1 deletion(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index f2e0446..c5518fd 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -44,6 +44,28 @@ can be used to store lists of items, but runtime developers should take care wit Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list in your runtime may result in exceeding the block production time. +#### Methods + +Refer to the Storage Value documentation for +[a comprehensive list of the methods that Storage Values expose](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#required-methods). +Some of the most important methods are summarized here: + +##### `get()` + +Load the value from storage. + +##### `put(val)` + +Store the provided value. + +##### `mutate(fn)` + +Mutate the value with the provided function. + +##### `take()` + +Remove the value from storage. + ### Storage Maps Substrate maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to @@ -51,6 +73,28 @@ give blockchain engineers increased control over the way in which these data str developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. +#### Methods + +[Storage Maps expose an API](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#required-methods) +that is similar to that of Storage Values. The selected methods referenced below all use a single key; use two keys for +[Storage Double Maps](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#required-methods): + +##### `get(key)` + +Load the value associated with the provided key from storage. + +##### `store(key, val)` + +Store the provided value by associating it with the given key. + +##### `mutate(key, fn)` + +Use the provided function to mutate the value associated with the given key. + +##### `take(key)` + +Remove the value associated with the given key from storage. + ### Iterable Storage Maps The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of @@ -59,6 +103,26 @@ their entirety within the runtime. Furthermore, because maps are comprised of mo lists, they are significantly more costly than lists to iterate over with respect to time. Depending on the hashing algorithm that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. +#### Methods + +[Iterable Storage Maps expose the following methods](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#required-methods) +in addition to the other map methods: + +##### `iter()` + +Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined +results. + +##### `drain()` + +Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while +doing this, you'll get undefined results. + +##### `translate(fn)` + +Use the provided function to translate all elements of the map, in no particular order. To remove an element from the +map, return `None` from the translation function. + ## Declaring Storage Items You can use the `decl_storage!` macro to easily create new runtime storage items. Here is an example of what it looks @@ -76,7 +140,22 @@ decl_storage! { ``` Notice that the last item, the `double_map` specifies the "hasher" (the hash algorithm) that should be used to generate -one of its key. Keep reading for more information about the different hashing algorithms and when to use them. +one of its key. Keep reading for [more information about the different hashing algorithms](#Hashing-Algorithms) and when +to use them. + +### Getter Functions + +The `decl_storage` macro makes it easy to define and implement getter functions for your storage items. + +Here is an example of adding a getter function for a simple Storage Value: + +```rust +decl_storage! { + trait Store for Module as Example { + pub SomeValue get(someValue): u64; + } +} +``` ### Default Values @@ -93,6 +172,14 @@ decl_storage! { } ``` +### Genesis Config + +You can define +[an optional `GenesisConfig`](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html#genesisconfig) +struct in order to initialize Storage Items in the genesis block of your blockchain. + +// TODO + ## Hashing Algorithms As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing From 230932bfbd65cddb78ec7c9bc98a6009f91a6cad Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 14 Apr 2020 23:48:53 -0700 Subject: [PATCH 05/26] Small changes to runtime storage document --- current/runtime/storage.md | 67 +++++++++++++++++++------------------- 1 file changed, 33 insertions(+), 34 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index c5518fd..9772300 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -17,22 +17,20 @@ information and best practices about Substrate's runtime storage interfaces. Ple ## Storage Items [The `storage` module in the Substrate Support pallet](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) -gives runtime developers access to Substrate's flexible storage APIs: +gives runtime developers access to Substrate's flexible storage APIs. Any value which can be encoded by the +[Parity SCALE codec](../advanced/codec) is supported by these storage APIs: * [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - A single value * [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - A key-value hash map * [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - An implementation of a map with two keys that provides the important ability to efficiently remove all entries that have - a common first key + a common first key * Iterable Storage Maps - Implementations of [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html) and - [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html) - whose keys and values can be iterated over + [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html) + whose keys and values can be iterated over -Any value which can be encoded by the [Parity SCALE codec](../advanced/codec) is supported by these -storage APIs. - -The type of storage item you use should depend on the logical way in which the value will be used by your runtime. +The type of Storage Item you select should depend on the logical way in which the value will be used by your runtime. ### Storage Value @@ -68,7 +66,7 @@ Remove the value from storage. ### Storage Maps -Substrate maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to +Storage Maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to give blockchain engineers increased control over the way in which these data structures are stored, Substrate allows developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. @@ -100,8 +98,9 @@ Remove the value associated with the given key from storage. The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native -lists, they are significantly more costly than lists to iterate over with respect to time. Depending on the hashing -algorithm that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. +lists, they are significantly more costly than lists to iterate over with respect to time. Depending on +[the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a map's keys, you may be able to +iterate across its keys as well as its values. #### Methods @@ -125,8 +124,8 @@ map, return `None` from the translation function. ## Declaring Storage Items -You can use the `decl_storage!` macro to easily create new runtime storage items. Here is an example of what it looks -like to declare each type of storage item: +You can use [the `decl_storage` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) to +easily create new runtime storage items. Here is an example of what it looks like to declare each type of storage item: ```rust decl_storage! { @@ -139,20 +138,20 @@ decl_storage! { } ``` -Notice that the last item, the `double_map` specifies the "hasher" (the hash algorithm) that should be used to generate +Notice that the last item (the `double_map`) specifies the "hasher" (the hash algorithm) that should be used to generate one of its key. Keep reading for [more information about the different hashing algorithms](#Hashing-Algorithms) and when to use them. -### Getter Functions +### Custom Getter Method Names -The `decl_storage` macro makes it easy to define and implement getter functions for your storage items. +The `decl_storage` macro makes it easy to provide a custom name for a Storage Item's `get()` method. -Here is an example of adding a getter function for a simple Storage Value: +Here is an example of renaming the `get()` function of a Storage Value named `SomeValue` to `some_value()`: ```rust decl_storage! { trait Store for Module as Example { - pub SomeValue get(someValue): u64; + pub SomeValue get(some_value): u64; } } ``` @@ -207,13 +206,13 @@ counterparts, which is why Substrate allows developers to select when they are u A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. -This makes it trivial for users to retrieve a key's original unhashed value and verify it (by re-hashing it) if they'd -like. It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is +This makes it trivial for users to retrieve a key's original unhashed value and verify it if they'd like (by re-hashing +it). It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. ### Common Substrate Hashers -This table lists some common hashers used in Substrate and describes which ones are cryptographic and which ones are +This table lists some common hashers used in Substrate and denotes those that are cryptographic and those that are transparent: | Hasher | Cryptographic | Transparent | @@ -224,7 +223,7 @@ transparent: | [TwoX 64 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Twox64Concat.html) | | X | | [Identity](https://substrate.dev/rustdocs/master/frame_support/struct.Identity.html) | | | -The Identity hasher exposes a hashing algorithm that has an output equal to its input (the identity function).This type +The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the identity function).This type of hasher should only be used when the starting key is already a cryptographic hash. ## Querying Storage @@ -233,8 +232,8 @@ Blockchains that are built with Substrate expose a remote procedure call (RPC) s blockchain's runtime storage. One thing that will be important to understand before proceeding is that the Substrate runtime storage APIs are layers of abstraction that have been built upon a simple key-value database. When you use the Substrate RPC to query runtime storage, you only need to provide a single piece of information: the storage key. The rest -of this section will go over the basics of how to calculate storage keys for the different types of storage items. If -you'd like to learn more about how Substrate uses a key-value database to implement the different kinds of storage items, +of this section will go over the basics of how to calculate storage keys for the different types of Storage Items. If +you'd like to learn more about how Substrate uses a key-value database to implement the different kinds of Storage Items, refer to [the advanced storage documentation](../advanced/storage). ### Storage Value Keys @@ -270,10 +269,10 @@ potentially malicious users of your blockchain. Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double -Maps), append the hash of the first map key followed by the hash of the second map key to the map's storage key. Remember, -Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you will need to make sure -to use the correct hashing algorithm (the one that was declared in [the `decl_storage` macro](#Declaring-Storage-Items)) -when determining the hashed keys for the elements in a map. +Maps), append the hash of the first map key followed by the hash of the second map key to the Storage Double Map's storage +key. Remember, Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you will need +to make sure to use the correct hashing algorithm (the one that was declared in +[the `decl_storage` macro](#Declaring-Storage-Items)) when determining the hashed keys for the elements in a map. Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using @@ -327,7 +326,7 @@ previous example that this represents the Blake2 128 Concat hash of some [SCALE] As described above, the Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing -algorithm. In this example, after you remove the first 32 hexadecimal characters that represent a Blake2 128 hash (i.e. +algorithm. In this example, after you remove the first 32 hexadecimal characters that represent the Blake2 128 hash (i.e. `0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value `0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](../advanced/codec)-encoded account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account @@ -359,11 +358,11 @@ TODO * Visit the reference docs for the [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) more details - possible storage declarations. + possible storage declarations. * Visit the reference docs for [StorageValue](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html), - [StorageMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html), - [StorageLinkedMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageLinkedMap.html), and - [StorageDoubleMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) to learn - more about their APIs. + [StorageMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html), + [StorageLinkedMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageLinkedMap.html), and + [StorageDoubleMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) to learn + more about their APIs. From afde25b691c8c163a91c87778fd54d7ac1df867b Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Sun, 19 Apr 2020 08:20:54 -0700 Subject: [PATCH 06/26] Address @thiolliere PR comments * Deemphasize redudancy with Rust docs by collapsing sections about Storage Item APIs * Distinguish APIs for StorageMap and StorageDoubleMap * Clarify StorageMap implementation --- current/runtime/storage.md | 93 ++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 53 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 9772300..c17cafb 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -48,50 +48,40 @@ Refer to the Storage Value documentation for [a comprehensive list of the methods that Storage Values expose](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#required-methods). Some of the most important methods are summarized here: -##### `get()` - -Load the value from storage. - -##### `put(val)` - -Store the provided value. - -##### `mutate(fn)` - -Mutate the value with the provided function. - -##### `take()` - -Remove the value from storage. +* [`get()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.get) - Load + the value from storage. +* [`put(val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.put) - Store + the provided value. +* [`mutate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.mutate) - + Mutate the value with the provided function. +* [`take()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.take) - Remove + the value from storage. ### Storage Maps -Storage Maps are implemented as hash maps, which is a pattern that should be familiar to most developers. In order to -give blockchain engineers increased control over the way in which these data structures are stored, Substrate allows -developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing -sets of items whose elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. +Storage Maps are implemented as maps with hashed keys, which is a pattern that should be familiar to most developers. +In order to give blockchain engineers increased control over the way in which these data structures are stored, +Substrate allows developers to select the hashing algorithm that is used to generate map keys. Map data structures are +ideal for managing sets of items whose elements will be accessed randomly, as opposed to iterating over them +sequentially in their entirety. #### Methods [Storage Maps expose an API](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#required-methods) -that is similar to that of Storage Values. The selected methods referenced below all use a single key; use two keys for -[Storage Double Maps](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#required-methods): - -##### `get(key)` - -Load the value associated with the provided key from storage. - -##### `store(key, val)` - -Store the provided value by associating it with the given key. - -##### `mutate(key, fn)` - -Use the provided function to mutate the value associated with the given key. - -##### `take(key)` - -Remove the value associated with the given key from storage. +that is similar to that of Storage Values. + +* `get` - Load the value associated with the provided key from storage. Docs: + [`StorageMap#get(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.get), + [`StorageDoubleMap#get(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.get) +* `insert` - Store the provided value by associating it with the given key. Docs: + [`StorageMap#insert(key, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.insert), + [`StorageDoubleMap#insert(key1, key2, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.insert) +* `mutate` - Use the provided function to mutate the value associated with the given key. Docs: + [`StorageMap#mutate(key, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.mutate), + [`StorageDoubleMap#mutate(key1, key2, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.mutate) +* `take` - Remove the value associated with the given key from storage. Docs: + [`StorageMap#take(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.take), + [`StorageDoubleMap#take(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) ### Iterable Storage Maps @@ -105,22 +95,19 @@ iterate across its keys as well as its values. #### Methods [Iterable Storage Maps expose the following methods](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#required-methods) -in addition to the other map methods: - -##### `iter()` - -Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined -results. - -##### `drain()` - -Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while -doing this, you'll get undefined results. - -##### `translate(fn)` - -Use the provided function to translate all elements of the map, in no particular order. To remove an element from the -map, return `None` from the translation function. +in addition to the other map methods. Note that for Iterable Storage Double Maps, the `iter` and `drain` methods require a parameter, i.e. the first key: + +* `iter` - Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined + results. Docs: + [IterableStorageMap#iter()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), + [IterableStorageDoubleMap#iter(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) +* `drain` - Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while + doing this, you'll get undefined results. Docs: + [IterableStorageMap#drain()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), [IterableStorageDoubleMap#drain(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) +* `translate` - Use the provided function to translate all elements of the map, in no particular order. To remove an element from the + map, return `None` from the translation function. Docs: + [IterableStorageMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), + [IterableStorageDoubleMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) ## Declaring Storage Items From fc74903eac7e2687fae8059a11e3e12897c2d7ec Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 20 Apr 2020 05:58:03 -0700 Subject: [PATCH 07/26] Address @thiolliere PR comments * Clarify Storage Map implementation * Clarify iterable storage map principals --- current/runtime/storage.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index c17cafb..189a3e0 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -60,10 +60,13 @@ Some of the most important methods are summarized here: ### Storage Maps Storage Maps are implemented as maps with hashed keys, which is a pattern that should be familiar to most developers. -In order to give blockchain engineers increased control over the way in which these data structures are stored, -Substrate allows developers to select the hashing algorithm that is used to generate map keys. Map data structures are -ideal for managing sets of items whose elements will be accessed randomly, as opposed to iterating over them -sequentially in their entirety. +Unlike traditional hash maps, though, [Storage Maps in Substrate are simple key-value stores](../advanced/storage) that +do not take key collision into account. The hashing algorithms that Substrate supplies are designed so that runtime +developers do not need to worry about key collisions, but the implementation of Substrate Storage Maps does becomes +important when [querying storage](#Querying-Storage) to read the elements of a Storage Map. In order to give blockchain +engineers increased control over the way in which these data structures are stored, Substrate allows developers to select +the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose +elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. #### Methods @@ -88,7 +91,11 @@ that is similar to that of Storage Values. The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native -lists, they are significantly more costly than lists to iterate over with respect to time. Depending on +lists, they are significantly more costly than lists to iterate over with respect to time. This is not to say that it +is "wrong" to iterate over maps in your runtime; in general Substrate focuses on "first principals" as opposed to hard +and fast rules of right and wrong. Being efficient within the runtime of a blockchain is an important first principal +of Substrate and this information is designed to help you understand _all_ of Substrate's storage capabilities and use +them in a way that respects the important first principals around which they were designed. Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. From 8c0221290566b413e06180b3c72b41d54332e15b Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 21 Apr 2020 03:37:25 -0700 Subject: [PATCH 08/26] Formatting --- current/runtime/storage.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 189a3e0..100bc9c 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -110,7 +110,8 @@ in addition to the other map methods. Note that for Iterable Storage Double Maps [IterableStorageDoubleMap#iter(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) * `drain` - Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while doing this, you'll get undefined results. Docs: - [IterableStorageMap#drain()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), [IterableStorageDoubleMap#drain(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) + [IterableStorageMap#drain()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), + [IterableStorageDoubleMap#drain(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) * `translate` - Use the provided function to translate all elements of the map, in no particular order. To remove an element from the map, return `None` from the translation function. Docs: [IterableStorageMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), From 5d3503cce257d3f8bd559c98d7c498a9bf70bdd7 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 21 Apr 2020 03:51:45 -0700 Subject: [PATCH 09/26] Clarify the `get` extension to `decl_storage` --- current/runtime/storage.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 100bc9c..065448e 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -137,16 +137,23 @@ Notice that the last item (the `double_map`) specifies the "hasher" (the hash al one of its key. Keep reading for [more information about the different hashing algorithms](#Hashing-Algorithms) and when to use them. -### Custom Getter Method Names +### Getter Methods -The `decl_storage` macro makes it easy to provide a custom name for a Storage Item's `get()` method. +The `decl_storage` macro provides an optional `get` extension that can be used to implement a getter method for a storage +item on the module that contains that storage item; the extension takes the desired name of the getter function as an +argument. If you omit this optional extension, you will still be able to access the storage item's value, but you will +not be able to do so by way of a getter method implemented on the module; instead, you will need to need to use +[the storage item's `get` method](#Methods). Keep in mind that the optional `get` extension only impacts the way that +the storage item can be accessed from within Substrate code; you will always be able to +[query the storage of your runtime](#Querying-Storage) to get the value of a storage item. -Here is an example of renaming the `get()` function of a Storage Value named `SomeValue` to `some_value()`: +Here is an example that implements a getter method named `some_value` for a Storage Value named `SomeValue`. This module +would now have access to a `Self::some_value()` method in addition to the `SomeValue::get()` method: ```rust decl_storage! { trait Store for Module as Example { - pub SomeValue get(some_value): u64; + pub SomeValue get(fn some_value): u64; } } ``` From 3918880c3bb3318fdf18c58f1e47064db8edd3ea Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Thu, 23 Apr 2020 01:45:21 -0700 Subject: [PATCH 10/26] Clarifications and improvements from @joepetrowski --- current/runtime/storage.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 065448e..3819e55 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -10,14 +10,14 @@ statement is somewhat self-evident, since one of the primary objectives of a blo consensus about the state of the underlying storage. Furthermore, well designed storage systems reduce the load on nodes in the network, which will lower the overhead for participants in your blockchain. Substrate exposes a set of layered, modular storage APIs that allow runtime developers to make the storage decisions that suit them best. However, -the fundamental principal of blockchain runtime storage is to minimize its use. This document is intended to provide +the fundamental principle of blockchain runtime storage is to minimize its use. This document is intended to provide information and best practices about Substrate's runtime storage interfaces. Please refer to [the advanced storage documentation](../advanced/storage) for more information about how these interfaces are implemented. ## Storage Items -[The `storage` module in the Substrate Support pallet](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) -gives runtime developers access to Substrate's flexible storage APIs. Any value which can be encoded by the +The `storage` module in [FRAME Support](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) +gives runtime developers access to Substrate's flexible storage APIs. Any value that can be encoded by the [Parity SCALE codec](../advanced/codec) is supported by these storage APIs: * [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - A single value @@ -35,7 +35,7 @@ The type of Storage Item you select should depend on the logical way in which th ### Storage Value This type of storage item should be used for values that are viewed as a single unit by the runtime, whether that is a -single primitive value, a single `struct` or a single collection of related items. Although wrapping related items in +single primitive value, a single `struct`, or a single collection of related items. Although wrapping related items in a shared `struct` is an excellent way to reduce the number of storage reads (an important consideration), at some point the size of the object will begin to incur costs that may outweigh the optimization in storage reads. Storage values can be used to store lists of items, but runtime developers should take care with respect to the size of these lists. @@ -62,7 +62,7 @@ Some of the most important methods are summarized here: Storage Maps are implemented as maps with hashed keys, which is a pattern that should be familiar to most developers. Unlike traditional hash maps, though, [Storage Maps in Substrate are simple key-value stores](../advanced/storage) that do not take key collision into account. The hashing algorithms that Substrate supplies are designed so that runtime -developers do not need to worry about key collisions, but the implementation of Substrate Storage Maps does becomes +developers do not need to worry about key collisions, but the implementation of Substrate Storage Maps does become important when [querying storage](#Querying-Storage) to read the elements of a Storage Map. In order to give blockchain engineers increased control over the way in which these data structures are stored, Substrate allows developers to select the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose @@ -92,10 +92,10 @@ The Substrate storage API provides iterable map implementations. Because maps ar data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native lists, they are significantly more costly than lists to iterate over with respect to time. This is not to say that it -is "wrong" to iterate over maps in your runtime; in general Substrate focuses on "first principals" as opposed to hard -and fast rules of right and wrong. Being efficient within the runtime of a blockchain is an important first principal +is "wrong" to iterate over maps in your runtime; in general Substrate focuses on "first principles" as opposed to hard +and fast rules of right and wrong. Being efficient within the runtime of a blockchain is an important first principle of Substrate and this information is designed to help you understand _all_ of Substrate's storage capabilities and use -them in a way that respects the important first principals around which they were designed. Depending on +them in a way that respects the important first principles around which they were designed. Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a map's keys, you may be able to iterate across its keys as well as its values. @@ -359,8 +359,8 @@ TODO ### References * Visit the reference docs for the - [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) more details - possible storage declarations. + [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) for more details + about the available storage declarations. * Visit the reference docs for [StorageValue](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html), From f20df86f06c57952e9286283aea84c19fec77e7c Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 27 Apr 2020 11:56:39 -0700 Subject: [PATCH 11/26] Remove iterable storage maps from top-level listing of storage items Per suggestion of @thiolliere and @shawntabrizi --- current/runtime/storage.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 3819e55..046d841 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -25,10 +25,6 @@ gives runtime developers access to Substrate's flexible storage APIs. Any value * [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - An implementation of a map with two keys that provides the important ability to efficiently remove all entries that have a common first key -* Iterable Storage Maps - Implementations of - [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html) and - [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html) - whose keys and values can be iterated over The type of Storage Item you select should depend on the logical way in which the value will be used by your runtime. From c8d17568a86db1743cbe9f1e1b61c4b0328964e8 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 27 Apr 2020 17:53:32 -0700 Subject: [PATCH 12/26] Rework Iterable Storage Map documentation --- current/runtime/storage.md | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 046d841..2a482be 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -82,23 +82,22 @@ that is similar to that of Storage Values. [`StorageMap#take(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.take), [`StorageDoubleMap#take(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) -### Iterable Storage Maps - -The Substrate storage API provides iterable map implementations. Because maps are often used to track unbounded sets of -data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in -their entirety within the runtime. Furthermore, because maps are comprised of more layers of indirection than native -lists, they are significantly more costly than lists to iterate over with respect to time. This is not to say that it -is "wrong" to iterate over maps in your runtime; in general Substrate focuses on "first principles" as opposed to hard -and fast rules of right and wrong. Being efficient within the runtime of a blockchain is an important first principle -of Substrate and this information is designed to help you understand _all_ of Substrate's storage capabilities and use -them in a way that respects the important first principles around which they were designed. Depending on -[the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a map's keys, you may be able to -iterate across its keys as well as its values. - -#### Methods - -[Iterable Storage Maps expose the following methods](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#required-methods) -in addition to the other map methods. Note that for Iterable Storage Double Maps, the `iter` and `drain` methods require a parameter, i.e. the first key: +#### Iterable Storage Maps + +Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a map's keys, you may +be able to iterate across its keys and values. Because maps are often used to track unbounded sets of data (account +balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety +within the runtime. Furthermore, because maps are comprised of more layers of indirection than native lists, they are +significantly more costly than lists to iterate over with respect to time. This is not to say that it is "wrong" to +iterate over maps in your runtime; in general Substrate focuses on "first principles" as opposed to hard and fast rules +of right and wrong. Being efficient within the runtime of a blockchain is an important first principle of Substrate and +this information is designed to help you understand _all_ of Substrate's storage capabilities and use them in a way +that respects the important first principles around which they were designed. + +##### Iterable Storage Map Methods + +Substrate's Iterable Storage Map interfaces define the following methods. Note that for Iterable Storage Double Maps, +the `iter` and `drain` methods require a parameter, i.e. the first key: * `iter` - Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined results. Docs: From 5dae1818792d17cbaec995e1c04292247809afd0 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 28 Apr 2020 06:26:21 -0700 Subject: [PATCH 13/26] Structural changes per @shawntabrizi --- current/runtime/storage.md | 252 ++++++++++++------------------------- 1 file changed, 79 insertions(+), 173 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 2a482be..81067e2 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -36,7 +36,8 @@ a shared `struct` is an excellent way to reduce the number of storage reads (an the size of the object will begin to incur costs that may outweigh the optimization in storage reads. Storage values can be used to store lists of items, but runtime developers should take care with respect to the size of these lists. Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list in your runtime may -result in exceeding the block production time. +result in exceeding the block production time - if this occurs your blockchain will stop producing blocks, which means +that it will stop functioning. #### Methods @@ -55,14 +56,12 @@ Some of the most important methods are summarized here: ### Storage Maps -Storage Maps are implemented as maps with hashed keys, which is a pattern that should be familiar to most developers. -Unlike traditional hash maps, though, [Storage Maps in Substrate are simple key-value stores](../advanced/storage) that -do not take key collision into account. The hashing algorithms that Substrate supplies are designed so that runtime -developers do not need to worry about key collisions, but the implementation of Substrate Storage Maps does become -important when [querying storage](#Querying-Storage) to read the elements of a Storage Map. In order to give blockchain -engineers increased control over the way in which these data structures are stored, Substrate allows developers to select -the hashing algorithm that is used to generate map keys. Map data structures are ideal for managing sets of items whose -elements will be accessed randomly, as opposed to iterating over them sequentially in their entirety. +Map data structures are ideal for managing sets of items whose elements will be accessed randomly, as opposed to +iterating over them sequentially in their entirety. Storage Maps in Substrate are implemented as key-value hash maps, +which is a pattern that should be familiar to most developers. In order to give blockchain engineers increased control, +Substrate allows developers to select [the hashing algorithm](#Hashing-Algorithms) that is used to generate a map's +keys. Refer to [the advanced storage documentation](../advanced/storage) to learn more about how Substrate's Storage +Maps are implemented. #### Methods @@ -112,71 +111,7 @@ the `iter` and `drain` methods require a parameter, i.e. the first key: [IterableStorageMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), [IterableStorageDoubleMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) -## Declaring Storage Items - -You can use [the `decl_storage` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) to -easily create new runtime storage items. Here is an example of what it looks like to declare each type of storage item: - -```rust -decl_storage! { - trait Store for Module as Example { - pub SomeValue: u64; - pub SomeMap: map u64 => u64; - pub SomeLinkedMap: linked_map u64 => u64; - pub SomeDoubleMap: double_map u64, blake2_256(u64) => u64; - } -} -``` - -Notice that the last item (the `double_map`) specifies the "hasher" (the hash algorithm) that should be used to generate -one of its key. Keep reading for [more information about the different hashing algorithms](#Hashing-Algorithms) and when -to use them. - -### Getter Methods - -The `decl_storage` macro provides an optional `get` extension that can be used to implement a getter method for a storage -item on the module that contains that storage item; the extension takes the desired name of the getter function as an -argument. If you omit this optional extension, you will still be able to access the storage item's value, but you will -not be able to do so by way of a getter method implemented on the module; instead, you will need to need to use -[the storage item's `get` method](#Methods). Keep in mind that the optional `get` extension only impacts the way that -the storage item can be accessed from within Substrate code; you will always be able to -[query the storage of your runtime](#Querying-Storage) to get the value of a storage item. - -Here is an example that implements a getter method named `some_value` for a Storage Value named `SomeValue`. This module -would now have access to a `Self::some_value()` method in addition to the `SomeValue::get()` method: - -```rust -decl_storage! { - trait Store for Module as Example { - pub SomeValue get(fn some_value): u64; - } -} -``` - -### Default Values - -Substrate allows you to specify a default value that is returned when a storage item's value is not set. The default -value does **not** actually occupy runtime storage, but runtime logic will see this value during execution. - -Here is an example of specifying the default value for all items in a map: - -```rust -decl_storage! { - trait Store for Module as Example { - pub SomeMap: map u64 => u64 = 1337; - } -} -``` - -### Genesis Config - -You can define -[an optional `GenesisConfig`](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html#genesisconfig) -struct in order to initialize Storage Items in the genesis block of your blockchain. - -// TODO - -## Hashing Algorithms +#### Hashing Algorithms As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is @@ -186,7 +121,7 @@ document you will find information about the pros and cons of the different appr [the advanced storage document](../advanced/storage) if you would like a better understanding of _why_ the different algorithms may behave differently. -### Cryptographic Hashing Algorithms +##### Cryptographic Hashing Algorithms Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of @@ -199,15 +134,16 @@ account numbers; without a cryptographic hash algorithm this would create an imb suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic counterparts, which is why Substrate allows developers to select when they are used. -### Transparent Hashing Algorithms +##### Transparent Hashing Algorithms A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a key's original unhashed value and verify it if they'd like (by re-hashing it). It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is -necessary to use a transparent hashing algorithm if you would like to iterate over the keys in a map. +_necessary_ to use a transparent hashing algorithm if you would like access [iterable map](#Iterable-Storage-Maps) +capabilities. -### Common Substrate Hashers +##### Common Substrate Hashers This table lists some common hashers used in Substrate and denotes those that are cryptographic and those that are transparent: @@ -223,111 +159,78 @@ transparent: The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the identity function).This type of hasher should only be used when the starting key is already a cryptographic hash. -## Querying Storage - -Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query your -blockchain's runtime storage. One thing that will be important to understand before proceeding is that the Substrate -runtime storage APIs are layers of abstraction that have been built upon a simple key-value database. When you use the -Substrate RPC to query runtime storage, you only need to provide a single piece of information: the storage key. The rest -of this section will go over the basics of how to calculate storage keys for the different types of Storage Items. If -you'd like to learn more about how Substrate uses a key-value database to implement the different kinds of Storage Items, -refer to [the advanced storage documentation](../advanced/storage). - -### Storage Value Keys +## Declaring Storage Items -To calculate the key for a simple Storage Value, take the TwoX 128 hash of the name of the module that contains the -Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the -[Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage Value item named -[`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): +You can use [the `decl_storage` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) to +easily create new runtime storage items. Here is an example of what it looks like to declare each type of storage item: -``` -twox_128("Sudo") = "0x5c0d1176a568c1f92944340dbfed9e9c" -twox_128("Key) = "0x530ebca703c85910e7164cb7d1c9e47b" -twox_128("Sudo") + twox_128("Key") = "0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b" +```rust +decl_storage! { + trait Store for Module as Example { + pub SomePrimitiveValue get(fn some_primitive_value): u32; + // complex types are prefaced by T:: + pub SomeComplexValue: T::AccountId; + pub SomeMap get(fn some_map): map hasher(blake2_128_concat) str => u32; + pub SomeDoubleMap: double_map hasher(blake2_128_concat) str, hasher(blake2_128_concat) str => u32; + } +} ``` -If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo module's `Key` Storage -Value could be represented as: +Notice that the map storage items specify [the hashing algorithm](#Hashing-Algorithms) that will be used. -``` -state_getStorage("0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" -``` +### Getter Methods -In this case, the value that is returned (`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is -Alice's [SCALE](../advanced/codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). +The `decl_storage` macro provides an optional `get` extension that can be used to implement a getter method for a storage +item on the module that contains that storage item; the extension takes the desired name of the getter function as an +argument. If you omit this optional extension, you will still be able to access the storage item's value, but you will +not be able to do so by way of a getter method implemented on the module; instead, you will need to need to use +[the storage item's `get` method](#Methods). Keep in mind that the optional `get` extension only impacts the way that +the storage item can be accessed from within Substrate code; you will always be able to +[query the storage of your runtime](../advanced/storage#Querying-Storage) to get the value of a storage item. -You may have noticed that the non-cryptographic TwoX 128 hash algorithm is used to generate Storage Value keys. This is -because it is not necessary to pay the performance costs associated with a cryptographic hash function since the input -to the hash function (the names of the module and storage item) are determined by you, the runtime developer, and not by -potentially malicious users of your blockchain. +Here is an example that implements a getter method named `some_value` for a Storage Value named `SomeValue`. This module +would now have access to a `Self::some_value()` method in addition to the `SomeValue::get()` method: -### Storage Map Keys +```rust +decl_storage! { + trait Store for Module as Example { + pub SomeValue get(fn some_value): u64; + } +} +``` -Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains -the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply -append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double -Maps), append the hash of the first map key followed by the hash of the second map key to the Storage Double Map's storage -key. Remember, Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you will need -to make sure to use the correct hashing algorithm (the one that was declared in -[the `decl_storage` macro](#Declaring-Storage-Items)) when determining the hashed keys for the elements in a map. +### Default Values + +Substrate allows you to specify a default value that is returned when a storage item's value is not set. The default +value does **not** actually occupy runtime storage, but runtime logic will see this value during execution. -Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the -balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using -[the transparent Blake2 128 Concat hashing algorithm](#Transparent-Hashing-Algorithms): +Here is an example of specifying the default value for all items in a map: +```rust +decl_storage! { + trait Store for Module as Example { + pub SomeMap: map u64 => u64 = 1337; + } +} ``` -twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" -twox_128("FreeBalance") = "0x6482b9ade7bc6657aaca787ba1add3b4" -scale_encode("5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" -blake2_128_concat("0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0xde1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" +### Genesis Config -state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0x0000a0dec5adc9353600000000000000" -``` +You can define +[an optional `GenesisConfig`](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html#genesisconfig) +struct in order to initialize Storage Items in the genesis block of your blockchain. -The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the -[SCALE](../advanced/codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice -that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` -function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat -is [a transparent hashing algorithm as describe above](#Transparent-Hashing-Algorithms). Although the above example may -make this characteristic seem superfluous, its utility becomes more apparent when the goal is to iterate over the keys -in a map (as opposed to retrieving the value associated with a single key). The ability to iterate over the keys in a -map is a common requirement in order to allow _people_ to use the map in a way that seems natural (such as UIs): first, -a user is presented with a list of elements in the map, then, that user can select the element that they are interested -in and query the map for more details about that particular element. Here is another example that uses the same example -Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing algorithm in a module named "Balances") -that will demonstrate using the Substrate RPC to query a Storage Map for its list of keys via the `state_getKeys` RPC -endpoint: +// TODO -``` -twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" -twox_128("FreeBalance") = "0x6482b9ade7bc6657aaca787ba1add3b4" - -state_getKeys("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4") = [ - "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d", - "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b432a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f", - ... -] -``` +## Accessing Storage Items -Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be directly used as input -for the RPC's `state_getStorage` endpoint. In fact, the first element in the example list above is equal to the input -used for the `state_getStorage` query in the previous example (the one used to find the balance for `Alice`). Because -the map that these keys belong to uses a transparent hashing algorithm to generate its keys, it is possible to determine -the account associated with the second element in the list. Notice that each element in the list is a hexadecimal value -that begins with the same 64 characters; this is because each list element represents a key in the same map, and that -map is identified by concatenating two TwoX 128 hashes, each of which are 128-bits or 32 hexadecimal characters. After -discarding this portion of the second element in the list, you are left with -`0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. You saw in the -previous example that this represents the Blake2 128 Concat hash of some [SCALE](../advanced/codec)-encoded account ID. -As described above, the Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's -input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat -hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing -algorithm. In this example, after you remove the first 32 hexadecimal characters that represent the Blake2 128 hash (i.e. -`0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value -`0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](../advanced/codec)-encoded -account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account -ID for the familiar `Alice_Stash` account. +Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query your +blockchain's runtime storage. You can use software libraries like [Polkadot JS](https://polkadot.js.org/) to easily +interact with the RPC server from your code and access storage items. The Polkadot JS team also maintains +[the Polkadot Apps UI](https://polkadot.js.org/apps), which is a fully-featured web app for interacting with +Substrate-based blockchains, including querying storage. Refer to +[the advanced storage documentation](../advanced/storage) to learn more about how Substrate uses a key-value database +to implement the different kinds of Storage Items and how to query this database directly by way of the RPC server. ## Child Storage Tries @@ -337,7 +240,11 @@ TODO TODO -## Verify First, Write Last +## Best Practices + +TODO + +### Verify First, Write Last TODO @@ -345,11 +252,11 @@ TODO ### Learn More -TODO +Read [the advanced storage documentation](../advanced/storage). ### Examples -- View this example to see how you can use a `double_map` to act as a `killable` single-map. +Check out [the Substrate Recipes section on storage](https://substrate.dev/recipes/3-entrees/storage-api/index.html). ### References @@ -358,8 +265,7 @@ TODO about the available storage declarations. * Visit the reference docs for - [StorageValue](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html), - [StorageMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html), - [StorageLinkedMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageLinkedMap.html), and - [StorageDoubleMap](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) to learn + [StorageValue](https://crates.parity.io/frame_support/storage/trait.StorageValue.html), + [StorageMap](https://crates.parity.io/frame_support/storage/trait.StorageMap.html) and + [StorageDoubleMap](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html) to learn more about their APIs. From bd4f8f1e3bf8c0abb9f424b24dd8aaf75bbe0ff0 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 28 Apr 2020 06:32:41 -0700 Subject: [PATCH 14/26] Structural changes per @shawntabrizi (advanced storage) --- current/advanced/storage.md | 114 ++++++++++++++++++++++++++++++++++-- 1 file changed, 110 insertions(+), 4 deletions(-) diff --git a/current/advanced/storage.md b/current/advanced/storage.md index 7ddfe4f..7ae8e30 100644 --- a/current/advanced/storage.md +++ b/current/advanced/storage.md @@ -4,7 +4,10 @@ lang: en title: Storage --- -Substrate uses a simple key-value data store implemented as a database-backed, modified Merkle tree. +Substrate uses a simple key-value data store implemented as a database-backed, +modified Merkle tree. All of Substrate's +[higher-lever storage abstractions](../runtime/storage) are built on top of +this simple key-value store. ## Key-Value Database @@ -64,11 +67,114 @@ can use to verify the specific content in that trie. Subsections of a trie do no root-hash-like representation that satisfy these needs automatically; thus a child trie is used instead. +## Querying Storage + +Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query runtime +storage. When you use the Substrate RPC to access a storage item, you only need to provide +[the key](#Key-Value-Database) associated with that item. + +### Storage Value Keys + +To calculate the key for a simple Storage Value, take the [TwoX 128 hash](https://github.com/Cyan4973/xxHash) of the +name of the module that contains the Storage Value and append to it the TwoX 128 hash of the name of the Storage Value +itself. For example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage +Value item named [`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): + +``` +twox_128("Sudo") = "0x5c0d1176a568c1f92944340dbfed9e9c" +twox_128("Key) = "0x530ebca703c85910e7164cb7d1c9e47b" +twox_128("Sudo") + twox_128("Key") = "0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b" +``` + +If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo module's `Key` Storage +Value could be represented as: + +``` +state_getStorage("0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" +``` + +In this case, the value that is returned (`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is +Alice's [SCALE](./codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). + +You may have noticed that the [non-cryptographic](../runtime/storage#Cryptographic-Hashing-Algorithms) TwoX 128 hash +algorithm is used to generate Storage Value keys. This is because it is not necessary to pay the performance costs +associated with a cryptographic hash function since the input to the hash function (the names of the module and storage +item) are determined by the runtime developer and not by potentially malicious users of your blockchain. + +### Storage Map Keys + +Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains +the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply +append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double +Maps), append the hash of the first map key followed by the hash of the second map key to the Storage Double Map's +storage key. Remember, Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you +will need to make sure to use the correct hashing algorithm (the one that was declared in +[the `decl_storage` macro](../runtime/storage#Declaring-Storage-Items)) when determining the hashed keys for the +elements in a map. + +Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the +balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using +[the transparent Blake2 128 Concat hashing algorithm](../runtime/storage#Transparent-Hashing-Algorithms): + +``` +twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" +twox_128("FreeBalance") = "0x6482b9ade7bc6657aaca787ba1add3b4" +scale_encode("5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" + +blake2_128_concat("0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0xde1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" + +state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0x0000a0dec5adc9353600000000000000" +``` + +The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the +[SCALE](./codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice +that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` +function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat +is [a transparent hashing algorithm as describe above](../runtime/storage#Transparent-Hashing-Algorithms). Although the +above example may make this characteristic seem superfluous, its utility becomes more apparent when the goal is to +iterate over the keys in a map (as opposed to retrieving the value associated with a single key). The ability to +iterate over the keys in a map is a common requirement in order to allow _people_ to use the map in a way that seems +natural (such as UIs): first, a user is presented with a list of elements in the map, then, that user can select the +element that they are interested in and query the map for more details about that particular element. Here is another +example that uses the same example Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing +algorithm in a module named "Balances") that will demonstrate using the Substrate RPC to query a Storage Map for its +list of keys via the `state_getKeys` RPC endpoint: + +``` +twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" +twox_128("FreeBalance") = "0x6482b9ade7bc6657aaca787ba1add3b4" + +state_getKeys("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4") = [ + "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d", + "0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b432a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f", + ... +] +``` + +Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be directly used as input +for the RPC's `state_getStorage` endpoint. In fact, the first element in the example list above is equal to the input +used for the `state_getStorage` query in the previous example (the one used to find the balance for `Alice`). Because +the map that these keys belong to uses a transparent hashing algorithm to generate its keys, it is possible to determine +the account associated with the second element in the list. Notice that each element in the list is a hexadecimal value +that begins with the same 64 characters; this is because each list element represents a key in the same map, and that +map is identified by concatenating two TwoX 128 hashes, each of which are 128-bits or 32 hexadecimal characters. After +discarding this portion of the second element in the list, you are left with +`0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. You saw in the +previous example that this represents the Blake2 128 Concat hash of some [SCALE](./codec)-encoded account ID. +The Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's +input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat +hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing +algorithm. In this example, after you remove the first 32 hexadecimal characters that represent the Blake2 128 hash (i.e. +`0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value +`0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](./codec)-encoded +account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account +ID for the familiar `Alice_Stash` account. + ## Runtime Storage API -The Substrate's [Support module](https://substrate.dev/rustdocs/master/frame_support/index.html) -provides utilities to generate unique, deterministic keys for your runtime module storage items. -These storage items are placed in the state trie and are accessible by querying the trie by key. +The Substrate [FRAME Support pallet](https://substrate.dev/rustdocs/master/frame_support/index.html) provides utilities +for generating unique, deterministic keys for your runtime's storage items. These storage items are placed in +the state trie[#Trie-Abstraction] and are accessible by [querying the trie by key](#Querying-Storage). ## Next Steps From 1dc686ee82187c28344a7387f58efae57db01de5 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 28 Apr 2020 07:13:17 -0700 Subject: [PATCH 15/26] Small changes for readability and elaboration --- current/advanced/storage.md | 34 +++++++++++++++++++--------------- current/runtime/storage.md | 5 +---- 2 files changed, 20 insertions(+), 19 deletions(-) diff --git a/current/advanced/storage.md b/current/advanced/storage.md index 7ae8e30..cad82b2 100644 --- a/current/advanced/storage.md +++ b/current/advanced/storage.md @@ -71,14 +71,17 @@ instead. Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query runtime storage. When you use the Substrate RPC to access a storage item, you only need to provide -[the key](#Key-Value-Database) associated with that item. +[the key](#Key-Value-Database) associated with that item. [Substrate's runtime storage APIs](../runtime/storage) expose +a number of storage item types; keep reading to learn how to calculate storage keys for the different types of storage +items. ### Storage Value Keys -To calculate the key for a simple Storage Value, take the [TwoX 128 hash](https://github.com/Cyan4973/xxHash) of the -name of the module that contains the Storage Value and append to it the TwoX 128 hash of the name of the Storage Value -itself. For example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage -Value item named [`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): +To calculate the key for a simple [Storage Value](../runtime/storage#Storage-Value), take the +[TwoX 128 hash](https://github.com/Cyan4973/xxHash) of the name of the module that contains the Storage Value and +append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the +[Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage Value item named +[`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): ``` twox_128("Sudo") = "0x5c0d1176a568c1f92944340dbfed9e9c" @@ -103,12 +106,13 @@ item) are determined by the runtime developer and not by potentially malicious u ### Storage Map Keys -Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the module that contains -the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply -append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double -Maps), append the hash of the first map key followed by the hash of the second map key to the Storage Double Map's -storage key. Remember, Substrate will use the TwoX 128 hashing algorithm for the module and storage item names, but you -will need to make sure to use the correct hashing algorithm (the one that was declared in +Like Storage Values, the keys for [Storage Maps](../runtime/storage#StorageMaps) are equal to the TwoX 128 hash of the +name of the module that contains the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To +retrieve an element from a map, simply append the hash of the desired map key to the storage key of the Storage Map. +For maps with two keys (Storage Double Maps), append the hash of the first map key followed by the hash of the second +map key to the Storage Double Map's storage key. Like Storage Values, Substrate will use the TwoX 128 hashing algorithm +for the module and Storage Map names, but you will need to make sure to use the correct +[hashing algorithm](../runtime/storage#Hashing-Algorithms) (the one that was declared in [the `decl_storage` macro](../runtime/storage#Declaring-Storage-Items)) when determining the hashed keys for the elements in a map. @@ -128,10 +132,10 @@ state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1ad The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the [SCALE](./codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice -that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` -function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat -is [a transparent hashing algorithm as describe above](../runtime/storage#Transparent-Hashing-Algorithms). Although the -above example may make this characteristic seem superfluous, its utility becomes more apparent when the goal is to +that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the +`blake2_128_concat` function consists of 32 hexadecimal characters followed by the function's input. This is because +the Blake2 128 Concat is [a transparent hashing algorithm](../runtime/storage#Transparent-Hashing-Algorithms). Although +the above example may make this characteristic seem superfluous, its utility becomes more apparent when the goal is to iterate over the keys in a map (as opposed to retrieving the value associated with a single key). The ability to iterate over the keys in a map is a common requirement in order to allow _people_ to use the map in a way that seems natural (such as UIs): first, a user is presented with a list of elements in the map, then, that user can select the diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 81067e2..49ef0cd 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -116,10 +116,7 @@ the `iter` and `drain` methods require a parameter, i.e. the first key: As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described -in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. In this -document you will find information about the pros and cons of the different approaches to hashing; read -[the advanced storage document](../advanced/storage) if you would like a better understanding of _why_ the different -algorithms may behave differently. +in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. ##### Cryptographic Hashing Algorithms From e90f59cc0fe97c7bbeaf5da8694b0f439deb386b Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 28 Apr 2020 13:53:12 -0700 Subject: [PATCH 16/26] Prettier --- current/advanced/storage.md | 140 ++++++++++--------- current/runtime/storage.md | 266 ++++++++++++++++++++---------------- 2 files changed, 222 insertions(+), 184 deletions(-) diff --git a/current/advanced/storage.md b/current/advanced/storage.md index cad82b2..1040d99 100644 --- a/current/advanced/storage.md +++ b/current/advanced/storage.md @@ -4,10 +4,9 @@ lang: en title: Storage --- -Substrate uses a simple key-value data store implemented as a database-backed, -modified Merkle tree. All of Substrate's -[higher-lever storage abstractions](../runtime/storage) are built on top of -this simple key-value store. +Substrate uses a simple key-value data store implemented as a database-backed, modified Merkle tree. +All of Substrate's [higher-lever storage abstractions](../runtime/storage) are built on top of this +simple key-value store. ## Key-Value Database @@ -69,18 +68,19 @@ instead. ## Querying Storage -Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query runtime -storage. When you use the Substrate RPC to access a storage item, you only need to provide -[the key](#Key-Value-Database) associated with that item. [Substrate's runtime storage APIs](../runtime/storage) expose -a number of storage item types; keep reading to learn how to calculate storage keys for the different types of storage -items. +Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be +used to query runtime storage. When you use the Substrate RPC to access a storage item, you only +need to provide [the key](#Key-Value-Database) associated with that item. +[Substrate's runtime storage APIs](../runtime/storage) expose a number of storage item types; keep +reading to learn how to calculate storage keys for the different types of storage items. ### Storage Value Keys To calculate the key for a simple [Storage Value](../runtime/storage#Storage-Value), take the -[TwoX 128 hash](https://github.com/Cyan4973/xxHash) of the name of the module that contains the Storage Value and -append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the -[Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a Storage Value item named +[TwoX 128 hash](https://github.com/Cyan4973/xxHash) of the name of the module that contains the +Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For +example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a +Storage Value item named [`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): ``` @@ -89,35 +89,40 @@ twox_128("Key) = "0x530ebca703c85910e7164cb7d1c9e47b" twox_128("Sudo") + twox_128("Key") = "0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b" ``` -If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo module's `Key` Storage -Value could be represented as: +If the familiar `Alice` account is the sudo user, an RPC request and response to read the Sudo +module's `Key` Storage Value could be represented as: ``` state_getStorage("0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" ``` -In this case, the value that is returned (`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is -Alice's [SCALE](./codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). +In this case, the value that is returned +(`"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"`) is Alice's +[SCALE](./codec)-encoded account ID (`5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`). -You may have noticed that the [non-cryptographic](../runtime/storage#Cryptographic-Hashing-Algorithms) TwoX 128 hash -algorithm is used to generate Storage Value keys. This is because it is not necessary to pay the performance costs -associated with a cryptographic hash function since the input to the hash function (the names of the module and storage -item) are determined by the runtime developer and not by potentially malicious users of your blockchain. +You may have noticed that the +[non-cryptographic](../runtime/storage#Cryptographic-Hashing-Algorithms) TwoX 128 hash algorithm is +used to generate Storage Value keys. This is because it is not necessary to pay the performance +costs associated with a cryptographic hash function since the input to the hash function (the names +of the module and storage item) are determined by the runtime developer and not by potentially +malicious users of your blockchain. ### Storage Map Keys -Like Storage Values, the keys for [Storage Maps](../runtime/storage#StorageMaps) are equal to the TwoX 128 hash of the -name of the module that contains the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To -retrieve an element from a map, simply append the hash of the desired map key to the storage key of the Storage Map. -For maps with two keys (Storage Double Maps), append the hash of the first map key followed by the hash of the second -map key to the Storage Double Map's storage key. Like Storage Values, Substrate will use the TwoX 128 hashing algorithm -for the module and Storage Map names, but you will need to make sure to use the correct +Like Storage Values, the keys for [Storage Maps](../runtime/storage#StorageMaps) are equal to the +TwoX 128 hash of the name of the module that contains the map prepended to the TwoX 128 hash of the +name of the Storage Map itself. To retrieve an element from a map, simply append the hash of the +desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double Maps), +append the hash of the first map key followed by the hash of the second map key to the Storage +Double Map's storage key. Like Storage Values, Substrate will use the TwoX 128 hashing algorithm for +the module and Storage Map names, but you will need to make sure to use the correct [hashing algorithm](../runtime/storage#Hashing-Algorithms) (the one that was declared in -[the `decl_storage` macro](../runtime/storage#Declaring-Storage-Items)) when determining the hashed keys for the -elements in a map. +[the `decl_storage` macro](../runtime/storage#Declaring-Storage-Items)) when determining the hashed +keys for the elements in a map. -Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named "Balances" for the -balance of the familiar `Alice` account. In this example, the `FreeBalance` map is using +Here is an example that illustrates querying a Storage Map named `FreeBalance` from a module named +"Balances" for the balance of the familiar `Alice` account. In this example, the `FreeBalance` map +is using [the transparent Blake2 128 Concat hashing algorithm](../runtime/storage#Transparent-Hashing-Algorithms): ``` @@ -130,19 +135,21 @@ blake2_128_concat("0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56 state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0x0000a0dec5adc9353600000000000000" ``` -The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the example above) is the -[SCALE](./codec)-encoded value of Alice's account balance (`"1000000000000000000000"` in this example). Notice -that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the -`blake2_128_concat` function consists of 32 hexadecimal characters followed by the function's input. This is because -the Blake2 128 Concat is [a transparent hashing algorithm](../runtime/storage#Transparent-Hashing-Algorithms). Although -the above example may make this characteristic seem superfluous, its utility becomes more apparent when the goal is to -iterate over the keys in a map (as opposed to retrieving the value associated with a single key). The ability to -iterate over the keys in a map is a common requirement in order to allow _people_ to use the map in a way that seems -natural (such as UIs): first, a user is presented with a list of elements in the map, then, that user can select the -element that they are interested in and query the map for more details about that particular element. Here is another -example that uses the same example Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing -algorithm in a module named "Balances") that will demonstrate using the Substrate RPC to query a Storage Map for its -list of keys via the `state_getKeys` RPC endpoint: +The value that is returned from the storage query (`"0x0000a0dec5adc9353600000000000000"` in the +example above) is the [SCALE](./codec)-encoded value of Alice's account balance +(`"1000000000000000000000"` in this example). Notice that before hashing Alice's account ID it has +to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` function consists of 32 +hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat is +[a transparent hashing algorithm](../runtime/storage#Transparent-Hashing-Algorithms). Although the +above example may make this characteristic seem superfluous, its utility becomes more apparent when +the goal is to iterate over the keys in a map (as opposed to retrieving the value associated with a +single key). The ability to iterate over the keys in a map is a common requirement in order to allow +_people_ to use the map in a way that seems natural (such as UIs): first, a user is presented with a +list of elements in the map, then, that user can select the element that they are interested in and +query the map for more details about that particular element. Here is another example that uses the +same example Storage Map (a map named `FreeBalances` that uses a Blake2 128 Concat hashing algorithm +in a module named "Balances") that will demonstrate using the Substrate RPC to query a Storage Map +for its list of keys via the `state_getKeys` RPC endpoint: ``` twox_128("Balances) = "0xc2261276cc9d1f8598ea4b6a74b15c2f" @@ -155,30 +162,35 @@ state_getKeys("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b ] ``` -Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be directly used as input -for the RPC's `state_getStorage` endpoint. In fact, the first element in the example list above is equal to the input -used for the `state_getStorage` query in the previous example (the one used to find the balance for `Alice`). Because -the map that these keys belong to uses a transparent hashing algorithm to generate its keys, it is possible to determine -the account associated with the second element in the list. Notice that each element in the list is a hexadecimal value -that begins with the same 64 characters; this is because each list element represents a key in the same map, and that -map is identified by concatenating two TwoX 128 hashes, each of which are 128-bits or 32 hexadecimal characters. After -discarding this portion of the second element in the list, you are left with -`0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. You saw in the -previous example that this represents the Blake2 128 Concat hash of some [SCALE](./codec)-encoded account ID. -The Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's -input to its Blake 128 hash. This means that the first 128 bits (or 32 hexadecimal characters) of a Blake2 128 Concat -hash represents a Blake2 128 hash, and the remainder represents the value that was passed to the Blake 2 128 hashing -algorithm. In this example, after you remove the first 32 hexadecimal characters that represent the Blake2 128 hash (i.e. -`0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value -`0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a [SCALE](./codec)-encoded -account ID. Decoding this value yields the result `5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account -ID for the familiar `Alice_Stash` account. +Each element in the list that is returned by the Substrate RPC's `state_getKeys` endpoint can be +directly used as input for the RPC's `state_getStorage` endpoint. In fact, the first element in the +example list above is equal to the input used for the `state_getStorage` query in the previous +example (the one used to find the balance for `Alice`). Because the map that these keys belong to +uses a transparent hashing algorithm to generate its keys, it is possible to determine the account +associated with the second element in the list. Notice that each element in the list is a +hexadecimal value that begins with the same 64 characters; this is because each list element +represents a key in the same map, and that map is identified by concatenating two TwoX 128 hashes, +each of which are 128-bits or 32 hexadecimal characters. After discarding this portion of the second +element in the list, you are left with +`0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. +You saw in the previous example that this represents the Blake2 128 Concat hash of some +[SCALE](./codec)-encoded account ID. The Blake 128 Concat hashing algorithm consists of appending +(concatenating) the hashing algorithm's input to its Blake 128 hash. This means that the first 128 +bits (or 32 hexadecimal characters) of a Blake2 128 Concat hash represents a Blake2 128 hash, and +the remainder represents the value that was passed to the Blake 2 128 hashing algorithm. In this +example, after you remove the first 32 hexadecimal characters that represent the Blake2 128 hash +(i.e. `0x32a5935f6edc617ae178fef9eb1e211f`) what is left is the hexadecimal value +`0xbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`, which is a +[SCALE](./codec)-encoded account ID. Decoding this value yields the result +`5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY`, which is the account ID for the familiar +`Alice_Stash` account. ## Runtime Storage API -The Substrate [FRAME Support pallet](https://substrate.dev/rustdocs/master/frame_support/index.html) provides utilities -for generating unique, deterministic keys for your runtime's storage items. These storage items are placed in -the state trie[#Trie-Abstraction] and are accessible by [querying the trie by key](#Querying-Storage). +The Substrate [FRAME Support pallet](https://substrate.dev/rustdocs/master/frame_support/index.html) +provides utilities for generating unique, deterministic keys for your runtime's storage items. These +storage items are placed in the state trie[#Trie-Abstraction] and are accessible by +[querying the trie by key](#Querying-Storage). ## Next Steps diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 49ef0cd..81a932a 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -4,40 +4,47 @@ lang: en title: Runtime Storage --- -Runtime storage allows you to store data in your blockchain that is persisted between blocks and can be accessed from -within your runtime logic. Storage should be one of the most critical concerns of a blockchain runtime developer. This -statement is somewhat self-evident, since one of the primary objectives of a blockchain is to provide decentralized -consensus about the state of the underlying storage. Furthermore, well designed storage systems reduce the load on -nodes in the network, which will lower the overhead for participants in your blockchain. Substrate exposes a set of -layered, modular storage APIs that allow runtime developers to make the storage decisions that suit them best. However, -the fundamental principle of blockchain runtime storage is to minimize its use. This document is intended to provide -information and best practices about Substrate's runtime storage interfaces. Please refer to -[the advanced storage documentation](../advanced/storage) for more information about how these interfaces are implemented. +Runtime storage allows you to store data in your blockchain that is persisted between blocks and can +be accessed from within your runtime logic. Storage should be one of the most critical concerns of a +blockchain runtime developer. This statement is somewhat self-evident, since one of the primary +objectives of a blockchain is to provide decentralized consensus about the state of the underlying +storage. Furthermore, well designed storage systems reduce the load on nodes in the network, which +will lower the overhead for participants in your blockchain. Substrate exposes a set of layered, +modular storage APIs that allow runtime developers to make the storage decisions that suit them +best. However, the fundamental principle of blockchain runtime storage is to minimize its use. This +document is intended to provide information and best practices about Substrate's runtime storage +interfaces. Please refer to [the advanced storage documentation](../advanced/storage) for more +information about how these interfaces are implemented. ## Storage Items -The `storage` module in [FRAME Support](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) -gives runtime developers access to Substrate's flexible storage APIs. Any value that can be encoded by the +The `storage` module in +[FRAME Support](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) gives +runtime developers access to Substrate's flexible storage APIs. Any value that can be encoded by the [Parity SCALE codec](../advanced/codec) is supported by these storage APIs: -* [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - A single value -* [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - A key-value hash map -* [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - An - implementation of a map with two keys that provides the important ability to efficiently remove all entries that have - a common first key +- [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - + A single value +- [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - + A key-value hash map +- [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - + An implementation of a map with two keys that provides the important ability to efficiently remove + all entries that have a common first key -The type of Storage Item you select should depend on the logical way in which the value will be used by your runtime. +The type of Storage Item you select should depend on the logical way in which the value will be used +by your runtime. ### Storage Value -This type of storage item should be used for values that are viewed as a single unit by the runtime, whether that is a -single primitive value, a single `struct`, or a single collection of related items. Although wrapping related items in -a shared `struct` is an excellent way to reduce the number of storage reads (an important consideration), at some point -the size of the object will begin to incur costs that may outweigh the optimization in storage reads. Storage values -can be used to store lists of items, but runtime developers should take care with respect to the size of these lists. -Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list in your runtime may -result in exceeding the block production time - if this occurs your blockchain will stop producing blocks, which means -that it will stop functioning. +This type of storage item should be used for values that are viewed as a single unit by the runtime, +whether that is a single primitive value, a single `struct`, or a single collection of related +items. Although wrapping related items in a shared `struct` is an excellent way to reduce the number +of storage reads (an important consideration), at some point the size of the object will begin to +incur costs that may outweigh the optimization in storage reads. Storage values can be used to store +lists of items, but runtime developers should take care with respect to the size of these lists. +Large lists incur storage costs just like large `structs`. Furthermore, iterating over a large list +in your runtime may result in exceeding the block production time - if this occurs your blockchain +will stop producing blocks, which means that it will stop functioning. #### Methods @@ -45,105 +52,113 @@ Refer to the Storage Value documentation for [a comprehensive list of the methods that Storage Values expose](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#required-methods). Some of the most important methods are summarized here: -* [`get()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.get) - Load - the value from storage. -* [`put(val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.put) - Store - the provided value. -* [`mutate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.mutate) - +- [`get()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.get) - + Load the value from storage. +- [`put(val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.put) - + Store the provided value. +- [`mutate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.mutate) - Mutate the value with the provided function. -* [`take()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.take) - Remove - the value from storage. +- [`take()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.take) - + Remove the value from storage. ### Storage Maps -Map data structures are ideal for managing sets of items whose elements will be accessed randomly, as opposed to -iterating over them sequentially in their entirety. Storage Maps in Substrate are implemented as key-value hash maps, -which is a pattern that should be familiar to most developers. In order to give blockchain engineers increased control, -Substrate allows developers to select [the hashing algorithm](#Hashing-Algorithms) that is used to generate a map's -keys. Refer to [the advanced storage documentation](../advanced/storage) to learn more about how Substrate's Storage -Maps are implemented. +Map data structures are ideal for managing sets of items whose elements will be accessed randomly, +as opposed to iterating over them sequentially in their entirety. Storage Maps in Substrate are +implemented as key-value hash maps, which is a pattern that should be familiar to most developers. +In order to give blockchain engineers increased control, Substrate allows developers to select +[the hashing algorithm](#Hashing-Algorithms) that is used to generate a map's keys. Refer to +[the advanced storage documentation](../advanced/storage) to learn more about how Substrate's +Storage Maps are implemented. #### Methods [Storage Maps expose an API](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#required-methods) that is similar to that of Storage Values. -* `get` - Load the value associated with the provided key from storage. Docs: +- `get` - Load the value associated with the provided key from storage. Docs: [`StorageMap#get(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.get), - [`StorageDoubleMap#get(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.get) -* `insert` - Store the provided value by associating it with the given key. Docs: + [`StorageDoubleMap#get(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.get) +- `insert` - Store the provided value by associating it with the given key. Docs: [`StorageMap#insert(key, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.insert), - [`StorageDoubleMap#insert(key1, key2, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.insert) -* `mutate` - Use the provided function to mutate the value associated with the given key. Docs: + [`StorageDoubleMap#insert(key1, key2, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.insert) +- `mutate` - Use the provided function to mutate the value associated with the given key. Docs: [`StorageMap#mutate(key, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.mutate), - [`StorageDoubleMap#mutate(key1, key2, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.mutate) -* `take` - Remove the value associated with the given key from storage. Docs: + [`StorageDoubleMap#mutate(key1, key2, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.mutate) +- `take` - Remove the value associated with the given key from storage. Docs: [`StorageMap#take(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.take), - [`StorageDoubleMap#take(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) + [`StorageDoubleMap#take(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) #### Iterable Storage Maps -Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a map's keys, you may -be able to iterate across its keys and values. Because maps are often used to track unbounded sets of data (account -balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety -within the runtime. Furthermore, because maps are comprised of more layers of indirection than native lists, they are -significantly more costly than lists to iterate over with respect to time. This is not to say that it is "wrong" to -iterate over maps in your runtime; in general Substrate focuses on "first principles" as opposed to hard and fast rules -of right and wrong. Being efficient within the runtime of a blockchain is an important first principle of Substrate and -this information is designed to help you understand _all_ of Substrate's storage capabilities and use them in a way -that respects the important first principles around which they were designed. +Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a +map's keys, you may be able to iterate across its keys and values. Because maps are often used to +track unbounded sets of data (account balances, for example) it is especially likely to exceed block +production time by iterating over maps in their entirety within the runtime. Furthermore, because +maps are comprised of more layers of indirection than native lists, they are significantly more +costly than lists to iterate over with respect to time. This is not to say that it is "wrong" to +iterate over maps in your runtime; in general Substrate focuses on "first principles" as opposed to +hard and fast rules of right and wrong. Being efficient within the runtime of a blockchain is an +important first principle of Substrate and this information is designed to help you understand _all_ +of Substrate's storage capabilities and use them in a way that respects the important first +principles around which they were designed. ##### Iterable Storage Map Methods -Substrate's Iterable Storage Map interfaces define the following methods. Note that for Iterable Storage Double Maps, -the `iter` and `drain` methods require a parameter, i.e. the first key: +Substrate's Iterable Storage Map interfaces define the following methods. Note that for Iterable +Storage Double Maps, the `iter` and `drain` methods require a parameter, i.e. the first key: -* `iter` - Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined - results. Docs: - [IterableStorageMap#iter()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), - [IterableStorageDoubleMap#iter(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) -* `drain` - Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while +- `iter` - Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined results. Docs: - [IterableStorageMap#drain()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), - [IterableStorageDoubleMap#drain(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) -* `translate` - Use the provided function to translate all elements of the map, in no particular order. To remove an element from the - map, return `None` from the translation function. Docs: - [IterableStorageMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), - [IterableStorageDoubleMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) + [IterableStorageMap#iter()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), + [IterableStorageDoubleMap#iter(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) +- `drain` - Remove all elements from the map and iterate through them in no particular order. If you + add elements to the map while doing this, you'll get undefined results. Docs: + [IterableStorageMap#drain()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), + [IterableStorageDoubleMap#drain(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) +- `translate` - Use the provided function to translate all elements of the map, in no particular + order. To remove an element from the map, return `None` from the translation function. Docs: + [IterableStorageMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), + [IterableStorageDoubleMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) #### Hashing Algorithms -As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to specify the hashing -algorithm that will be used when generating a map's keys. A Rust object that is used to encapsulate hashing logic is -referred to as a "hasher". Broadly speaking, the hashers that are available to Substrate developers can be described -in two ways: whether or not they are cryptographic and whether or not they produce output that is transparent. +As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to +specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is +used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that +are available to Substrate developers can be described in two ways: whether or not they are +cryptographic and whether or not they produce output that is transparent. ##### Cryptographic Hashing Algorithms -Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the input to the hashing -algorithm to influence its output. For example, a cryptographic hashing algorithm would produce a wide distribution of -outputs even if the inputs were the numbers 1 through 10. It is critical to use cryptographic hashing algorithms when -users are able to influence the keys of a Storage Map. Failure to do so creates an attack vector that makes it easy for -malicious actors to degrade the performance of your blockchain network. An example of a map that should use a cryptographic -hash algorithm to generate its keys is a map used to track account balances. In this case, it is important to use a -cryptographic hashing algorithm so that an attacker cannot bombard your system with many small transfers to sequential -account numbers; without a cryptographic hash algorithm this would create an imbalanced storage structure that would -suffer in performance. Cryptographic hashing algorithms are more complex and resource-intensive than their non-cryptographic -counterparts, which is why Substrate allows developers to select when they are used. +Cryptographic hashing algorithms are those that use cryptography to make it challenging to use the +input to the hashing algorithm to influence its output. For example, a cryptographic hashing +algorithm would produce a wide distribution of outputs even if the inputs were the numbers 1 +through 10. It is critical to use cryptographic hashing algorithms when users are able to influence +the keys of a Storage Map. Failure to do so creates an attack vector that makes it easy for +malicious actors to degrade the performance of your blockchain network. An example of a map that +should use a cryptographic hash algorithm to generate its keys is a map used to track account +balances. In this case, it is important to use a cryptographic hashing algorithm so that an attacker +cannot bombard your system with many small transfers to sequential account numbers; without a +cryptographic hash algorithm this would create an imbalanced storage structure that would suffer in +performance. Cryptographic hashing algorithms are more complex and resource-intensive than their +non-cryptographic counterparts, which is why Substrate allows developers to select when they are +used. ##### Transparent Hashing Algorithms -A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a -given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. -This makes it trivial for users to retrieve a key's original unhashed value and verify it if they'd like (by re-hashing -it). It is generally recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is -_necessary_ to use a transparent hashing algorithm if you would like access [iterable map](#Iterable-Storage-Maps) -capabilities. +A transparent hashing algorithm is one that makes it easy to discover and verify the input that was +used to generate a given output. In Substrate, hashing algorithms are made transparent by +concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a +key's original unhashed value and verify it if they'd like (by re-hashing it). It is generally +recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is +_necessary_ to use a transparent hashing algorithm if you would like access +[iterable map](#Iterable-Storage-Maps) capabilities. ##### Common Substrate Hashers -This table lists some common hashers used in Substrate and denotes those that are cryptographic and those that are -transparent: +This table lists some common hashers used in Substrate and denotes those that are cryptographic and +those that are transparent: | Hasher | Cryptographic | Transparent | | ----------------------------------------------------------------------------------------------------- | ------------- | ----------- | @@ -153,13 +168,16 @@ transparent: | [TwoX 64 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Twox64Concat.html) | | X | | [Identity](https://substrate.dev/rustdocs/master/frame_support/struct.Identity.html) | | | -The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the identity function).This type -of hasher should only be used when the starting key is already a cryptographic hash. +The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the +identity function).This type of hasher should only be used when the starting key is already a +cryptographic hash. ## Declaring Storage Items -You can use [the `decl_storage` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) to -easily create new runtime storage items. Here is an example of what it looks like to declare each type of storage item: +You can use +[the `decl_storage` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) +to easily create new runtime storage items. Here is an example of what it looks like to declare each +type of storage item: ```rust decl_storage! { @@ -173,20 +191,24 @@ decl_storage! { } ``` -Notice that the map storage items specify [the hashing algorithm](#Hashing-Algorithms) that will be used. +Notice that the map storage items specify [the hashing algorithm](#Hashing-Algorithms) that will be +used. ### Getter Methods -The `decl_storage` macro provides an optional `get` extension that can be used to implement a getter method for a storage -item on the module that contains that storage item; the extension takes the desired name of the getter function as an -argument. If you omit this optional extension, you will still be able to access the storage item's value, but you will -not be able to do so by way of a getter method implemented on the module; instead, you will need to need to use -[the storage item's `get` method](#Methods). Keep in mind that the optional `get` extension only impacts the way that -the storage item can be accessed from within Substrate code; you will always be able to -[query the storage of your runtime](../advanced/storage#Querying-Storage) to get the value of a storage item. +The `decl_storage` macro provides an optional `get` extension that can be used to implement a getter +method for a storage item on the module that contains that storage item; the extension takes the +desired name of the getter function as an argument. If you omit this optional extension, you will +still be able to access the storage item's value, but you will not be able to do so by way of a +getter method implemented on the module; instead, you will need to need to use +[the storage item's `get` method](#Methods). Keep in mind that the optional `get` extension only +impacts the way that the storage item can be accessed from within Substrate code; you will always be +able to [query the storage of your runtime](../advanced/storage#Querying-Storage) to get the value +of a storage item. -Here is an example that implements a getter method named `some_value` for a Storage Value named `SomeValue`. This module -would now have access to a `Self::some_value()` method in addition to the `SomeValue::get()` method: +Here is an example that implements a getter method named `some_value` for a Storage Value named +`SomeValue`. This module would now have access to a `Self::some_value()` method in addition to the +`SomeValue::get()` method: ```rust decl_storage! { @@ -198,8 +220,9 @@ decl_storage! { ### Default Values -Substrate allows you to specify a default value that is returned when a storage item's value is not set. The default -value does **not** actually occupy runtime storage, but runtime logic will see this value during execution. +Substrate allows you to specify a default value that is returned when a storage item's value is not +set. The default value does **not** actually occupy runtime storage, but runtime logic will see this +value during execution. Here is an example of specifying the default value for all items in a map: @@ -221,13 +244,15 @@ struct in order to initialize Storage Items in the genesis block of your blockch ## Accessing Storage Items -Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query your -blockchain's runtime storage. You can use software libraries like [Polkadot JS](https://polkadot.js.org/) to easily -interact with the RPC server from your code and access storage items. The Polkadot JS team also maintains -[the Polkadot Apps UI](https://polkadot.js.org/apps), which is a fully-featured web app for interacting with -Substrate-based blockchains, including querying storage. Refer to -[the advanced storage documentation](../advanced/storage) to learn more about how Substrate uses a key-value database -to implement the different kinds of Storage Items and how to query this database directly by way of the RPC server. +Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be +used to query your blockchain's runtime storage. You can use software libraries like +[Polkadot JS](https://polkadot.js.org/) to easily interact with the RPC server from your code and +access storage items. The Polkadot JS team also maintains +[the Polkadot Apps UI](https://polkadot.js.org/apps), which is a fully-featured web app for +interacting with Substrate-based blockchains, including querying storage. Refer to +[the advanced storage documentation](../advanced/storage) to learn more about how Substrate uses a +key-value database to implement the different kinds of Storage Items and how to query this database +directly by way of the RPC server. ## Child Storage Tries @@ -253,16 +278,17 @@ Read [the advanced storage documentation](../advanced/storage). ### Examples -Check out [the Substrate Recipes section on storage](https://substrate.dev/recipes/3-entrees/storage-api/index.html). +Check out +[the Substrate Recipes section on storage](https://substrate.dev/recipes/3-entrees/storage-api/index.html). ### References -* Visit the reference docs for the - [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) for more details - about the available storage declarations. +- Visit the reference docs for the + [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) + for more details about the available storage declarations. -* Visit the reference docs for +- Visit the reference docs for [StorageValue](https://crates.parity.io/frame_support/storage/trait.StorageValue.html), [StorageMap](https://crates.parity.io/frame_support/storage/trait.StorageMap.html) and - [StorageDoubleMap](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html) to learn - more about their APIs. + [StorageDoubleMap](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html) to + learn more about their APIs. From a54712a0b2502fd96e93f951362501168473ffaf Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Thu, 30 Apr 2020 07:26:19 -0700 Subject: [PATCH 17/26] Add section on best practices and address suggestions from @joepetrowski --- current/advanced/storage.md | 4 +- current/runtime/storage.md | 126 +++++++++++++++++++++++++++++------- 2 files changed, 104 insertions(+), 26 deletions(-) diff --git a/current/advanced/storage.md b/current/advanced/storage.md index 1040d99..5dbd4f4 100644 --- a/current/advanced/storage.md +++ b/current/advanced/storage.md @@ -79,7 +79,7 @@ reading to learn how to calculate storage keys for the different types of storag To calculate the key for a simple [Storage Value](../runtime/storage#Storage-Value), take the [TwoX 128 hash](https://github.com/Cyan4973/xxHash) of the name of the module that contains the Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For -example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) module exposes a +example, the [Sudo](https://substrate.dev/rustdocs/master/pallet_sudo/index.html) pallet exposes a Storage Value item named [`Key`](https://substrate.dev/rustdocs/master/pallet_sudo/struct.Module.html#method.key): @@ -187,7 +187,7 @@ example, after you remove the first 32 hexadecimal characters that represent the ## Runtime Storage API -The Substrate [FRAME Support pallet](https://substrate.dev/rustdocs/master/frame_support/index.html) +Substrate's [FRAME Support crate](https://substrate.dev/rustdocs/master/frame_support/index.html) provides utilities for generating unique, deterministic keys for your runtime's storage items. These storage items are placed in the state trie[#Trie-Abstraction] and are accessible by [querying the trie by key](#Querying-Storage). diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 81a932a..6a335ec 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -59,7 +59,7 @@ Some of the most important methods are summarized here: - [`mutate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.mutate) - Mutate the value with the provided function. - [`take()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.take) - - Remove the value from storage. + Load the value and remove it from storage. ### Storage Maps @@ -85,7 +85,7 @@ that is similar to that of Storage Values. - `mutate` - Use the provided function to mutate the value associated with the given key. Docs: [`StorageMap#mutate(key, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.mutate), [`StorageDoubleMap#mutate(key1, key2, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.mutate) -- `take` - Remove the value associated with the given key from storage. Docs: +- `take` - Load the value associated with the given key and remove it from storage. Docs: [`StorageMap#take(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.take), [`StorageDoubleMap#take(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) @@ -95,13 +95,14 @@ Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you s map's keys, you may be able to iterate across its keys and values. Because maps are often used to track unbounded sets of data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because -maps are comprised of more layers of indirection than native lists, they are significantly more -costly than lists to iterate over with respect to time. This is not to say that it is "wrong" to -iterate over maps in your runtime; in general Substrate focuses on "first principles" as opposed to -hard and fast rules of right and wrong. Being efficient within the runtime of a blockchain is an -important first principle of Substrate and this information is designed to help you understand _all_ -of Substrate's storage capabilities and use them in a way that respects the important first -principles around which they were designed. +accessing the elements of a map requires more pointer dereferencing than accessing the elements of a +native list, maps are significantly _more_ costly than lists to iterate over with respect to time. +This is not to say that it is "wrong" to iterate over maps in your runtime; in general Substrate +focuses on "[first principles](#Best-Practices)" as opposed to hard and fast rules of right and +wrong. Being efficient within the runtime of a blockchain is an important first principle of +Substrate and this information is designed to help you understand _all_ of Substrate's storage +capabilities and use them in a way that respects the important first principles around which they +were designed. ##### Iterable Storage Map Methods @@ -110,16 +111,16 @@ Storage Double Maps, the `iter` and `drain` methods require a parameter, i.e. th - `iter` - Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined results. Docs: - [IterableStorageMap#iter()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), - [IterableStorageDoubleMap#iter(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) + [`IterableStorageMap#iter()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), + [`IterableStorageDoubleMap#iter(key1)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) - `drain` - Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while doing this, you'll get undefined results. Docs: - [IterableStorageMap#drain()](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), - [IterableStorageDoubleMap#drain(key1)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) + [`IterableStorageMap#drain()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), + [`IterableStorageDoubleMap#drain(key1)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) - `translate` - Use the provided function to translate all elements of the map, in no particular order. To remove an element from the map, return `None` from the translation function. Docs: - [IterableStorageMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), - [IterableStorageDoubleMap#translate(fn)](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) + [`IterableStorageMap#translate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), + [`IterableStorageDoubleMap#translate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) #### Hashing Algorithms @@ -169,7 +170,7 @@ those that are transparent: | [Identity](https://substrate.dev/rustdocs/master/frame_support/struct.Identity.html) | | | The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the -identity function).This type of hasher should only be used when the starting key is already a +identity function). This type of hasher should only be used when the starting key is already a cryptographic hash. ## Declaring Storage Items @@ -182,11 +183,12 @@ type of storage item: ```rust decl_storage! { trait Store for Module as Example { + SomePrivateValue: u32; pub SomePrimitiveValue get(fn some_primitive_value): u32; // complex types are prefaced by T:: pub SomeComplexValue: T::AccountId; - pub SomeMap get(fn some_map): map hasher(blake2_128_concat) str => u32; - pub SomeDoubleMap: double_map hasher(blake2_128_concat) str, hasher(blake2_128_concat) str => u32; + pub SomeMap get(fn some_map): map hasher(blake2_128_concat) T::AccountId => u32; + pub SomeDoubleMap: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::AccountId => u32; } } ``` @@ -194,6 +196,14 @@ decl_storage! { Notice that the map storage items specify [the hashing algorithm](#Hashing-Algorithms) that will be used. +### Visibility + +In the example above, all the storage items except `SomePrivateValue` are made public by way of the +`pub` keyword. Blockchain storage is always publicly +[visible from _outside_ of the runtime](#Accessing-Storage-Items); the visibility of Substrate +storage items only impacts whether or not other runtime pallets will be able to access the storage +item. + ### Getter Methods The `decl_storage` macro provides an optional `get` extension that can be used to implement a getter @@ -254,19 +264,87 @@ interacting with Substrate-based blockchains, including querying storage. Refer key-value database to implement the different kinds of Storage Items and how to query this database directly by way of the RPC server. -## Child Storage Tries +## Best Practices -TODO +Substrate's goal is to provide a flexible framework that allows people to build the blockchain that +suits their needs - the creators of Substrate tend not to think in terms of "right" or "wrong". That +being said, the Substrate codebase adheres to a number of best practices in order to promote the +creation of blockchain networks that are secure, performant and maintainable in the long-term. The +following sections outline best practices for using Substrate storage and also describe the +important first principles that motivated them. + +### What to Store + +Remember, the fundamental principle of blockchain runtime storage is to minimize its use. Only +_consensus-critical_ data should be stored in your runtime. When possible, use techniques like +hashing to reduce the amount of data you must store. For instance, many of Substrate's governance +capabilities (e.g. +[the Democracy pallet's `propose` dispatchable](https://crates.parity.io/pallet_democracy/enum.Call.html#variant.propose)) +allow network participants to vote on the _hash_ of a dispatchable call, which is always bounded in +size, as opposed to the call itself, which may be unbounded in length. This is especially true in +the case of runtime upgrades where the dispatchable call takes an entire runtime WASM blob as its +parameter. Because these governance mechanisms are implemented _on-chain_, all the information that +is needed to come to consensus on the state of a given proposal must also be stored on-chain - this +includes _what_ is being voted on. However, by binding an on-chain proposal to its hash, Substrate's +governance mechanisms allow this to be done in a way that defers bringing all the data associated +with a proposal on-chain until _after_ it has been approved. This means that storage is not wasted +on proposals that fail. Once a proposal has passed, someone can initiate the actual dispatchable +call (including all its parameters), which will be hashed and compared to the hash in the proposal. +Another common pattern for using hashes to minimize data that is stored on-chain is to store the +metadata associated with an object in [IPFS](https://ipfs.io/); this means that only the IPFS +location (a hash that is bounded in size) needs to be stored on-chain. + +Hashes are only one mechanism that can be used to control the size of runtime storage. An example of +another mechanism is [bounds](#Create-Bounds). -## Storage Cache +### Verify First, Write Last -TODO +The state of a blockchain network's storage is immutable; data can be changed, but there will always +be a record of these changes, and making them typically incurs costs. Because of this, it is +important that data is only persisted to runtime storage when it is certain that all preconditions +have been met. In general, code blocks that may result in adding data to storage should be +structured as follows: -## Best Practices +```rust +{ + // all checks and throwing code go here + + // all storage writes go here; no throwing code below this line + + // all event emissions go here +} +``` + +Do not use runtime storage to store intermediate or transient data within the context of an +operation that is logically atomic or data that will not be needed if the operation is to fail. This +does not mean that runtime storage should not be used to track the state of ongoing actions that +require multiple atomic operations, as in the case of +[the multi-signature capabilities from the Utility pallet](https://crates.parity.io/pallet_utility/enum.Call.html#variant.as_multi). +In this case, runtime storage is used to track the signatories on a dispatchable call even though a +given call may never receive enough signatures to actually be invoked. In this case, each signature +is considered an atomic event in the ongoing multi-signature operation; the data needed to record a +single signature is not stored until after all the preconditions associated with that signature have +been met. + +### Create Bounds + +Creating bounds on the size of storage items is an extremely effective way to control the use of +runtime storage and one that is used repeatedly throughout the Substrate codebase. In general, any +storage item whose size is determined by user action should have a bound on it. +[The multi-signature capabilities from the Utility pallet](https://crates.parity.io/pallet_utility/trait.Trait.html#associatedtype.MaxSignatories) +that were described above are one such example. In this case, the list of signatories associated +with a multi-signature operation is provided by the multi-signature participants. Because this +signatory list is [necessary to come to consensus](#What-to-Store) on the state of the +multi-signature operation, it must be stored in the runtime. However, in order to give runtime +developers control over how much space in storage these lists may occupy, the Utility pallet +requires users to configure a bound on this number that will be included as a +[precondition](#Verify-First-Write-Last) before anything is written to storage. + +## Child Storage Tries TODO -### Verify First, Write Last +## Storage Cache TODO From c95681c000105533d82d4e9c985fd99dc490d779 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Thu, 30 Apr 2020 11:07:08 -0700 Subject: [PATCH 18/26] Small clarifications --- current/runtime/storage.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 6a335ec..768edd2 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -302,7 +302,7 @@ another mechanism is [bounds](#Create-Bounds). The state of a blockchain network's storage is immutable; data can be changed, but there will always be a record of these changes, and making them typically incurs costs. Because of this, it is important that data is only persisted to runtime storage when it is certain that all preconditions -have been met. In general, code blocks that may result in adding data to storage should be +have been met. In general, code blocks that may result in mutating storage should be structured as follows: ```rust @@ -344,10 +344,6 @@ requires users to configure a bound on this number that will be included as a TODO -## Storage Cache - -TODO - ## Next Steps ### Learn More From 89a31fa33d51e3f99ccb6b8a34cd6a174357c9c8 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 4 May 2020 06:21:12 -0700 Subject: [PATCH 19/26] Storage item is not a proper noun --- current/runtime/storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 768edd2..5b12425 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -31,7 +31,7 @@ runtime developers access to Substrate's flexible storage APIs. Any value that c An implementation of a map with two keys that provides the important ability to efficiently remove all entries that have a common first key -The type of Storage Item you select should depend on the logical way in which the value will be used +The type of storage item you select should depend on the logical way in which the value will be used by your runtime. ### Storage Value @@ -248,7 +248,7 @@ decl_storage! { You can define [an optional `GenesisConfig`](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html#genesisconfig) -struct in order to initialize Storage Items in the genesis block of your blockchain. +struct in order to initialize storage items in the genesis block of your blockchain. // TODO @@ -261,7 +261,7 @@ access storage items. The Polkadot JS team also maintains [the Polkadot Apps UI](https://polkadot.js.org/apps), which is a fully-featured web app for interacting with Substrate-based blockchains, including querying storage. Refer to [the advanced storage documentation](../advanced/storage) to learn more about how Substrate uses a -key-value database to implement the different kinds of Storage Items and how to query this database +key-value database to implement the different kinds of storage items and how to query this database directly by way of the RPC server. ## Best Practices From 3b469529e70a123a8c9b615ccb068431e5c8537e Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 4 May 2020 16:37:58 -0700 Subject: [PATCH 20/26] Use crates.parity.io for Rustdoc links --- current/runtime/storage.md | 89 +++++++++++++++++++------------------- 1 file changed, 44 insertions(+), 45 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 5b12425..569749e 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -18,16 +18,15 @@ information about how these interfaces are implemented. ## Storage Items -The `storage` module in -[FRAME Support](https://substrate.dev/rustdocs/master/frame_support/storage/index.html) gives -runtime developers access to Substrate's flexible storage APIs. Any value that can be encoded by the -[Parity SCALE codec](../advanced/codec) is supported by these storage APIs: - -- [Storage Value](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html) - - A single value -- [Storage Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html) - - A key-value hash map -- [Storage Double Map](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html) - +The `storage` module in [FRAME Support](https://crates.parity.io/frame_support/storage/index.html) +gives runtime developers access to Substrate's flexible storage APIs. Any value that can be encoded +by the [Parity SCALE codec](../advanced/codec) is supported by these storage APIs: + +- [Storage Value](https://crates.parity.io/frame_support/storage/trait.StorageValue.html) - A single + value +- [Storage Map](https://crates.parity.io/frame_support/storage/trait.StorageMap.html) - A key-value + hash map +- [Storage Double Map](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html) - An implementation of a map with two keys that provides the important ability to efficiently remove all entries that have a common first key @@ -49,16 +48,16 @@ will stop producing blocks, which means that it will stop functioning. #### Methods Refer to the Storage Value documentation for -[a comprehensive list of the methods that Storage Values expose](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#required-methods). +[a comprehensive list of the methods that Storage Values expose](https://crates.parity.io/frame_support/storage/trait.StorageValue.html#required-methods). Some of the most important methods are summarized here: -- [`get()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.get) - +- [`get()`](https://crates.parity.io/frame_support/storage/trait.StorageValue.html#tymethod.get) - Load the value from storage. -- [`put(val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.put) - +- [`put(val)`](https://crates.parity.io/frame_support/storage/trait.StorageValue.html#tymethod.put) - Store the provided value. -- [`mutate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.mutate) - +- [`mutate(fn)`](https://crates.parity.io/frame_support/storage/trait.StorageValue.html#tymethod.mutate) - Mutate the value with the provided function. -- [`take()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageValue.html#tymethod.take) - +- [`take()`](https://crates.parity.io/frame_support/storage/trait.StorageValue.html#tymethod.take) - Load the value and remove it from storage. ### Storage Maps @@ -73,21 +72,21 @@ Storage Maps are implemented. #### Methods -[Storage Maps expose an API](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#required-methods) +[Storage Maps expose an API](https://crates.parity.io/frame_support/storage/trait.StorageMap.html#required-methods) that is similar to that of Storage Values. - `get` - Load the value associated with the provided key from storage. Docs: - [`StorageMap#get(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.get), - [`StorageDoubleMap#get(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.get) + [`StorageMap#get(key)`](https://crates.parity.io/frame_support/storage/trait.StorageMap.html#tymethod.get), + [`StorageDoubleMap#get(key1, key2)`](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html#tymethod.get) - `insert` - Store the provided value by associating it with the given key. Docs: - [`StorageMap#insert(key, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.insert), - [`StorageDoubleMap#insert(key1, key2, val)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.insert) + [`StorageMap#insert(key, val)`](https://crates.parity.io/frame_support/storage/trait.StorageMap.html#tymethod.insert), + [`StorageDoubleMap#insert(key1, key2, val)`](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html#tymethod.insert) - `mutate` - Use the provided function to mutate the value associated with the given key. Docs: - [`StorageMap#mutate(key, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.mutate), - [`StorageDoubleMap#mutate(key1, key2, fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.mutate) + [`StorageMap#mutate(key, fn)`](https://crates.parity.io/frame_support/storage/trait.StorageMap.html#tymethod.mutate), + [`StorageDoubleMap#mutate(key1, key2, fn)`](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html#tymethod.mutate) - `take` - Load the value associated with the given key and remove it from storage. Docs: - [`StorageMap#take(key)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageMap.html#tymethod.take), - [`StorageDoubleMap#take(key1, key2)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) + [`StorageMap#take(key)`](https://crates.parity.io/frame_support/storage/trait.StorageMap.html#tymethod.take), + [`StorageDoubleMap#take(key1, key2)`](https://crates.parity.io/frame_support/storage/trait.StorageDoubleMap.html#tymethod.take) #### Iterable Storage Maps @@ -111,16 +110,16 @@ Storage Double Maps, the `iter` and `drain` methods require a parameter, i.e. th - `iter` - Enumerate all elements in the map in no particular order. If you alter the map while doing this, you'll get undefined results. Docs: - [`IterableStorageMap#iter()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), - [`IterableStorageDoubleMap#iter(key1)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) + [`IterableStorageMap#iter()`](https://crates.parity.io/frame_support/storage/trait.IterableStorageMap.html#tymethod.iter), + [`IterableStorageDoubleMap#iter(key1)`](https://crates.parity.io/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.iter) - `drain` - Remove all elements from the map and iterate through them in no particular order. If you add elements to the map while doing this, you'll get undefined results. Docs: - [`IterableStorageMap#drain()`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), - [`IterableStorageDoubleMap#drain(key1)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) + [`IterableStorageMap#drain()`](https://crates.parity.io/frame_support/storage/trait.IterableStorageMap.html#tymethod.drain), + [`IterableStorageDoubleMap#drain(key1)`](https://crates.parity.io/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.drain) - `translate` - Use the provided function to translate all elements of the map, in no particular order. To remove an element from the map, return `None` from the translation function. Docs: - [`IterableStorageMap#translate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), - [`IterableStorageDoubleMap#translate(fn)`](https://substrate.dev/rustdocs/master/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) + [`IterableStorageMap#translate(fn)`](https://crates.parity.io/frame_support/storage/trait.IterableStorageMap.html#tymethod.translate), + [`IterableStorageDoubleMap#translate(fn)`](https://crates.parity.io/frame_support/storage/trait.IterableStorageDoubleMap.html#tymethod.translate) #### Hashing Algorithms @@ -161,13 +160,13 @@ _necessary_ to use a transparent hashing algorithm if you would like access This table lists some common hashers used in Substrate and denotes those that are cryptographic and those that are transparent: -| Hasher | Cryptographic | Transparent | -| ----------------------------------------------------------------------------------------------------- | ------------- | ----------- | -| [Blake2 128](https://substrate.dev/rustdocs/master/frame_support/struct.Blake2_128.html) | X | | -| [TwoX 128](https://substrate.dev/rustdocs/master/frame_support/struct.Twox128.html) | | | -| [Blake2 128 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Blake2_128Concat.html) | X | X | -| [TwoX 64 Concat](https://substrate.dev/rustdocs/master/frame_support/struct.Twox64Concat.html) | | X | -| [Identity](https://substrate.dev/rustdocs/master/frame_support/struct.Identity.html) | | | +| Hasher | Cryptographic | Transparent | +| ---------------------------------------------------------------------------------------- | ------------- | ----------- | +| [Blake2 128](https://crates.parity.io/frame_support/struct.Blake2_128.html) | X | | +| [TwoX 128](https://crates.parity.io/frame_support/struct.Twox128.html) | | | +| [Blake2 128 Concat](https://crates.parity.io/frame_support/struct.Blake2_128Concat.html) | X | X | +| [TwoX 64 Concat](https://crates.parity.io/frame_support/struct.Twox64Concat.html) | | X | +| [Identity](https://crates.parity.io/frame_support/struct.Identity.html) | | | The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the identity function). This type of hasher should only be used when the starting key is already a @@ -176,9 +175,9 @@ cryptographic hash. ## Declaring Storage Items You can use -[the `decl_storage` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) -to easily create new runtime storage items. Here is an example of what it looks like to declare each -type of storage item: +[the `decl_storage` macro](https://crates.parity.io/frame_support/macro.decl_storage.html) to easily +create new runtime storage items. Here is an example of what it looks like to declare each type of +storage item: ```rust decl_storage! { @@ -247,7 +246,7 @@ decl_storage! { ### Genesis Config You can define -[an optional `GenesisConfig`](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html#genesisconfig) +[an optional `GenesisConfig`](https://crates.parity.io/frame_support/macro.decl_storage.html#genesisconfig) struct in order to initialize storage items in the genesis block of your blockchain. // TODO @@ -302,8 +301,8 @@ another mechanism is [bounds](#Create-Bounds). The state of a blockchain network's storage is immutable; data can be changed, but there will always be a record of these changes, and making them typically incurs costs. Because of this, it is important that data is only persisted to runtime storage when it is certain that all preconditions -have been met. In general, code blocks that may result in mutating storage should be -structured as follows: +have been met. In general, code blocks that may result in mutating storage should be structured as +follows: ```rust { @@ -358,8 +357,8 @@ Check out ### References - Visit the reference docs for the - [`decl_storage!` macro](https://substrate.dev/rustdocs/master/frame_support/macro.decl_storage.html) - for more details about the available storage declarations. + [`decl_storage!` macro](https://crates.parity.io/frame_support/macro.decl_storage.html) for more + details about the available storage declarations. - Visit the reference docs for [StorageValue](https://crates.parity.io/frame_support/storage/trait.StorageValue.html), From c14a42ac4f7708f29d9538954e02807f86c2d8c5 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Mon, 4 May 2020 18:52:36 -0700 Subject: [PATCH 21/26] Genesis configuration --- current/runtime/storage.md | 149 +++++++++++++++++++++++++++++++++++-- 1 file changed, 144 insertions(+), 5 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 569749e..19f81ac 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -243,13 +243,152 @@ decl_storage! { } ``` -### Genesis Config +### Genesis Configuration + +Substrate's runtime storage APIs include capabilities to initialize storage items in the genesis +block of your blockchain. The genesis storage configuration APIs expose a number of mechanisms for +initializing storage, all of which have entry points in the `decl_storage` macro. These mechanisms +all result in the creation of a `GenesisConfig` data type that implements +[the `sp_runtime::BuildModuleGenesisStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildModuleGenesisStorage.html) +and will be added to the storage item's module (e.g. +[`Struct pallet_balances::GenesisConfig`](https://crates.parity.io/pallet_balances/struct.GenesisConfig.html)); +storage items that are tagged for genesis configuration will have a corresponding attribute on this +data type. In order to consume a module's genesis configuration capabilities, you must include the +`Config` element when adding the module to your runtime with +[the `construct_runtime` macro](https://crates.parity.io/frame_support/macro.construct_runtime.html). +All the `GenesisConfig` types for the modules that inform a runtime will be aggregated into a single +`GenesisConfig` type for that runtime, which implements +[the `sp_runtime::BuildStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildStorage.html) +(e.g. +[`Struct node_template_runtime::GenesisConfig`](https://crates.parity.io/node_template_runtime/struct.GenesisConfig.html)); +each attribute on this type corresponds to a `GenesisConfig` from one of the runtime's modules. +Ultimately, the runtime's `GenesisConfig` is exposed by way of +[the `sc_chain_spec::ChainSpec` trait](https://crates.parity.io/sc_chain_spec/trait.ChainSpec.html). +For a complete and concrete example of using Substrate's genesis storage configuration capabilities, +refer to the `decl_storage` macro in +[the Society pallet](https://github.com/paritytech/substrate/blob/master/frame/society/src/lib.rs) +as well as the genesis configuration for the Society module's storage in +[the chain specification that ships with the Substrate code base](https://github.com/paritytech/substrate/blob/master/bin/node/cli/src/chain_spec.rs). +Keep reading for more detailed descriptions of these capabilities. + +#### `config` + +When you use the `decl_storage` macro to declare a storage item, you can provide an optional +`config` extension that will add an attribute to the module's `GenesisConfig` data type; the value +of this attribute will be used as the initial value of the storage item in your chain's genesis +block. The `config` extension takes a parameter that will determine the name of the attribute on the +`GenesisConfig` data type; this parameter is optional if [the `get` extension](#Getter-Methods) is +provided (the name of the `get` function is used as the attribute's name). + +Here is an example that demonstrates using the `config` extension with a Storage Value named `MyVal` +to create an attribute named `init_val` on the `GenesisConfig` data type for the Storage Value's +module. This attribute is then used in an example that demonstrates using the `GenesisConfig` types +to set the Storage Value's initial value in your chain's genesis block. + +In `my_module/src/lib.rs`: -You can define -[an optional `GenesisConfig`](https://crates.parity.io/frame_support/macro.decl_storage.html#genesisconfig) -struct in order to initialize storage items in the genesis block of your blockchain. +```rust +decl_storage! { + trait Store for Module as MyModule { + pub MyVal get(fn my_val) config(init_val): u64; + } +} +``` + +In `chain_spec.rs`: + +```rust +GenesisConfig { + my_module: Some(MyModuleConfig { + init_val: 221u64 + SOME_CONSTANT_VALUE, + }), +} +``` + +#### `build` + +Whereas [the `config` extension](#config) to the `decl_storage` macro allows you to configure a +module's genesis storage state within a chain specification, the `build` extension allows you to +perform this same task within the module itself (this gives you access to the module's private +functions). Like `config`, the `build` extension accepts a single parameter, but in this case the +parameter is always required and must be a closure, which is essentially a function. The `build` +closure will be invoked with a single parameter whose type will be the module's `GenesisConfig` type +(this gives you easy access to all the attributes of the `GenesisConfig` type). You may use the +`build` extension along with the `config` extension for a single storage item; in this case, the +module's `GenesisConfig` type will have an attribute that corresponds to what was set using `config` +whose value will be set in the chain specification, but it will be the value returned by the `build` +closure that will be used to set the storage item's genesis value. + +Here is an example that demonstrates using `build` to set the initial value of a storage item. In +this case, the example involves two storage items: one that represents a list of member account IDs +and another that designates a special member from the list, the prime member. The list of members is +provided by way of the `config` extension and the prime member, who is assumed to be the first +element in the list of members, is set using the `build` extension. + +In `my_module/src/lib.rs`: + +```rust +decl_storage! { + trait Store for Module as MyModule { + pub Members config(orig_ids): Vec; + pub Prime build(|config: &GenesisConfig| config.orig_ids.first().cloned()): T::AccountId; + } +} +``` -// TODO +In `chain_spec.rs`: + +```rust +GenesisConfig { + my_module: Some(MyModuleConfig { + orig_ids: LIST_OF_IDS, + }), +} +``` + +#### `add_extra_genesis` + +The `add_extra_genesis` extension to the `decl_storage` macro allows you to define a scope where +[`config`](#config) and [`build`](#build) extensions can be provided without the need to bind them +to specific storage items. You can use `config` within an `add_extra_genesis` scope to add an +attribute to the module's `GenesisConfig` data type that can be used within any `build` closure. The +`build` closures that are defined within an `add_extra_genesis` scope can be used to execute logic +without binding that logic's return value to the value of a particular storage item; this may be +desireable if you wish to invoke a private helper function within your module that sets several +storage items or invoke a function defined on some other module included within your module. + +Here is an example that encapsulates the same use case described above in the example for `build`: a +module that maintains a list of member account IDs along with a designated prime member. In this +case, however, the `add_extra_genesis` extension is used to define a `GenesisConfig` attribute that +is not bound to particular storage item as well as a `build` closure that will call a private +function on the module to set the values of multiple storage items. For the purposes of this +example, the implementation of the private helper function (`initialize_members`) is left to your +imagination. + +In `my_module/src/lib.rs`: + +```js +decl_storage! { + trait Store for Module as MyModule { + pub Members: Vec; + pub Prime: T::AccountId; + } + add_extra_genesis { + config(orig_ids): Vec; + build(|config| Module::::initialize_members(&config.members)) + } +} +``` + +In `chain_spec.rs`: + +```rust +GenesisConfig { + my_module: Some(MyModuleConfig { + orig_ids: LIST_OF_IDS, + }), +} +``` ## Accessing Storage Items From 05bc5136b0238adb48e4f3b09d2e0491a9a6658e Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 5 May 2020 09:08:33 -0700 Subject: [PATCH 22/26] Small enhancements per @kianenigma --- current/runtime/storage.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 19f81ac..e3c8a73 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -94,14 +94,13 @@ Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you s map's keys, you may be able to iterate across its keys and values. Because maps are often used to track unbounded sets of data (account balances, for example) it is especially likely to exceed block production time by iterating over maps in their entirety within the runtime. Furthermore, because -accessing the elements of a map requires more pointer dereferencing than accessing the elements of a -native list, maps are significantly _more_ costly than lists to iterate over with respect to time. -This is not to say that it is "wrong" to iterate over maps in your runtime; in general Substrate -focuses on "[first principles](#Best-Practices)" as opposed to hard and fast rules of right and -wrong. Being efficient within the runtime of a blockchain is an important first principle of -Substrate and this information is designed to help you understand _all_ of Substrate's storage -capabilities and use them in a way that respects the important first principles around which they -were designed. +accessing the elements of a map requires more database reads than accessing the elements of a native +list, maps are significantly _more_ costly than lists to iterate over with respect to time. This is +not to say that it is "wrong" to iterate over maps in your runtime; in general Substrate focuses on +"[first principles](#Best-Practices)" as opposed to hard and fast rules of right and wrong. Being +efficient within the runtime of a blockchain is an important first principle of Substrate and this +information is designed to help you understand _all_ of Substrate's storage capabilities and use +them in a way that respects the important first principles around which they were designed. ##### Iterable Storage Map Methods @@ -200,8 +199,8 @@ used. In the example above, all the storage items except `SomePrivateValue` are made public by way of the `pub` keyword. Blockchain storage is always publicly [visible from _outside_ of the runtime](#Accessing-Storage-Items); the visibility of Substrate -storage items only impacts whether or not other runtime pallets will be able to access the storage -item. +storage items only impacts whether or not other pallets _within_ the runtime will be able to access +the storage item. ### Getter Methods From ddef06da1ed7ffab17dbff1f6b4ee8671db9adcc Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Tue, 5 May 2020 10:05:01 -0700 Subject: [PATCH 23/26] Non-transparent hashers are deprecated --- current/runtime/storage.md | 57 +++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 26 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index e3c8a73..e1a542f 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -90,17 +90,17 @@ that is similar to that of Storage Values. #### Iterable Storage Maps -Depending on [the hashing algorithm](#Transparent-Hashing-Algorithms) that you select to generate a -map's keys, you may be able to iterate across its keys and values. Because maps are often used to -track unbounded sets of data (account balances, for example) it is especially likely to exceed block -production time by iterating over maps in their entirety within the runtime. Furthermore, because -accessing the elements of a map requires more database reads than accessing the elements of a native -list, maps are significantly _more_ costly than lists to iterate over with respect to time. This is -not to say that it is "wrong" to iterate over maps in your runtime; in general Substrate focuses on -"[first principles](#Best-Practices)" as opposed to hard and fast rules of right and wrong. Being -efficient within the runtime of a blockchain is an important first principle of Substrate and this -information is designed to help you understand _all_ of Substrate's storage capabilities and use -them in a way that respects the important first principles around which they were designed. +Substrate Storage Maps are iterable with respect to their keys and values. Because maps are often +used to track unbounded sets of data (account balances, for example) it is especially likely to +exceed block production time by iterating over maps in their entirety within the runtime. +Furthermore, because accessing the elements of a map requires more database reads than accessing the +elements of a native list, maps are significantly _more_ costly than lists to iterate over with +respect to time. This is not to say that it is "wrong" to iterate over maps in your runtime; in +general Substrate focuses on "[first principles](#Best-Practices)" as opposed to hard and fast rules +of right and wrong. Being efficient within the runtime of a blockchain is an important first +principle of Substrate and this information is designed to help you understand _all_ of Substrate's +storage capabilities and use them in a way that respects the important first principles around which +they were designed. ##### Iterable Storage Map Methods @@ -123,10 +123,13 @@ Storage Double Maps, the `iter` and `drain` methods require a parameter, i.e. th #### Hashing Algorithms As mentioned above, a novel feature of Substrate Storage Maps is that they allow developers to -specify the hashing algorithm that will be used when generating a map's keys. A Rust object that is -used to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that -are available to Substrate developers can be described in two ways: whether or not they are -cryptographic and whether or not they produce output that is transparent. +specify the hashing algorithm that will be used to generate a map's keys. A Rust object that is used +to encapsulate hashing logic is referred to as a "hasher". Broadly speaking, the hashers that are +available to Substrate developers can be described in two ways: whether or not they are +cryptographic and whether or not they produce output that is transparent. For the sake of +completeness, the characteristics of non-transparent hashing algorithms are described below, but +keep in mind that any hasher that does not produce transparent output has been deprecated for use +within FRAME-based blockchains. ##### Cryptographic Hashing Algorithms @@ -149,23 +152,25 @@ used. A transparent hashing algorithm is one that makes it easy to discover and verify the input that was used to generate a given output. In Substrate, hashing algorithms are made transparent by concatenating the algorithm's input to its output. This makes it trivial for users to retrieve a -key's original unhashed value and verify it if they'd like (by re-hashing it). It is generally -recommended to use transparent hashing algorithms for your runtime's Storage Maps. In fact, it is -_necessary_ to use a transparent hashing algorithm if you would like access -[iterable map](#Iterable-Storage-Maps) capabilities. +key's original unhashed value and verify it if they'd like (by re-hashing it). The creators of +Substrate have **deprecated the use of non-transparent hashers** within FRAME-based runtimes, so +this information is provided primarily for completeness. In fact, it is _necessary_ to use a +transparent hashing algorithm if you would like access [iterable map](#Iterable-Storage-Maps) +capabilities. Refer to [the advanced storage documentation](../advanced/storage#Storage-Map-Keys) to +learn more about the important capabilities that transparent hashing algorithms expose. ##### Common Substrate Hashers This table lists some common hashers used in Substrate and denotes those that are cryptographic and those that are transparent: -| Hasher | Cryptographic | Transparent | -| ---------------------------------------------------------------------------------------- | ------------- | ----------- | -| [Blake2 128](https://crates.parity.io/frame_support/struct.Blake2_128.html) | X | | -| [TwoX 128](https://crates.parity.io/frame_support/struct.Twox128.html) | | | -| [Blake2 128 Concat](https://crates.parity.io/frame_support/struct.Blake2_128Concat.html) | X | X | -| [TwoX 64 Concat](https://crates.parity.io/frame_support/struct.Twox64Concat.html) | | X | -| [Identity](https://crates.parity.io/frame_support/struct.Identity.html) | | | +| Hasher | Cryptographic | Transparent | +| ------------------------------------------------------------------------------------------ | ------------- | ----------- | +| [Blake2 128 Concat](https://crates.parity.io/frame_support/struct.Blake2_128Concat.html) | X | X | +| [TwoX 64 Concat](https://crates.parity.io/frame_support/struct.Twox64Concat.html) | | X | +| [Identity](https://crates.parity.io/frame_support/struct.Identity.html) | | | +| [Blake2 128](https://crates.parity.io/frame_support/struct.Blake2_128.html) **DEPRECATED** | X | | +| [TwoX 128](https://crates.parity.io/frame_support/struct.Twox128.html) **DEPRECATED** | | | The Identity hasher encapsulates a hashing algorithm that has an output equal to its input (the identity function). This type of hasher should only be used when the starting key is already a From 4b4b0655ac05601c1185734838b7a3b6d1127942 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Thu, 7 May 2020 19:07:14 -0700 Subject: [PATCH 24/26] Clarify generic storage types Co-authored-by: thiolliere --- current/runtime/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index e1a542f..21635f3 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -188,7 +188,7 @@ decl_storage! { trait Store for Module as Example { SomePrivateValue: u32; pub SomePrimitiveValue get(fn some_primitive_value): u32; - // complex types are prefaced by T:: + // types can make use of generic `T: Trait` pub SomeComplexValue: T::AccountId; pub SomeMap get(fn some_map): map hasher(blake2_128_concat) T::AccountId => u32; pub SomeDoubleMap: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::AccountId => u32; From ebdea8e58cd7515bc7445dfdc2189f3a3ff3a261 Mon Sep 17 00:00:00 2001 From: Dan Forbes Date: Thu, 7 May 2020 19:15:03 -0700 Subject: [PATCH 25/26] Clarifications per review by @thiolliere --- current/runtime/storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 21635f3..8891101 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -254,7 +254,7 @@ block of your blockchain. The genesis storage configuration APIs expose a number initializing storage, all of which have entry points in the `decl_storage` macro. These mechanisms all result in the creation of a `GenesisConfig` data type that implements [the `sp_runtime::BuildModuleGenesisStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildModuleGenesisStorage.html) -and will be added to the storage item's module (e.g. +and will be added to the module that contains the storage items (e.g. [`Struct pallet_balances::GenesisConfig`](https://crates.parity.io/pallet_balances/struct.GenesisConfig.html)); storage items that are tagged for genesis configuration will have a corresponding attribute on this data type. In order to consume a module's genesis configuration capabilities, you must include the @@ -451,9 +451,9 @@ follows: { // all checks and throwing code go here - // all storage writes go here; no throwing code below this line + // ** no throwing code below this line ** - // all event emissions go here + // all event emissions & storage writes go here } ``` From bb20b77e7c1aac5ee06232d5a6bc035a1fb8ed78 Mon Sep 17 00:00:00 2001 From: joepetrowski Date: Fri, 8 May 2020 11:34:49 +0200 Subject: [PATCH 26/26] final nits --- current/advanced/storage.md | 17 ++--- current/runtime/storage.md | 127 +++++++++++++++++------------------- 2 files changed, 69 insertions(+), 75 deletions(-) diff --git a/current/advanced/storage.md b/current/advanced/storage.md index 5dbd4f4..9479245 100644 --- a/current/advanced/storage.md +++ b/current/advanced/storage.md @@ -11,9 +11,10 @@ simple key-value store. ## Key-Value Database Substrate implements its storage database with [RocksDB](https://rocksdb.org/), a persistent -key-value store for fast storage environments. +key-value store for fast storage environments. It also supports an experimental +[Parity DB](https://github.com/paritytech/parity-db). -This is used for all the components of Substrate that require persistent storage, such as: +The DB is used for all the components of Substrate that require persistent storage, such as: - Substrate clients - Substrate light-clients @@ -36,14 +37,13 @@ simply comparing their trie roots. Accessing trie data is costly. Each read operation takes O(log N) time, where N is the number of elements stored in the trie. To mitigate this, we use a key-value cache. -All trie nodes are stored in RocksDB and part of the trie state can get pruned, i.e. a key-value -pair can be deleted from the storage when it is out of pruning range for non-archive nodes. We do -not use [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance -reasons. +All trie nodes are stored in the DB and part of the trie state can get pruned, i.e. a key-value pair +can be deleted from storage when it is out of pruning range for non-archive nodes. We do not use +[reference counting](http://en.wikipedia.org/wiki/Reference_counting) for performance reasons. ### State Trie -Substrate based chains have a single main trie, called the state trie, whose root hash is placed in +Substrate-based chains have a single main trie, called the state trie, whose root hash is placed in each block header. This is used to easily verify the state of the blockchain and provide a basis for light clients to verify proofs. @@ -173,6 +173,7 @@ represents a key in the same map, and that map is identified by concatenating tw each of which are 128-bits or 32 hexadecimal characters. After discarding this portion of the second element in the list, you are left with `0x32a5935f6edc617ae178fef9eb1e211fbe5ddb1579b72e84524fc29e78609e3caf42e85aa118ebfe0b0ad404b5bdd25f`. + You saw in the previous example that this represents the Blake2 128 Concat hash of some [SCALE](./codec)-encoded account ID. The Blake 128 Concat hashing algorithm consists of appending (concatenating) the hashing algorithm's input to its Blake 128 hash. This means that the first 128 @@ -189,7 +190,7 @@ example, after you remove the first 32 hexadecimal characters that represent the Substrate's [FRAME Support crate](https://substrate.dev/rustdocs/master/frame_support/index.html) provides utilities for generating unique, deterministic keys for your runtime's storage items. These -storage items are placed in the state trie[#Trie-Abstraction] and are accessible by +storage items are placed in the [state trie](#Trie-Abstraction) and are accessible by [querying the trie by key](#Querying-Storage). ## Next Steps diff --git a/current/runtime/storage.md b/current/runtime/storage.md index 8891101..6f4e71f 100644 --- a/current/runtime/storage.md +++ b/current/runtime/storage.md @@ -155,7 +155,7 @@ concatenating the algorithm's input to its output. This makes it trivial for use key's original unhashed value and verify it if they'd like (by re-hashing it). The creators of Substrate have **deprecated the use of non-transparent hashers** within FRAME-based runtimes, so this information is provided primarily for completeness. In fact, it is _necessary_ to use a -transparent hashing algorithm if you would like access [iterable map](#Iterable-Storage-Maps) +transparent hashing algorithm if you would like to access [iterable map](#Iterable-Storage-Maps) capabilities. Refer to [the advanced storage documentation](../advanced/storage#Storage-Map-Keys) to learn more about the important capabilities that transparent hashing algorithms expose. @@ -185,19 +185,19 @@ storage item: ```rust decl_storage! { - trait Store for Module as Example { - SomePrivateValue: u32; - pub SomePrimitiveValue get(fn some_primitive_value): u32; - // types can make use of generic `T: Trait` - pub SomeComplexValue: T::AccountId; - pub SomeMap get(fn some_map): map hasher(blake2_128_concat) T::AccountId => u32; - pub SomeDoubleMap: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::AccountId => u32; - } + trait Store for Module as Example { + SomePrivateValue: u32; + pub SomePrimitiveValue get(fn some_primitive_value): u32; + // types can make use of the generic `T: Trait` + pub SomeComplexValue: T::AccountId; + pub SomeMap get(fn some_map): map hasher(blake2_128_concat) T::AccountId => u32; + pub SomeDoubleMap: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::AccountId => u32; + } } ``` -Notice that the map storage items specify [the hashing algorithm](#Hashing-Algorithms) that will be -used. +Notice that the map's storage items specify [the hashing algorithm](#Hashing-Algorithms) that will +be used. ### Visibility @@ -205,7 +205,7 @@ In the example above, all the storage items except `SomePrivateValue` are made p `pub` keyword. Blockchain storage is always publicly [visible from _outside_ of the runtime](#Accessing-Storage-Items); the visibility of Substrate storage items only impacts whether or not other pallets _within_ the runtime will be able to access -the storage item. +a storage item. ### Getter Methods @@ -225,9 +225,9 @@ Here is an example that implements a getter method named `some_value` for a Stor ```rust decl_storage! { - trait Store for Module as Example { - pub SomeValue get(fn some_value): u64; - } + trait Store for Module as Example { + pub SomeValue get(fn some_value): u64; + } } ``` @@ -241,9 +241,9 @@ Here is an example of specifying the default value for all items in a map: ```rust decl_storage! { - trait Store for Module as Example { - pub SomeMap: map u64 => u64 = 1337; - } + trait Store for Module as Example { + pub SomeMap: map u64 => u64 = 1337; + } } ``` @@ -253,7 +253,7 @@ Substrate's runtime storage APIs include capabilities to initialize storage item block of your blockchain. The genesis storage configuration APIs expose a number of mechanisms for initializing storage, all of which have entry points in the `decl_storage` macro. These mechanisms all result in the creation of a `GenesisConfig` data type that implements -[the `sp_runtime::BuildModuleGenesisStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildModuleGenesisStorage.html) +[the `BuildModuleGenesisStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildModuleGenesisStorage.html) and will be added to the module that contains the storage items (e.g. [`Struct pallet_balances::GenesisConfig`](https://crates.parity.io/pallet_balances/struct.GenesisConfig.html)); storage items that are tagged for genesis configuration will have a corresponding attribute on this @@ -262,23 +262,22 @@ data type. In order to consume a module's genesis configuration capabilities, yo [the `construct_runtime` macro](https://crates.parity.io/frame_support/macro.construct_runtime.html). All the `GenesisConfig` types for the modules that inform a runtime will be aggregated into a single `GenesisConfig` type for that runtime, which implements -[the `sp_runtime::BuildStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildStorage.html) -(e.g. +[the `BuildStorage` trait](https://crates.parity.io/sp_runtime/trait.BuildStorage.html) (e.g. [`Struct node_template_runtime::GenesisConfig`](https://crates.parity.io/node_template_runtime/struct.GenesisConfig.html)); each attribute on this type corresponds to a `GenesisConfig` from one of the runtime's modules. Ultimately, the runtime's `GenesisConfig` is exposed by way of -[the `sc_chain_spec::ChainSpec` trait](https://crates.parity.io/sc_chain_spec/trait.ChainSpec.html). -For a complete and concrete example of using Substrate's genesis storage configuration capabilities, -refer to the `decl_storage` macro in +[the `ChainSpec` trait](https://crates.parity.io/sc_chain_spec/trait.ChainSpec.html). For a complete +and concrete example of using Substrate's genesis storage configuration capabilities, refer to the +`decl_storage` macro in [the Society pallet](https://github.com/paritytech/substrate/blob/master/frame/society/src/lib.rs) -as well as the genesis configuration for the Society module's storage in +as well as the genesis configuration for the Society pallet's storage in [the chain specification that ships with the Substrate code base](https://github.com/paritytech/substrate/blob/master/bin/node/cli/src/chain_spec.rs). Keep reading for more detailed descriptions of these capabilities. #### `config` When you use the `decl_storage` macro to declare a storage item, you can provide an optional -`config` extension that will add an attribute to the module's `GenesisConfig` data type; the value +`config` extension that will add an attribute to the pallet's `GenesisConfig` data type; the value of this attribute will be used as the initial value of the storage item in your chain's genesis block. The `config` extension takes a parameter that will determine the name of the attribute on the `GenesisConfig` data type; this parameter is optional if [the `get` extension](#Getter-Methods) is @@ -293,9 +292,9 @@ In `my_module/src/lib.rs`: ```rust decl_storage! { - trait Store for Module as MyModule { - pub MyVal get(fn my_val) config(init_val): u64; - } + trait Store for Module as MyModule { + pub MyVal get(fn my_val) config(init_val): u64; + } } ``` @@ -303,9 +302,9 @@ In `chain_spec.rs`: ```rust GenesisConfig { - my_module: Some(MyModuleConfig { - init_val: 221u64 + SOME_CONSTANT_VALUE, - }), + my_module: Some(MyModuleConfig { + init_val: 221u64 + SOME_CONSTANT_VALUE, + }), } ``` @@ -316,10 +315,10 @@ module's genesis storage state within a chain specification, the `build` extensi perform this same task within the module itself (this gives you access to the module's private functions). Like `config`, the `build` extension accepts a single parameter, but in this case the parameter is always required and must be a closure, which is essentially a function. The `build` -closure will be invoked with a single parameter whose type will be the module's `GenesisConfig` type +closure will be invoked with a single parameter whose type will be the pallet's `GenesisConfig` type (this gives you easy access to all the attributes of the `GenesisConfig` type). You may use the `build` extension along with the `config` extension for a single storage item; in this case, the -module's `GenesisConfig` type will have an attribute that corresponds to what was set using `config` +pallet's `GenesisConfig` type will have an attribute that corresponds to what was set using `config` whose value will be set in the chain specification, but it will be the value returned by the `build` closure that will be used to set the storage item's genesis value. @@ -333,10 +332,10 @@ In `my_module/src/lib.rs`: ```rust decl_storage! { - trait Store for Module as MyModule { - pub Members config(orig_ids): Vec; - pub Prime build(|config: &GenesisConfig| config.orig_ids.first().cloned()): T::AccountId; - } + trait Store for Module as MyModule { + pub Members config(orig_ids): Vec; + pub Prime build(|config: &GenesisConfig| config.orig_ids.first().cloned()): T::AccountId; + } } ``` @@ -344,18 +343,18 @@ In `chain_spec.rs`: ```rust GenesisConfig { - my_module: Some(MyModuleConfig { - orig_ids: LIST_OF_IDS, - }), + my_module: Some(MyModuleConfig { + orig_ids: LIST_OF_IDS, + }), } ``` #### `add_extra_genesis` -The `add_extra_genesis` extension to the `decl_storage` macro allows you to define a scope where +The `add_extra_genesis` extension to the `decl_storage` macro allows you to define a scope where the [`config`](#config) and [`build`](#build) extensions can be provided without the need to bind them to specific storage items. You can use `config` within an `add_extra_genesis` scope to add an -attribute to the module's `GenesisConfig` data type that can be used within any `build` closure. The +attribute to the pallet's `GenesisConfig` data type that can be used within any `build` closure. The `build` closures that are defined within an `add_extra_genesis` scope can be used to execute logic without binding that logic's return value to the value of a particular storage item; this may be desireable if you wish to invoke a private helper function within your module that sets several @@ -373,14 +372,14 @@ In `my_module/src/lib.rs`: ```js decl_storage! { - trait Store for Module as MyModule { - pub Members: Vec; - pub Prime: T::AccountId; - } - add_extra_genesis { - config(orig_ids): Vec; - build(|config| Module::::initialize_members(&config.members)) - } + trait Store for Module as MyModule { + pub Members: Vec; + pub Prime: T::AccountId; + } + add_extra_genesis { + config(orig_ids): Vec; + build(|config| Module::::initialize_members(&config.members)) + } } ``` @@ -388,9 +387,9 @@ In `chain_spec.rs`: ```rust GenesisConfig { - my_module: Some(MyModuleConfig { - orig_ids: LIST_OF_IDS, - }), + my_module: Some(MyModuleConfig { + orig_ids: LIST_OF_IDS, + }), } ``` @@ -411,7 +410,7 @@ directly by way of the RPC server. Substrate's goal is to provide a flexible framework that allows people to build the blockchain that suits their needs - the creators of Substrate tend not to think in terms of "right" or "wrong". That being said, the Substrate codebase adheres to a number of best practices in order to promote the -creation of blockchain networks that are secure, performant and maintainable in the long-term. The +creation of blockchain networks that are secure, performant, and maintainable in the long-term. The following sections outline best practices for using Substrate storage and also describe the important first principles that motivated them. @@ -424,7 +423,7 @@ capabilities (e.g. [the Democracy pallet's `propose` dispatchable](https://crates.parity.io/pallet_democracy/enum.Call.html#variant.propose)) allow network participants to vote on the _hash_ of a dispatchable call, which is always bounded in size, as opposed to the call itself, which may be unbounded in length. This is especially true in -the case of runtime upgrades where the dispatchable call takes an entire runtime WASM blob as its +the case of runtime upgrades where the dispatchable call takes an entire runtime Wasm blob as its parameter. Because these governance mechanisms are implemented _on-chain_, all the information that is needed to come to consensus on the state of a given proposal must also be stored on-chain - this includes _what_ is being voted on. However, by binding an on-chain proposal to its hash, Substrate's @@ -433,7 +432,7 @@ with a proposal on-chain until _after_ it has been approved. This means that sto on proposals that fail. Once a proposal has passed, someone can initiate the actual dispatchable call (including all its parameters), which will be hashed and compared to the hash in the proposal. Another common pattern for using hashes to minimize data that is stored on-chain is to store the -metadata associated with an object in [IPFS](https://ipfs.io/); this means that only the IPFS +pre-image associated with an object in [IPFS](https://ipfs.io/); this means that only the IPFS location (a hash that is bounded in size) needs to be stored on-chain. Hashes are only one mechanism that can be used to control the size of runtime storage. An example of @@ -441,11 +440,10 @@ another mechanism is [bounds](#Create-Bounds). ### Verify First, Write Last -The state of a blockchain network's storage is immutable; data can be changed, but there will always -be a record of these changes, and making them typically incurs costs. Because of this, it is -important that data is only persisted to runtime storage when it is certain that all preconditions -have been met. In general, code blocks that may result in mutating storage should be structured as -follows: +Substrate does not cache state prior to extrinsic dispatch. Instead, it applies changes directly as +they are invoked. If an extrinsic fails, any state changes will persist. Because of this, it is +important not to make any storage mutations until it is certain that all preconditions have been +met. In general, code blocks that may result in mutating storage should be structured as follows: ```rust { @@ -482,10 +480,6 @@ developers control over how much space in storage these lists may occupy, the Ut requires users to configure a bound on this number that will be included as a [precondition](#Verify-First-Write-Last) before anything is written to storage. -## Child Storage Tries - -TODO - ## Next Steps ### Learn More @@ -502,7 +496,6 @@ Check out - Visit the reference docs for the [`decl_storage!` macro](https://crates.parity.io/frame_support/macro.decl_storage.html) for more details about the available storage declarations. - - Visit the reference docs for [StorageValue](https://crates.parity.io/frame_support/storage/trait.StorageValue.html), [StorageMap](https://crates.parity.io/frame_support/storage/trait.StorageMap.html) and