From ae45cd4cb814b59ab4918e6a0ac4fd300b86bf8c Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Fri, 7 May 2021 21:07:50 +0800 Subject: [PATCH 01/19] Add proposal for supporting big transactions --- docs/src/proposals/big-transactions.md | 44 ++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/src/proposals/big-transactions.md diff --git a/docs/src/proposals/big-transactions.md b/docs/src/proposals/big-transactions.md new file mode 100644 index 00000000000000..89d2aea3149c24 --- /dev/null +++ b/docs/src/proposals/big-transactions.md @@ -0,0 +1,44 @@ +# Big Transactions + +## Problem +Solana networking packets may not exceed the ipv6 MTU size to ensure fast and +reliable network transmission of cluster information over UDP. Solana's +networking stack uses a conservative MTU size of 1280 bytes which after +accounting for networking headers, leaves 1232 bytes for packet data like +serialized transactions. + +Developers building applications on Solana have had to design their on-chain +programs in a way that works within the transaction size limit constraint. One +common work-around is to store temporary state on-chain and process the data in +later transactions. This is the approach used by the BPF loader program for +deploying Solana programs. + +However, this workaround doesn't work well when users want to compose many +on-chain programs in a single atomic transaction because with more composition, +comes more account inputs. Each transaction account input takes up 32 bytes and +as programs get more complex, especially defi protocols, the number accounts +needed increases quite quickly. There is no available workaround for increasing +the number of accounts used in a single transaction, it's capped at less than 30 +accounts after signatures and other metadata is accounted for. + +## Proposed Solutions + +1) Account input compression + +Solana can provide a way to map commonly used addresses to small integer values. +This mapping can be computed on epoch boundaries by all validators or registered +on-chain in state managed by a new on-chain program. + +This approach will require a new serialization format for compressed +transactions which can fit within a single network packet. Compressed +transactions will need to be identified as different from standard transactions +somehow. Either by a special encoding or using dedicated UDP ports for this new +type of transaction. + +2) Transaction builder program + +Solana can provide a new on-chain program which allows "Big" transactions to be +constructed on-chain by normal transactions. Once the transaction is +constructed, a final "Execute" transaction can trigger a node to process the big +transaction as a normal transaction without needing to fit it into an MTU sized +packet. \ No newline at end of file From 9a5dabfd9eadf5c2146bc208b80bc0c5a8d3bab1 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Fri, 21 May 2021 15:25:34 -0700 Subject: [PATCH 02/19] account index program --- docs/src/proposals/big-transactions.md | 196 +++++++++++++++++++++---- 1 file changed, 169 insertions(+), 27 deletions(-) diff --git a/docs/src/proposals/big-transactions.md b/docs/src/proposals/big-transactions.md index 89d2aea3149c24..78d47715ac40b1 100644 --- a/docs/src/proposals/big-transactions.md +++ b/docs/src/proposals/big-transactions.md @@ -1,39 +1,162 @@ # Big Transactions ## Problem -Solana networking packets may not exceed the ipv6 MTU size to ensure fast and -reliable network transmission of cluster information over UDP. Solana's -networking stack uses a conservative MTU size of 1280 bytes which after -accounting for networking headers, leaves 1232 bytes for packet data like -serialized transactions. - -Developers building applications on Solana have had to design their on-chain -programs in a way that works within the transaction size limit constraint. One -common work-around is to store temporary state on-chain and process the data in + +Messages transmitted to Solana validators must not exceed the IPv6 MTU size to +ensure fast and reliable network transmission of cluster info over UDP. +Solana's networking stack uses a conservative MTU size of 1280 bytes which, +after accounting for headers, leaves 1232 bytes for packet data like serialized +transactions. + +Developers building applications on Solana must design their on-chain program +interfaces within the above transaction size limit constraint. One common +work-around is to store state temporarily on-chain and consume that state in later transactions. This is the approach used by the BPF loader program for deploying Solana programs. -However, this workaround doesn't work well when users want to compose many -on-chain programs in a single atomic transaction because with more composition, -comes more account inputs. Each transaction account input takes up 32 bytes and -as programs get more complex, especially defi protocols, the number accounts -needed increases quite quickly. There is no available workaround for increasing -the number of accounts used in a single transaction, it's capped at less than 30 -accounts after signatures and other metadata is accounted for. +However, this workaround doesn't work well when developers compose many on-chain +programs in a single atomic transaction. With more composition comes more +account inputs, each of which takes up 32 bytes. There is currently no available +workaround for increasing the number of accounts used in a single transaction +since each transaction must list all accounts that it needs to properly lock +accounts for parallel execution. Therefore the current cap is about 35 accounts +after accounting for signatures and other transaction metadata. + +## Proposed Solution + +Introduce a new on-chain account indexing program which stores account address +mappings and add a new transaction format which supports concise account +references through the new on-chain account indexes. + +### Account Indexing Program + +Here we describe a contract-based solution to the problem, whereby a protocol +developer or end-user can create collections of related accounts on-chain for +concise use in a transaction's account inputs. This approach is similar to page +tables used in operating systems to succinctly map virtual addresses to physical +memory. + +After addresses are stored on-chain in an index account, they may be succinctly +referenced from a transaction using an index rather than a full 32 byte address. +This will require a new transaction format to make use of these succinct indexes +as well as runtime handling for looking up and loading accounts from the +on-chain indexes. + +#### State + +Once created, index accounts may not be deleted. Stored addresses should be +append only so that once an address is stored in an index account, it may not be +removed later. + +Since transactions use a u16 offset to look up addresses, index accounts can +store up to 2^16 addresses each. Anyone may create an index account of any size +as long as its big enough to store the necessary metadata. In addition to +stored addresses, index accounts must also track the latest count of stored +addresses and an authority which must be a present signer for all index +modifications. + +#### Program controlled indexes + +If the authority of an index account is controlled by a program, more +sophisticated indexes could be built with governance features or price curves +for new index addresses. + +### Versioned Transactions + +In order to allow accounts to be referenced more succinctly, the structure of +serialized transactions must be modified. This means that the new transaction +format must be distinguished from the current transaction format. + +Current transactions can fit at most 19 signatures (64-bytes each) but the +message format encodes the number of required signers as a `u8`. Since the +upper bit of the `u8` will never be set for a valid transaction, we can enable +it to denote whether a transaction should be decoded with the versioned format +or not. + +#### New Transaction Format + +```rust +pub struct VersionedMessage { + /// Version of encoded message. + /// The max encoded version is 2^7 - 1 due to the ignored upper disambiguation bit + pub version: u8, + pub header: MessageHeader, + /// Number of read-only account inputs specified thru indexes + pub num_readonly_indexed_accounts: u8, + #[serde(with = "short_vec")] + pub account_keys: Vec, + /// All the account indexes used by this transaction + #[serde(with = "short_vec")] + pub account_indexes: Vec, + pub recent_blockhash: Hash, + /// Compiled instructions stay the same, account indexes continue to be stored + /// as a u8 which means the max number of account_indexes + account_keys is 256. + #[serde(with = "short_vec")] + pub instructions: Vec, +} + +pub struct AccountIndex { + pub account_key_offset: u8, + // 1-3 bytes used to lookup address in index account + pub index_account_offset: CompactU16, +} +``` -## Proposed Solutions +#### Size changes -1) Account input compression +- Extra byte for version field +- Extra byte for number of total account index inputs +- Extra byte for number of readonly account index inputs +- Most indexes will be compact and use 2 bytes + index address +- Cost of each additional index account is ~2 bytes -Solana can provide a way to map commonly used addresses to small integer values. -This mapping can be computed on epoch boundaries by all validators or registered -on-chain in state managed by a new on-chain program. +### Limitations -This approach will require a new serialization format for compressed -transactions which can fit within a single network packet. Compressed -transactions will need to be identified as different from standard transactions -somehow. Either by a special encoding or using dedicated UDP ports for this new -type of transaction. +- Max of 256 accounts may be specified in a transaction because u8 is used by compiled +instructions to index into transaction message account keys. +Indexes can hold up to 2^16 keys. Smaller indexes is ok. Each index is then u16 +- Transaction signers may not be specified using an on-chain account index, the +full address of each signer must be serialized in the transaction. This ensures +that the performance of transaction signature checks is not affected. + +## Security Concerns + +### Resource consumption + +Enabling more account inputs in a transaction allows for more program +invocations, write-locks, and data reads / writes. Before indexes are live, we +need transaction-wide compute limits and increased costs for write locks and +data reads. + +### Front running + +If the addresses listed within an index account are modifiable, front running +attacks could modify which index accounts are accessed from a later transaction. +For this reason, we propose that any stored address is immutable and that index +accounts themselves may not be removed. + +### Denial of service + +Index accounts will be read very frequently and will therefore be a more high +profile target for denial of service attacks through write locks similar to +sysvar accounts. + +Since stored accounts inside index accounts are immutable, reads and writes +to index accounts could be parallelized as long as all referenced addresses +are for indexes less than the current number of addresses stored. + +## Other Proposals + +1) Account prefixes + +Needing to pre-register accounts in an on-chain index is cumbersome because it +adds an extra step for transaction processing. Instead, Solana transactions +could use variable length address prefixes to specify accounts. These prefix +shortcuts can save on data usage without needing to setup on-chain state. + +However, this model requires nodes to keep a mapping of prefixes to active account +addresses. Attackers can create accounts with the same prefix as a popular account +to disrupt transactions. 2) Transaction builder program @@ -41,4 +164,23 @@ Solana can provide a new on-chain program which allows "Big" transactions to be constructed on-chain by normal transactions. Once the transaction is constructed, a final "Execute" transaction can trigger a node to process the big transaction as a normal transaction without needing to fit it into an MTU sized -packet. \ No newline at end of file +packet. + +The UX of this approach is tricky. A user could in theory sign a big transaction +but it wouldn't be great if they had to use their wallet to sign multiple +transactions to build that transaction that they already signed and approved. This +could be a use-case for transaction relay services, though. A user could pay a +relayer to construct the large pre-signed transaction on-chain for them. + +In order to prevent the large transaction from being reconstructed and replayed, +a nonce counter system would be necessary as well. + +3) Epoch account indexes + +Similarly to leader schedule calculation, validators could create a global index +of the most accessed accounts in the previous epoch and make that index +available to transactions in the following epoch. + +This approach has a downside of only updating the index at epoch boundaries +which means there would be a few day delay before popular new accounts could be +referenced. From 006608eafb7a9d31fdfaadee829a2a20c8e2961f Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Fri, 21 May 2021 15:28:49 -0700 Subject: [PATCH 03/19] fix formatting --- docs/src/proposals/big-transactions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/proposals/big-transactions.md b/docs/src/proposals/big-transactions.md index 78d47715ac40b1..da779b6508c6ed 100644 --- a/docs/src/proposals/big-transactions.md +++ b/docs/src/proposals/big-transactions.md @@ -81,7 +81,7 @@ pub struct VersionedMessage { /// The max encoded version is 2^7 - 1 due to the ignored upper disambiguation bit pub version: u8, pub header: MessageHeader, - /// Number of read-only account inputs specified thru indexes + /// Number of read-only account inputs specified thru indexes pub num_readonly_indexed_accounts: u8, #[serde(with = "short_vec")] pub account_keys: Vec, From 94d9a744f199a34571461ec2dc7ea590a1aa1abd Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Wed, 2 Jun 2021 15:33:53 -0700 Subject: [PATCH 04/19] review feedback --- ...ctions.md => page-indexed-transactions.md} | 47 +++++++++++++------ 1 file changed, 33 insertions(+), 14 deletions(-) rename docs/src/proposals/{big-transactions.md => page-indexed-transactions.md} (79%) diff --git a/docs/src/proposals/big-transactions.md b/docs/src/proposals/page-indexed-transactions.md similarity index 79% rename from docs/src/proposals/big-transactions.md rename to docs/src/proposals/page-indexed-transactions.md index da779b6508c6ed..d15fc1edda4d53 100644 --- a/docs/src/proposals/big-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -1,4 +1,4 @@ -# Big Transactions +# Page-Indexed Transactions ## Problem @@ -26,7 +26,7 @@ after accounting for signatures and other transaction metadata. Introduce a new on-chain account indexing program which stores account address mappings and add a new transaction format which supports concise account -references through the new on-chain account indexes. +references through on-chain account indexes. ### Account Indexing Program @@ -44,9 +44,10 @@ on-chain indexes. #### State -Once created, index accounts may not be deleted. Stored addresses should be -append only so that once an address is stored in an index account, it may not be -removed later. +Index accounts must be rent-exempt and may not be deleted. Stored addresses +should be append only so that once an address is stored in an index account, it +may not be removed later. Index accounts must be pre-allocated since Solana does +not support reallocation for accounts yet. Since transactions use a u16 offset to look up addresses, index accounts can store up to 2^16 addresses each. Anyone may create an index account of any size @@ -64,14 +65,16 @@ for new index addresses. ### Versioned Transactions In order to allow accounts to be referenced more succinctly, the structure of -serialized transactions must be modified. This means that the new transaction -format must be distinguished from the current transaction format. +serialized transactions must be modified. The new transaction format should not +affect transaction processing in the Solana VM beyond the increased capacity for +accounts and instruction invocations. Invoked programs will be unaware of which +transaction format was used. -Current transactions can fit at most 19 signatures (64-bytes each) but the -message format encodes the number of required signers as a `u8`. Since the -upper bit of the `u8` will never be set for a valid transaction, we can enable -it to denote whether a transaction should be decoded with the versioned format -or not. +The new transaction format must be distinguished from the current transaction +format. Current transactions can fit at most 19 signatures (64-bytes each) but +the message header encodes `num_required_signatures` as a `u8`. Since the upper +bit of the `u8` will never be set for a valid transaction, we can enable it to +denote whether a transaction should be decoded with the versioned format or not. #### New Transaction Format @@ -110,6 +113,12 @@ pub struct AccountIndex { - Most indexes will be compact and use 2 bytes + index address - Cost of each additional index account is ~2 bytes +#### RPC changes + +The RPC API should support returning both compressed transactions for +verification as well as decompressed transactions which are easier for clients +to handle and abstract away the indexing details. + ### Limitations - Max of 256 accounts may be specified in a transaction because u8 is used by compiled @@ -118,6 +127,9 @@ Indexes can hold up to 2^16 keys. Smaller indexes is ok. Each index is then u16 - Transaction signers may not be specified using an on-chain account index, the full address of each signer must be serialized in the transaction. This ensures that the performance of transaction signature checks is not affected. +- Hardware wallets will probably not be able to display details about accounts +referenced with an index due to inability to verify on-chain data. +- Only single level indexes can be used. Recursive indexes will not be supported. ## Security Concerns @@ -145,6 +157,12 @@ Since stored accounts inside index accounts are immutable, reads and writes to index accounts could be parallelized as long as all referenced addresses are for indexes less than the current number of addresses stored. +### Duplicate accounts + +If the same account is referenced in a transaction by address as well as through +an index, the transaction should be rejected to avoid conflicts when determining +if the account is a signer or writeable. + ## Other Proposals 1) Account prefixes @@ -173,7 +191,7 @@ could be a use-case for transaction relay services, though. A user could pay a relayer to construct the large pre-signed transaction on-chain for them. In order to prevent the large transaction from being reconstructed and replayed, -a nonce counter system would be necessary as well. +its message hash will need to be added to the status cache when executed. 3) Epoch account indexes @@ -183,4 +201,5 @@ available to transactions in the following epoch. This approach has a downside of only updating the index at epoch boundaries which means there would be a few day delay before popular new accounts could be -referenced. +referenced. It also needs to be consistently generated by all validators by +using some criteria like adding accounts in order by access count. From b77af6bf43f18cc5cb2573f7b2fe960fc3eb481f Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Thu, 3 Jun 2021 09:40:37 -0700 Subject: [PATCH 05/19] Add cost changes section --- docs/src/proposals/page-indexed-transactions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index d15fc1edda4d53..c30206199bb505 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -113,6 +113,11 @@ pub struct AccountIndex { - Most indexes will be compact and use 2 bytes + index address - Cost of each additional index account is ~2 bytes +#### Cost changes + +Accessing an index account in a transaction should incur an extra cost due to +the extra work validators need to do to load and cache index accounts. + #### RPC changes The RPC API should support returning both compressed transactions for From d29d11bfa3be27f5fd8fb39f8e75e95a54159e16 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Sat, 5 Jun 2021 21:19:08 -0700 Subject: [PATCH 06/19] Add cost section and more attack details --- docs/src/proposals/page-indexed-transactions.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index c30206199bb505..341a4543667945 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -56,6 +56,11 @@ stored addresses, index accounts must also track the latest count of stored addresses and an authority which must be a present signer for all index modifications. +#### Cost + +Since index accounts require caching and special handling in the runtime, they should incur +higher costs for storage. Cost structure design is TBD. + #### Program controlled indexes If the authority of an index account is controlled by a program, more @@ -152,6 +157,13 @@ attacks could modify which index accounts are accessed from a later transaction. For this reason, we propose that any stored address is immutable and that index accounts themselves may not be removed. +Additionally, a malicious actor could try to fork the chain immediately after a +new index account is added to a block. If successful, they could add a different +unexpected index account in the fork. In order to deter this attack, clients +should wait for indexes to be finalized before using them in a transaction. +Clients may also append integrity check instructions to the transaction which +verify that the correct accounts are used. + ### Denial of service Index accounts will be read very frequently and will therefore be a more high From bdbc2c46ef063e8ad79bbc01edb2318c431f175f Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Sat, 5 Jun 2021 21:24:21 -0700 Subject: [PATCH 07/19] fix lint --- docs/src/proposals/page-indexed-transactions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index 341a4543667945..125e74da636af9 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -59,7 +59,7 @@ modifications. #### Cost Since index accounts require caching and special handling in the runtime, they should incur -higher costs for storage. Cost structure design is TBD. +higher costs for storage. Cost structure design will be added later. #### Program controlled indexes From 54d8fdf4867fceb2fa129d34ce2256437c9f8c3c Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Mon, 7 Jun 2021 14:41:26 -0700 Subject: [PATCH 08/19] document metadata changes --- docs/src/proposals/page-indexed-transactions.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index 125e74da636af9..7d5a3e3c855475 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -123,11 +123,18 @@ pub struct AccountIndex { Accessing an index account in a transaction should incur an extra cost due to the extra work validators need to do to load and cache index accounts. +#### Metadata changes + +Each account accessed via an index should be stored in the transaction metadata +for quick reference. This will avoid the need for clients to make multiple RPC +round trips to fetch all accounts referenced in a page-indexed transaction. It +will also make it easier to use the ledger tool to analyze account access +patterns. + #### RPC changes -The RPC API should support returning both compressed transactions for -verification as well as decompressed transactions which are easier for clients -to handle and abstract away the indexing details. +The RPC API should also support an option for returning fully decompressed +transactions to abstract away the indexing details from downstream clients. ### Limitations From ad42eb5cefd0dab023bdbb208c85f7a7b5258d46 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Mon, 7 Jun 2021 14:42:04 -0700 Subject: [PATCH 09/19] nit --- docs/src/proposals/page-indexed-transactions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index 7d5a3e3c855475..18dfe8cfe6e561 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -54,7 +54,7 @@ store up to 2^16 addresses each. Anyone may create an index account of any size as long as its big enough to store the necessary metadata. In addition to stored addresses, index accounts must also track the latest count of stored addresses and an authority which must be a present signer for all index -modifications. +additions. #### Cost From 50dd231cce2ca272d960459bf39846387dcb2afd Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Mon, 7 Jun 2021 14:50:12 -0700 Subject: [PATCH 10/19] rpc details --- docs/src/proposals/page-indexed-transactions.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index 18dfe8cfe6e561..8186eb79cf691d 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -133,6 +133,12 @@ patterns. #### RPC changes +Fetched transaction responses will likely require a new version field to +indicate to clients which transaction structure to use for deserialization. +Clients using pre-existing RPC methods will receive error responses when +attempting to fetch a versioned transaction which will indicate that they +must upgrade. + The RPC API should also support an option for returning fully decompressed transactions to abstract away the indexing details from downstream clients. From 03aaf4cbc92bbe61f327708ae7245c8b7687a499 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Mon, 7 Jun 2021 16:15:48 -0700 Subject: [PATCH 11/19] add index meta struct --- docs/src/proposals/page-indexed-transactions.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index 8186eb79cf691d..853db6b922af24 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -56,6 +56,22 @@ stored addresses, index accounts must also track the latest count of stored addresses and an authority which must be a present signer for all index additions. +Index additions require one slot to activate and so the index data should track +how many additions are still pending activation in on-chain data. + +```rust +struct IndexMeta { + // authority must sign for each addition + authority: Pubkey, + // incremented on each addition + len: u16, + // always set to the current slot + last_update_slot: Slot, + // incremented if current slot is equal to `last_update_slot` + last_update_slot_additions: u16, +} +``` + #### Cost Since index accounts require caching and special handling in the runtime, they should incur From 39e2cbdf626847ae220161a842f64124c1364408 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Mon, 7 Jun 2021 17:12:11 -0700 Subject: [PATCH 12/19] add additional proposal and chagne title --- .../src/proposals/page-indexed-transactions.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/page-indexed-transactions.md index 853db6b922af24..f6b0b545186eda 100644 --- a/docs/src/proposals/page-indexed-transactions.md +++ b/docs/src/proposals/page-indexed-transactions.md @@ -1,4 +1,4 @@ -# Page-Indexed Transactions +# Transactions v2 - Compressed account inputs ## Problem @@ -249,3 +249,19 @@ This approach has a downside of only updating the index at epoch boundaries which means there would be a few day delay before popular new accounts could be referenced. It also needs to be consistently generated by all validators by using some criteria like adding accounts in order by access count. + +4) Address lists + +Extend the transaction structure to support addresses that, when loaded, expand +to a list of addresses. After expansion, all account inputs are concatenated to +form a single list of account keys which can be indexed into by instructions. +Address lists would likely need to be immutable to prevent attacks. They would +also need to be limited in length to limit resource consumption. + +This proposal can be thought of a special case of the proposed index account +approach. Since the full account list would be expanded, there's no need to add +additional offsets that use up the limited space in a serialized +transaction. However, the expected size of an address list may need to be +encoded into the transaction to aid the sanitization of account indexes. +Additionally, special attention must be given to watch out for accounts that +exist in multiple account lists. From 70de74c28b3e50335842e3ee2eb8453175b1d653 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Mon, 7 Jun 2021 17:12:42 -0700 Subject: [PATCH 13/19] rename proposal file --- .../{page-indexed-transactions.md => transactions-v2.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/src/proposals/{page-indexed-transactions.md => transactions-v2.md} (100%) diff --git a/docs/src/proposals/page-indexed-transactions.md b/docs/src/proposals/transactions-v2.md similarity index 100% rename from docs/src/proposals/page-indexed-transactions.md rename to docs/src/proposals/transactions-v2.md From 8a858f6346611c5b684a9f76b183664e4fa29e4a Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Tue, 8 Jun 2021 19:56:16 -0700 Subject: [PATCH 14/19] rename to address map and rewrite tx format --- docs/src/proposals/transactions-v2.md | 209 ++++++++++++++------------ 1 file changed, 110 insertions(+), 99 deletions(-) diff --git a/docs/src/proposals/transactions-v2.md b/docs/src/proposals/transactions-v2.md index f6b0b545186eda..194298fb901f30 100644 --- a/docs/src/proposals/transactions-v2.md +++ b/docs/src/proposals/transactions-v2.md @@ -1,4 +1,4 @@ -# Transactions v2 - Compressed account inputs +# Transactions v2 - Address maps ## Problem @@ -24,75 +24,78 @@ after accounting for signatures and other transaction metadata. ## Proposed Solution -Introduce a new on-chain account indexing program which stores account address -mappings and add a new transaction format which supports concise account -references through on-chain account indexes. +Introduce a new on-chain program which stores account address maps and add a new +transaction format which supports concise account references through the +on-chain address maps. -### Account Indexing Program +### Address Map Program -Here we describe a contract-based solution to the problem, whereby a protocol -developer or end-user can create collections of related accounts on-chain for +Here we describe a program-based solution to the problem, whereby a protocol +developer or end-user can create collections of related addresses on-chain for concise use in a transaction's account inputs. This approach is similar to page tables used in operating systems to succinctly map virtual addresses to physical memory. -After addresses are stored on-chain in an index account, they may be succinctly -referenced from a transaction using an index rather than a full 32 byte address. -This will require a new transaction format to make use of these succinct indexes -as well as runtime handling for looking up and loading accounts from the -on-chain indexes. +After addresses are stored on-chain in an address map account, they may be +succinctly referenced in a transaction using a 2-byte u16 index rather than a +full 32-byte address. This will require a new transaction format to make use of +these succinct references as well as runtime handling for looking up and loading +accounts from the on-chain mappings. #### State -Index accounts must be rent-exempt and may not be deleted. Stored addresses -should be append only so that once an address is stored in an index account, it -may not be removed later. Index accounts must be pre-allocated since Solana does -not support reallocation for accounts yet. +Address map accounts must be rent-exempt and may not be deleted. Address maps +should be append only so that once an address is stored, it may not be removed +later. Address map accounts must be pre-allocated since Solana does not support +reallocation for accounts yet. -Since transactions use a u16 offset to look up addresses, index accounts can -store up to 2^16 addresses each. Anyone may create an index account of any size -as long as its big enough to store the necessary metadata. In addition to -stored addresses, index accounts must also track the latest count of stored -addresses and an authority which must be a present signer for all index -additions. +Since transactions use a u16 offset to look up addresses, accounts can store up +to 2^16 addresses each. Anyone may create an address map account of any size as +long as its big enough to store the necessary metadata. In addition to stored +addresses, address map accounts must also track the latest count of stored +addresses and an authority which must be a present signer for all appended map +entries. -Index additions require one slot to activate and so the index data should track -how many additions are still pending activation in on-chain data. +Map additions require one slot to activate so each map should track how many +addresses are still pending activation in their on-chain state. ```rust -struct IndexMeta { +struct AddressMap { // authority must sign for each addition authority: Pubkey, - // incremented on each addition - len: u16, // always set to the current slot last_update_slot: Slot, // incremented if current slot is equal to `last_update_slot` last_update_slot_additions: u16, + // incremented on each addition + len: u16, + // list of entries + entries: [Pubkey; MAP_CAPACITY], } ``` #### Cost -Since index accounts require caching and special handling in the runtime, they should incur -higher costs for storage. Cost structure design will be added later. +Since address map accounts require caching and special handling in the runtime, they +should incur higher costs for storage. Cost structure design will be added +later. -#### Program controlled indexes +#### Program controlled address maps -If the authority of an index account is controlled by a program, more -sophisticated indexes could be built with governance features or price curves -for new index addresses. +If the authority of an address map account is controlled by a program, more +sophisticated maps could be built with governance features or price curves for +new additions. ### Versioned Transactions In order to allow accounts to be referenced more succinctly, the structure of serialized transactions must be modified. The new transaction format should not affect transaction processing in the Solana VM beyond the increased capacity for -accounts and instruction invocations. Invoked programs will be unaware of which +accounts and program invocations. Invoked programs will be unaware of which transaction format was used. The new transaction format must be distinguished from the current transaction -format. Current transactions can fit at most 19 signatures (64-bytes each) but +format. Current transactions can fit at most 19 signatures (64-bytes each) but the message header encodes `num_required_signatures` as a `u8`. Since the upper bit of the `u8` will never be set for a valid transaction, we can enable it to denote whether a transaction should be decoded with the versioned format or not. @@ -101,51 +104,59 @@ denote whether a transaction should be decoded with the versioned format or not. ```rust pub struct VersionedMessage { - /// Version of encoded message. - /// The max encoded version is 2^7 - 1 due to the ignored upper disambiguation bit - pub version: u8, - pub header: MessageHeader, - /// Number of read-only account inputs specified thru indexes - pub num_readonly_indexed_accounts: u8, - #[serde(with = "short_vec")] - pub account_keys: Vec, - /// All the account indexes used by this transaction - #[serde(with = "short_vec")] - pub account_indexes: Vec, - pub recent_blockhash: Hash, - /// Compiled instructions stay the same, account indexes continue to be stored - /// as a u8 which means the max number of account_indexes + account_keys is 256. - #[serde(with = "short_vec")] - pub instructions: Vec, + /// Version of encoded message. + /// The max encoded version is 2^7 - 1 due to the ignored upper disambiguation bit + pub version: u8, + + // unchanged + pub header: MessageHeader, + + // unchanged + #[serde(with = "short_vec")] + pub account_keys: Vec, + + /// The last `address_mappings.len()` number of readonly unsigned account_keys + /// should be loaded as address maps + #[serde(with = "short_vec")] + pub address_mappings: Vec, + + // unchanged + pub recent_blockhash: Hash, + + // unchanged. Account indices are still `u8` encoded so the max number of accounts + // in account_keys + address_mappings is limited to 256. + #[serde(with = "short_vec")] + pub instructions: Vec, } -pub struct AccountIndex { - pub account_key_offset: u8, - // 1-3 bytes used to lookup address in index account - pub index_account_offset: CompactU16, +pub struct AddressMapping { + /// The last num_readonly of address_entries are read-only + pub num_readonly: u8, + + /// List of mapping entries to load + #[serde(with = "short_vec")] + pub entries: Vec, } ``` #### Size changes -- Extra byte for version field -- Extra byte for number of total account index inputs -- Extra byte for number of readonly account index inputs -- Most indexes will be compact and use 2 bytes + index address -- Cost of each additional index account is ~2 bytes +- 1 byte for `version` field +- 1 byte for `account_mappings` length +- Each mapping requires 2 bytes for `indices` length and `num_readonly` +- Each mapping entry is 2 bytes (u16) #### Cost changes -Accessing an index account in a transaction should incur an extra cost due to -the extra work validators need to do to load and cache index accounts. +Using an address map in a transaction should incur an extra cost due to +the extra work validators need to do to load and cache them. #### Metadata changes -Each account accessed via an index should be stored in the transaction metadata +Each account accessed via a mapping should be stored in the transaction metadata for quick reference. This will avoid the need for clients to make multiple RPC -round trips to fetch all accounts referenced in a page-indexed transaction. It -will also make it easier to use the ledger tool to analyze account access -patterns. +round trips to fetch all accounts referenced in a v2 transaction. It will also +make it easier to use the ledger tool to analyze account access patterns. #### RPC changes @@ -155,68 +166,68 @@ Clients using pre-existing RPC methods will receive error responses when attempting to fetch a versioned transaction which will indicate that they must upgrade. -The RPC API should also support an option for returning fully decompressed -transactions to abstract away the indexing details from downstream clients. +The RPC API should also support an option for returning fully expanded +transactions to abstract away the mapping details from downstream clients. ### Limitations - Max of 256 accounts may be specified in a transaction because u8 is used by compiled instructions to index into transaction message account keys. -Indexes can hold up to 2^16 keys. Smaller indexes is ok. Each index is then u16 -- Transaction signers may not be specified using an on-chain account index, the -full address of each signer must be serialized in the transaction. This ensures -that the performance of transaction signature checks is not affected. +- Address maps can hold up to 2^16 addresses because mapping entries are encoded as +`u16` in transactions. +- Transaction signers may not be specified using a mapping, the full address of +each signer must be serialized in the transaction. This ensures that the +performance of transaction signature checks is not affected. - Hardware wallets will probably not be able to display details about accounts -referenced with an index due to inability to verify on-chain data. -- Only single level indexes can be used. Recursive indexes will not be supported. +referenced through mappings due to inability to verify on-chain data. +- Only single level address maps can be used. Recursive maps will not be supported. ## Security Concerns ### Resource consumption Enabling more account inputs in a transaction allows for more program -invocations, write-locks, and data reads / writes. Before indexes are live, we -need transaction-wide compute limits and increased costs for write locks and -data reads. +invocations, write-locks, and data reads / writes. Before address maps are +enabled, transaction-wide compute limits and increased costs for write locks and +data reads are required. ### Front running -If the addresses listed within an index account are modifiable, front running -attacks could modify which index accounts are accessed from a later transaction. -For this reason, we propose that any stored address is immutable and that index -accounts themselves may not be removed. +If the addresses listed within an address map account are modifiable, front +running attacks could modify which mapped accounts are resolved for a later +transaction. For this reason, we propose that any stored address is immutable +and that mapping accounts themselves may not be removed. Additionally, a malicious actor could try to fork the chain immediately after a -new index account is added to a block. If successful, they could add a different -unexpected index account in the fork. In order to deter this attack, clients -should wait for indexes to be finalized before using them in a transaction. -Clients may also append integrity check instructions to the transaction which -verify that the correct accounts are used. +new address map account is added to a block. If successful, they could add a +different unexpected map entry in the fork. In order to deter this attack, +clients should wait for mappings to be finalized before using them in a +transaction. Clients may also append integrity check instructions to the +transaction which verify that the correct accounts are used. ### Denial of service -Index accounts will be read very frequently and will therefore be a more high -profile target for denial of service attacks through write locks similar to -sysvar accounts. +Address map accounts will be read very frequently and will therefore be a +more high profile target for denial of service attacks through write locks +similar to sysvar accounts. -Since stored accounts inside index accounts are immutable, reads and writes -to index accounts could be parallelized as long as all referenced addresses -are for indexes less than the current number of addresses stored. +Since addresses stored inside map accounts are immutable, reads and writes +can be parallelized as long as all referenced map entries are activated. ### Duplicate accounts -If the same account is referenced in a transaction by address as well as through -an index, the transaction should be rejected to avoid conflicts when determining -if the account is a signer or writeable. +Transactions may not load an account more than once whether directly through +`account_keys` or indirectly through `address_mappings`. ## Other Proposals 1) Account prefixes -Needing to pre-register accounts in an on-chain index is cumbersome because it -adds an extra step for transaction processing. Instead, Solana transactions -could use variable length address prefixes to specify accounts. These prefix -shortcuts can save on data usage without needing to setup on-chain state. +Needing to pre-register accounts in an on-chain address map is cumbersome +because it adds an extra step for transaction processing. Instead, Solana +transactions could use variable length address prefixes to specify accounts. +These prefix shortcuts can save on data usage without needing to setup on-chain +state. However, this model requires nodes to keep a mapping of prefixes to active account addresses. Attackers can create accounts with the same prefix as a popular account From f76b2434ec24ec3219407d10abba195fbafa97bf Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Tue, 8 Jun 2021 23:14:43 -0700 Subject: [PATCH 15/19] no more appends, limit mapping size to 256 --- docs/src/proposals/transactions-v2.md | 64 +++++++++++++-------------- 1 file changed, 31 insertions(+), 33 deletions(-) diff --git a/docs/src/proposals/transactions-v2.md b/docs/src/proposals/transactions-v2.md index 194298fb901f30..ea3695aa29a3c9 100644 --- a/docs/src/proposals/transactions-v2.md +++ b/docs/src/proposals/transactions-v2.md @@ -37,54 +37,52 @@ tables used in operating systems to succinctly map virtual addresses to physical memory. After addresses are stored on-chain in an address map account, they may be -succinctly referenced in a transaction using a 2-byte u16 index rather than a +succinctly referenced in a transaction using a 1-byte u8 index rather than a full 32-byte address. This will require a new transaction format to make use of these succinct references as well as runtime handling for looking up and loading accounts from the on-chain mappings. #### State -Address map accounts must be rent-exempt and may not be deleted. Address maps -should be append only so that once an address is stored, it may not be removed -later. Address map accounts must be pre-allocated since Solana does not support -reallocation for accounts yet. +Address map accounts must be rent-exempt but may be closed with a one epoch +deactivation period. Address maps must be activated before use. -Since transactions use a u16 offset to look up addresses, accounts can store up -to 2^16 addresses each. Anyone may create an address map account of any size as -long as its big enough to store the necessary metadata. In addition to stored -addresses, address map accounts must also track the latest count of stored -addresses and an authority which must be a present signer for all appended map -entries. +Since transactions use a u8 offset to look up mapped addresses, accounts can +store up to 2^8 addresses each. Anyone may create an address map account of any +size as long as its big enough to store the necessary metadata. In addition to +stored addresses, address map accounts must also track the latest count of +stored addresses and an authority which must be a present signer for all +appended map entries. Map additions require one slot to activate so each map should track how many -addresses are still pending activation in their on-chain state. +addresses are still pending activation in their on-chain state: ```rust struct AddressMap { - // authority must sign for each addition + // authority must sign for each addition and to close the map account authority: Pubkey, - // always set to the current slot - last_update_slot: Slot, - // incremented if current slot is equal to `last_update_slot` - last_update_slot_additions: u16, - // incremented on each addition - len: u16, - // list of entries - entries: [Pubkey; MAP_CAPACITY], + // entries may not be modified once activated + activated: bool, + // list of entries, max capacity of u8::MAX + entries: Vec, } ``` -#### Cost +#### Cleanup -Since address map accounts require caching and special handling in the runtime, they -should incur higher costs for storage. Cost structure design will be added -later. +Once an address map gets stale and is no longer used, it can be reclaimed by the +authority withdrawing lamports but the remaining balance must be greater than +two epochs of rent. This ensures that it takes at least one full epoch to +deactivate a map. -#### Program controlled address maps +Maps may not be recreated because each new map must be created at a derived +address using a monotonically increase counter seed. -If the authority of an address map account is controlled by a program, more -sophisticated maps could be built with governance features or price curves for -new additions. +#### Cost + +Since address map accounts require caching and special handling in the runtime, +they should incur higher costs for storage. Cost structure design will be added +later. ### Versioned Transactions @@ -135,7 +133,7 @@ pub struct AddressMapping { /// List of mapping entries to load #[serde(with = "short_vec")] - pub entries: Vec, + pub entries: Vec, } ``` @@ -144,7 +142,7 @@ pub struct AddressMapping { - 1 byte for `version` field - 1 byte for `account_mappings` length - Each mapping requires 2 bytes for `indices` length and `num_readonly` -- Each mapping entry is 2 bytes (u16) +- Each mapping entry is 1 byte (u8) #### Cost changes @@ -173,8 +171,8 @@ transactions to abstract away the mapping details from downstream clients. - Max of 256 accounts may be specified in a transaction because u8 is used by compiled instructions to index into transaction message account keys. -- Address maps can hold up to 2^16 addresses because mapping entries are encoded as -`u16` in transactions. +- Address maps can hold up to 256 addresses because mapping entries are encoded as +`u8` in transactions. - Transaction signers may not be specified using a mapping, the full address of each signer must be serialized in the transaction. This ensures that the performance of transaction signature checks is not affected. From d5db732d163f4e35e48a021d88d5fbbb0a929697 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Tue, 8 Jun 2021 23:34:57 -0700 Subject: [PATCH 16/19] update dos section --- docs/src/proposals/transactions-v2.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/src/proposals/transactions-v2.md b/docs/src/proposals/transactions-v2.md index ea3695aa29a3c9..d4743dc0a8b6cf 100644 --- a/docs/src/proposals/transactions-v2.md +++ b/docs/src/proposals/transactions-v2.md @@ -61,6 +61,9 @@ addresses are still pending activation in their on-chain state: struct AddressMap { // authority must sign for each addition and to close the map account authority: Pubkey, + // record a deactivation epoch to help validators know when to remove + // the mapping from their caches. + deactivation_epoch: Epoch, // entries may not be modified once activated activated: bool, // list of entries, max capacity of u8::MAX @@ -76,7 +79,7 @@ two epochs of rent. This ensures that it takes at least one full epoch to deactivate a map. Maps may not be recreated because each new map must be created at a derived -address using a monotonically increase counter seed. +address using a monotonically increasing counter as a derivation seed. #### Cost @@ -209,8 +212,8 @@ Address map accounts will be read very frequently and will therefore be a more high profile target for denial of service attacks through write locks similar to sysvar accounts. -Since addresses stored inside map accounts are immutable, reads and writes -can be parallelized as long as all referenced map entries are activated. +For this reason, special handling should be given to address map lookups. +Address maps lookups should not be affected by account read/write locks. ### Duplicate accounts From 31d96c0b290f86ac5f2e85d36a542c29be3c30a6 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Wed, 9 Jun 2021 10:10:00 -0700 Subject: [PATCH 17/19] add note about readonly --- docs/src/proposals/transactions-v2.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/src/proposals/transactions-v2.md b/docs/src/proposals/transactions-v2.md index d4743dc0a8b6cf..b841edb4155b00 100644 --- a/docs/src/proposals/transactions-v2.md +++ b/docs/src/proposals/transactions-v2.md @@ -272,8 +272,9 @@ also need to be limited in length to limit resource consumption. This proposal can be thought of a special case of the proposed index account approach. Since the full account list would be expanded, there's no need to add -additional offsets that use up the limited space in a serialized -transaction. However, the expected size of an address list may need to be -encoded into the transaction to aid the sanitization of account indexes. -Additionally, special attention must be given to watch out for accounts that -exist in multiple account lists. +additional offsets that use up the limited space in a serialized transaction. +However, the expected size of an address list may need to be encoded into the +transaction to aid the sanitization of account indexes. We would also need to +encode how many addresses in the list should be loaded as readonly vs +read-write. Lastly, special attention must be given to watch out for addresses +that exist in multiple account lists. From 1a2432d2cd1a558fdbe7f2d2ee03329d4241aef0 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Wed, 9 Jun 2021 17:55:22 -0700 Subject: [PATCH 18/19] restructure message to use enum --- docs/src/proposals/transactions-v2.md | 33 +++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/docs/src/proposals/transactions-v2.md b/docs/src/proposals/transactions-v2.md index b841edb4155b00..63d321768f7203 100644 --- a/docs/src/proposals/transactions-v2.md +++ b/docs/src/proposals/transactions-v2.md @@ -104,11 +104,35 @@ denote whether a transaction should be decoded with the versioned format or not. #### New Transaction Format ```rust +#[derive(Serialize, Deserialize)] +pub struct Transaction { + #[serde(with = "short_vec")] + pub signatures: Vec, + /// The message to sign. + pub message: Message, +} + +// Uses custom deserialization. If the first bit is not set, deserialize as the +// original `Message` format. Otherwise deserialize as a versioned message. +pub enum Message { + Unversioned(UnversionedMessage), + Versioned(VersionedMessage), +} + +#[derive(Serialize, Deserialize)] pub struct VersionedMessage { - /// Version of encoded message. - /// The max encoded version is 2^7 - 1 due to the ignored upper disambiguation bit - pub version: u8, + // only used for differentiating between message types, must be greater than 2^7 + prefix: u8, + pub message: MessageVersions, +} + +// use bincode varint encoding to use u8 as enum discriminant +#[derive(Serialize, Deserialize)] +pub enum MessageVersions { + Current(Box) +} +pub struct MessageV2 { // unchanged pub header: MessageHeader, @@ -142,7 +166,8 @@ pub struct AddressMapping { #### Size changes -- 1 byte for `version` field +- 1 byte for `prefix` field +- 1 byte for version enum discriminant - 1 byte for `account_mappings` length - Each mapping requires 2 bytes for `indices` length and `num_readonly` - Each mapping entry is 1 byte (u8) From abada96762ad5dd24edd9704a52527f71566b1c6 Mon Sep 17 00:00:00 2001 From: Justin Starry Date: Thu, 10 Jun 2021 12:56:34 -0700 Subject: [PATCH 19/19] cleanup --- docs/src/proposals/transactions-v2.md | 71 +++++++++++++-------------- 1 file changed, 34 insertions(+), 37 deletions(-) diff --git a/docs/src/proposals/transactions-v2.md b/docs/src/proposals/transactions-v2.md index 63d321768f7203..6e4dfa930a5d8e 100644 --- a/docs/src/proposals/transactions-v2.md +++ b/docs/src/proposals/transactions-v2.md @@ -62,7 +62,7 @@ struct AddressMap { // authority must sign for each addition and to close the map account authority: Pubkey, // record a deactivation epoch to help validators know when to remove - // the mapping from their caches. + // the map from their caches. deactivation_epoch: Epoch, // entries may not be modified once activated activated: bool, @@ -112,26 +112,21 @@ pub struct Transaction { pub message: Message, } -// Uses custom deserialization. If the first bit is not set, deserialize as the -// original `Message` format. Otherwise deserialize as a versioned message. +// Uses custom serialization. If the first bit is set, a versioned message is +// encoded starting from the next byte. If the first bit is not set, all bytes +// are used to encode the original unversioned `Message` format. pub enum Message { Unversioned(UnversionedMessage), Versioned(VersionedMessage), } +// use bincode varint encoding to use u8 instead of u32 for enum tags #[derive(Serialize, Deserialize)] -pub struct VersionedMessage { - // only used for differentiating between message types, must be greater than 2^7 - prefix: u8, - pub message: MessageVersions, -} - -// use bincode varint encoding to use u8 as enum discriminant -#[derive(Serialize, Deserialize)] -pub enum MessageVersions { +pub enum VersionedMessage { Current(Box) } +#[derive(Serialize, Deserialize)] pub struct MessageV2 { // unchanged pub header: MessageHeader, @@ -140,25 +135,26 @@ pub struct MessageV2 { #[serde(with = "short_vec")] pub account_keys: Vec, - /// The last `address_mappings.len()` number of readonly unsigned account_keys + /// The last `address_maps.len()` number of readonly unsigned account_keys /// should be loaded as address maps #[serde(with = "short_vec")] - pub address_mappings: Vec, + pub address_maps: Vec, // unchanged pub recent_blockhash: Hash, // unchanged. Account indices are still `u8` encoded so the max number of accounts - // in account_keys + address_mappings is limited to 256. + // in account_keys + address_maps is limited to 256. #[serde(with = "short_vec")] pub instructions: Vec, } -pub struct AddressMapping { - /// The last num_readonly of address_entries are read-only - pub num_readonly: u8, +#[derive(Serialize, Deserialize)] +pub struct AddressMap { + /// The last num_readonly_entries of entries are read-only + pub num_readonly_entries: u8, - /// List of mapping entries to load + /// List of map entries to load #[serde(with = "short_vec")] pub entries: Vec, } @@ -168,9 +164,9 @@ pub struct AddressMapping { - 1 byte for `prefix` field - 1 byte for version enum discriminant -- 1 byte for `account_mappings` length -- Each mapping requires 2 bytes for `indices` length and `num_readonly` -- Each mapping entry is 1 byte (u8) +- 1 byte for `address_maps` length +- Each map requires 2 bytes for `entries` length and `num_readonly` +- Each map entry is 1 byte (u8) #### Cost changes @@ -179,10 +175,11 @@ the extra work validators need to do to load and cache them. #### Metadata changes -Each account accessed via a mapping should be stored in the transaction metadata -for quick reference. This will avoid the need for clients to make multiple RPC -round trips to fetch all accounts referenced in a v2 transaction. It will also -make it easier to use the ledger tool to analyze account access patterns. +Each account accessed via an address map should be stored in the transaction +metadata for quick reference. This will avoid the need for clients to make +multiple RPC round trips to fetch all accounts referenced in a v2 transaction. +It will also make it easier to use the ledger tool to analyze account access +patterns. #### RPC changes @@ -193,19 +190,19 @@ attempting to fetch a versioned transaction which will indicate that they must upgrade. The RPC API should also support an option for returning fully expanded -transactions to abstract away the mapping details from downstream clients. +transactions to abstract away the address map details from downstream clients. ### Limitations - Max of 256 accounts may be specified in a transaction because u8 is used by compiled instructions to index into transaction message account keys. -- Address maps can hold up to 256 addresses because mapping entries are encoded as -`u8` in transactions. -- Transaction signers may not be specified using a mapping, the full address of -each signer must be serialized in the transaction. This ensures that the -performance of transaction signature checks is not affected. +- Address maps can hold up to 256 addresses because references to map entries +are encoded as `u8` in transactions. +- Transaction signers may not be referenced with an address map, the full +address of each signer must be serialized in the transaction. This ensures that +the performance of transaction signature checks is not affected. - Hardware wallets will probably not be able to display details about accounts -referenced through mappings due to inability to verify on-chain data. +referenced through address maps due to inability to verify on-chain data. - Only single level address maps can be used. Recursive maps will not be supported. ## Security Concerns @@ -221,13 +218,13 @@ data reads are required. If the addresses listed within an address map account are modifiable, front running attacks could modify which mapped accounts are resolved for a later -transaction. For this reason, we propose that any stored address is immutable -and that mapping accounts themselves may not be removed. +transaction. For this reason, we propose that any stored address is immutable +and that address map accounts themselves may not be recreated. Additionally, a malicious actor could try to fork the chain immediately after a new address map account is added to a block. If successful, they could add a different unexpected map entry in the fork. In order to deter this attack, -clients should wait for mappings to be finalized before using them in a +clients should wait for address maps to be finalized before using them in a transaction. Clients may also append integrity check instructions to the transaction which verify that the correct accounts are used. @@ -243,7 +240,7 @@ Address maps lookups should not be affected by account read/write locks. ### Duplicate accounts Transactions may not load an account more than once whether directly through -`account_keys` or indirectly through `address_mappings`. +`account_keys` or indirectly through `address_maps`. ## Other Proposals