Migrate from address maps to address lookup tables #21634

jstarry · 2021-12-06T14:50:24Z

Problem

The transaction v2 proposal has new proposed changes that rename "address maps" to "address lookup tables" and the design of address lookup tables has shifted.

Summary of Changes

Rename "address map indexes" to "address table lookups" and "mapped addresses" to "loaded addresses"
Address map account keys used to be appended to message.account_keys but address table lookups track the account key separately.

Fixes #

jstarry · 2021-12-08T04:03:07Z

@lijunwangs can you please advise at what the impact of changing these Postgres types and tables will be?

lijunwangs · 2021-12-08T16:50:31Z

sdk/program/src/message/versions/v0/mod.rs

+
+        // there should be at least 1 RW fee-payer account.
+        if self.header.num_readonly_signed_accounts >= self.header.num_required_signatures {
+            return Err(SanitizeError::IndexOutOfBounds);


I think probably SanitizeError::InvalidValue is more appropriate.

lijunwangs · 2021-12-08T16:54:02Z

sdk/program/src/message/versions/v0/mod.rs

+            let num_table_loaded_accounts = lookup
+                .writable_indexes
+                .len()
+                .saturating_add(lookup.readonly_indexes.len());


Do we need validations for writable_indexes?

The writable and readonly indexes are used to lookup addresses in an on-chain address table. We can't sanitize those here, the runtime will fail in another location if the lookup fails. The indexes can be as high as u8::MAX so there's no need to sanitize them

lijunwangs · 2021-12-08T16:59:18Z

sdk/program/src/message/versions/v0/mod.rs

+            }
+        }
+
+        Ok(())


Most of the errors returned in this function is IndexOutOfBounds, when one does run into such error -- it is actually difficult to figure out what causes it to fail. I guess some debug logging before the return will help.

This sanitization is done on the validator to filter out invalid transactions. It's not a method that users will interface with

Heh, I meant for the developer doing test. I found myself adding println! code in the error conditions to figure out what offended the sanitizer. I know this is not new code -- just copied from earlier location. And can be addressed separately.

lijunwangs · 2021-12-08T17:35:58Z

sdk/program/src/message/versions/v0/mod.rs

+#[serde(rename_all = "camelCase")]
+pub struct MessageAddressTableLookup {
+    /// Address lookup table account key
+    pub account_key: Pubkey,


How is the account_key used? Where do we actually store the table?

There are many such tables (users can create their own as they please) and they are all on-chain accounts so they are identified by this account key.

Got it! Thanks

lijunwangs · 2021-12-08T17:41:27Z

accountsdb-plugin-postgres/scripts/create_schema.sql

-CREATE TYPE "AddressMapIndexes" AS (
-    writable SMALLINT[],
-    readonly SMALLINT[]
+CREATE TYPE "TransactionMessageAddressTableLookup" AS (


Is the TransactionMessageAddressTable a global construct or stored in the transaction itself? If it is the former, we will have to persist that global table as well otherwise the indexes cannot be used.

Is the TransactionMessageAddressTable a global construct

It's stored on-chain so the full address table won't be included in the transaction object, but those addresses are resolved into "LoadedAddresses"

Cool. It should work then

jstarry · 2021-12-09T15:43:53Z

@lijunwangs can you please advise at what the impact of changing these Postgres types and tables will be?

lijunwangs · 2021-12-09T17:41:26Z

sdk/program/src/message/versions/v0/mod.rs

+            }
+        }
+
+        Ok(())


Heh, I meant for the developer doing test. I found myself adding println! code in the error conditions to figure out what offended the sanitizer. I know this is not new code -- just copied from earlier location. And can be addressed separately.

lijunwangs · 2021-12-09T17:43:23Z

sdk/program/src/message/versions/v0/mod.rs

+#[serde(rename_all = "camelCase")]
+pub struct MessageAddressTableLookup {
+    /// Address lookup table account key
+    pub account_key: Pubkey,


Got it! Thanks

lijunwangs · 2021-12-09T17:44:08Z

accountsdb-plugin-postgres/scripts/create_schema.sql

-CREATE TYPE "AddressMapIndexes" AS (
-    writable SMALLINT[],
-    readonly SMALLINT[]
+CREATE TYPE "TransactionMessageAddressTableLookup" AS (


Cool. It should work then

Pull request has been modified.

lijunwangs · 2021-12-09T19:45:04Z

sdk/program/src/message/versions/mod.rs

@@ -28,7 +28,7 @@ pub const MESSAGE_VERSION_PREFIX: u8 = 0x80;
 /// which message version is serialized starting from version `0`. If the first
 /// is bit is not set, all bytes are used to encode the legacy `Message`
 /// format.
-#[frozen_abi(digest = "x2F3RG2RhJQWN6L2N3jebvcAvNYFrhE3sKTPJ4sENvL")]
+#[frozen_abi(digest = "G4EAiqmGgBprgf5ePYemLJcoFfx4R7rhC1Weo2FVJ7fn")]


Just out of curiosity, how is the new digest generated?

It's generated using some procedural macros that build a type graph and hash the output. I just ran ./test-abi.sh locally to see what the mismatch was and updated the value.

codecov · 2021-12-09T21:41:16Z

Codecov Report

Merging #21634 (dec2065) into master (0224a8b) will decrease coverage by 0.0%.
The diff coverage is 89.0%.

@@            Coverage Diff            @@
##           master   #21634     +/-   ##
=========================================
- Coverage    81.6%    81.6%   -0.1%     
=========================================
  Files         511      511             
  Lines      143320   143304     -16     
=========================================
- Hits       116976   116962     -14     
+ Misses      26344    26342      -2

* Migrate from address maps to address lookup tables * update sanitize error * cargo fmt * update abi (cherry picked from commit 6c108c8)

* Migrate from address maps to address lookup tables * update sanitize error * cargo fmt * update abi (cherry picked from commit 6c108c8) Co-authored-by: Justin Starry <[email protected]>

jstarry added the v1.9 label Dec 6, 2021

jstarry force-pushed the migrate-address-maps branch from 04e54e3 to 5c96e59 Compare December 8, 2021 04:00

jstarry marked this pull request as ready for review December 8, 2021 04:02

jstarry requested a review from lijunwangs December 8, 2021 04:03

Migrate from address maps to address lookup tables

4e6c9d2

jstarry force-pushed the migrate-address-maps branch from 5c96e59 to 4e6c9d2 Compare December 8, 2021 04:04

lijunwangs reviewed Dec 8, 2021

View reviewed changes

jstarry added 2 commits December 9, 2021 10:42

update sanitize error

483c790

cargo fmt

a3c531c

lijunwangs previously approved these changes Dec 9, 2021

View reviewed changes

update abi

dec2065

lijunwangs reviewed Dec 9, 2021

View reviewed changes

jstarry merged commit 6c108c8 into solana-labs:master Dec 10, 2021

jstarry deleted the migrate-address-maps branch December 10, 2021 16:04

mergify bot pushed a commit that referenced this pull request Dec 10, 2021

Migrate from address maps to address lookup tables (#21634)

b83f4ae

* Migrate from address maps to address lookup tables * update sanitize error * cargo fmt * update abi (cherry picked from commit 6c108c8)

mergify bot mentioned this pull request Dec 10, 2021

Migrate from address maps to address lookup tables (backport #21634) #21773

Merged

brooksprumo mentioned this pull request Jan 5, 2022

v1.9/fix build #22311

Closed

jstarry mentioned this pull request Jul 4, 2022

Versioned Transaction and Address Lookup Table feature activation #26391

Closed

65 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from address maps to address lookup tables #21634

Migrate from address maps to address lookup tables #21634

jstarry commented Dec 6, 2021 •

edited

Loading

jstarry commented Dec 8, 2021

lijunwangs Dec 8, 2021

lijunwangs Dec 8, 2021

jstarry Dec 9, 2021

lijunwangs Dec 8, 2021

jstarry Dec 9, 2021

lijunwangs Dec 9, 2021

lijunwangs Dec 8, 2021

jstarry Dec 9, 2021

lijunwangs Dec 9, 2021

lijunwangs Dec 8, 2021

jstarry Dec 9, 2021

lijunwangs Dec 9, 2021

jstarry commented Dec 9, 2021

lijunwangs Dec 9, 2021

lijunwangs Dec 9, 2021

lijunwangs Dec 9, 2021

lijunwangs Dec 9, 2021

jstarry Dec 9, 2021

codecov bot commented Dec 9, 2021

+                          }
+                      }
+                      Ok(())

Migrate from address maps to address lookup tables #21634

Migrate from address maps to address lookup tables #21634

Conversation

jstarry commented Dec 6, 2021 • edited Loading

Problem

Summary of Changes

jstarry commented Dec 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jstarry commented Dec 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 9, 2021

Codecov Report

jstarry commented Dec 6, 2021 •

edited

Loading