Add RPC support for versioned transactions

[jstarry]

Problem

• Clients have no way to tell RPC which transaction versions they support receiving
• RPC doesn't support sending transactions which use address lookup tables

Summary of Changes

• Added max_supported_transaction_version request parameter to enable receiving versioned transactions in RPC responses and pubsub notifications. This ensures that older clients will get an "unsupported version error" instead of a response for a transaction version they can't parse.
• Added RPC support for sending and simulating transactions which load addresses from on-chain lookup tables
• Added version field to transaction responses when the max_supported_transaction_version is used whose value is "legacy" string for legacy transactions and a number (ie 0) for versioned transactions.
• Added loaded_addresses field to transaction metadata responses for transactions with version 0
• Added address_table_lookups field to transaction message responses for transactions with version 0

Fixes #

[CriesofCarrots]

Sorry, I might be being dense, but I'm not sure I follow why this is necessary for either getBlock and getTransaction? We generally consider adding fields to rpc responses as non-breaking, and it looks to me like all the new rpc fields are additive (loadedAddresses and version for blocks, addressTableLookups for transactions).

[CriesofCarrots]

Is it that you think it's important for clients to receive an error rather than a v0 block/transaction that is just missing those fields? If so, I don't think I agree. Change my mind?

[jstarry]

My intent here is that we can totally diverge from keeping backwards compatible fields for future transaction versions if we want.

[jstarry]

Is it that you think it's important for clients to receive an error rather than a v0 block/transaction that is just missing those fields? If so, I don't think I agree. Change my mind?

And yes, I also think this is important. Clients might be parsing transactions for certain account keys and if they don't explicitly support the loaded addresses field for v0 transactions, they can totally miss important certain accounts

[CriesofCarrots]

Hmm, I still don't really love that it forces all clients to upgrade (unexpectedly) when the first v0 transaction comes down the pipe. But to confirm, this won't affect clients querying getBlock with transactionDetails: none|signatures, right?

[CriesofCarrots]

That said, I will start giving this a proper review now :)

[CriesofCarrots]

What do you envision the client upgrade / feature-activation process looking like? Basically every exchange will be affected.

[jstarry]

Hmm, I still don't really love that it forces all clients to upgrade (unexpectedly) when the first v0 transaction comes down the pipe

Yeah, I don't love the forced upgrade either but I do feel like this approach opens the door for us to start iterating on the transaction format. We will likely be changing other things about the format in the future and it's inevitable that clients will need to be upgraded to support new features.

What do you envision the client upgrade / feature-activation process looking like? Basically every exchange will be affected.

The rollout process should look like this:

1. Clients are updated with support for handling versioned transactions
2. RPC servers and runtime are updated to handle versioned transactions
3. Once clients and validators are updated, the feature can be activated
4. Users start processing these new versioned transactions
5. Old clients will start getting errors when they request blocks / transactions which they don't opt in to receiving
6. Users complain and realize they need to update their clients

But to confirm, this won't affect clients querying getBlock with transactionDetails: none|signatures, right?

Correct

[CriesofCarrots]

I didn't catch any issues with implementation, just maybe need a little more persuading on the approach ;) One test case would be nice to add.
I also want to sanity-test Blockstore-Bigtable writes and reads -- what's the easiest way to generate and submit a v0 transaction?

[CriesofCarrots]

I know it's a bit of a pita to construct, but would be nice to have a test case demonstrating unsupported tx error for getBlock to make the effect of one v0 tx clear.

[jstarry]

I rebased off master to get the rpc test refactorings and added a new test test_get_block_with_versioned_tx

[jstarry]

I also want to sanity-test Blockstore-Bigtable writes and reads -- what's the easiest way to generate and submit a v0 transaction?

This test is a good starting point:

solana/core/src/banking_stage.rs

Line 3242 in c69e3b7

fn test_write_persist_transaction_status() {

[CriesofCarrots]

Okay, thank you for your patience with me. Blockstore/bigtable flow all works, and I guess I am on board with the approach. It feels okay in practice, and I'm stuck coming up with a better one.

Just the question about the AddressLoader bank commitment on this PR

Oh also, incidentally: I hacked together solana-client methods that will submit a VersionedTransaction, and wrote a little program that creates an address-lookup account and submits a v0 transaction using that account. Can share, if it would be helpful.

[jstarry]

I hacked together solana-client methods that will submit a VersionedTransaction, and wrote a little program that creates an address-lookup account and submits a v0 transaction using that account. Can share, if it would be helpful.

Yes, that would be great, thanks!

[CriesofCarrots]

Yes, that would be great, thanks!

Here's the solana(-client) branch: https://github.com/CriesofCarrots/solana/tree/tx-v0-client
And gist: https://gist.github.com/CriesofCarrots/b8f29aba2b960956cc2482ddf73ed385