Enable indexing Merkle trees on other SVMs

[danenbm]

Notes

• This PR enables indexing mpl-bubblegum Merkle trees created on other SVMs, where spl-account-compression and spl-noop have not been deployed.
• The mpl forks do not change how the transactions are broadcast, so this indexing update simply allows parsing from either spl-noop or mpl-noop, with no changes to the structure other than using types associated with the forked programs.
  • The mpl forks can be found here: https://github.com/metaplex-foundation/mpl-account-compression
  • Note these are not going to Solana mainnet at this time, but are deployed to Solana devnet for testing and will be deployed to Eclipse.
• The Bubblegum version for Eclipse is on this svm branch: https://github.com/metaplex-foundation/mpl-bubblegum/tree/svm

Other changes

• Update DAS to use Solana 1.18.
• Update Docker build to download mpl-account-compression, mpl-noop, and mpl-bubblegum from devnet.
• Suppress clippy warning on use of deprecated Borsh 0.10 deserialization function.

Testing

• Works locally with new mpl programs.
• Added an integration test that creates a tree, mints an NFT, and transfers it using the new mpl programs.
• Regression testing: all existing integration tests pass.

Test results

running 48 tests
test cnft_tests::test_mint_redeem_cancel_redeem ... ok
test account_update_tests::test_account_updates ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_pubkey_in_rule_set ... ok
test cnft_tests::test_mint_verify_collection ... ok
test mpl_core_tests::test_mpl_core_get_assets_by_authority ... ok
test cnft_tests::test_mint_delegate_transfer ... ok
test mpl_core_tests::test_mpl_core_get_assets_by_group_with_oracle_and_custom_pda_all_seeds ... ok
test mpl_core_tests::test_mpl_core_get_assets_by_owner ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_two_oracle_external_plugins ... ok
test general_scenario_tests::test_creators_reordering ... ok
test mpl_core_tests::test_mpl_core_get_collection_with_linked_app_data_with_json_data_and_owner_is_data_authority ... ok
test mpl_core_tests::test_mpl_core_get_collection_with_linked_app_data_with_msg_pack_data_and_update_authority_is_data_authority ... ok
test mpl_core_tests::test_mpl_core_verified_creators_plugin ... ok
test mpl_core_tests::test_mpl_core_verified_creators_plugin_with_signature ... ok
test regular_nft_tests::test_reg_get_asset ... ok
test mpl_core_tests::test_mpl_core_get_assets_by_group ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_data_section_with_msg_pack_data ... ok
test regular_nft_tests::test_reg_search_assets ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_app_data_with_msg_pack_data_and_address_is_data_authority ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_oracle_multiple_lifecycle_events ... ok
test mpl_core_tests::test_mpl_core_autograph_plugin ... ok
test cnft_tests::test_mint_transfer_mpl_programs ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_multiple_internal_and_external_plugins ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_app_data_with_json_data_and_update_authority_is_data_authority ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_data_section_with_json_data ... ok
test regular_nft_tests::test_reg_get_asset_by_group ... ok
test mpl_core_tests::test_mpl_core_get_collection ... ok
test cnft_tests::test_mint_verify_creator ... ok
test regular_nft_tests::test_reg_get_asset_batch ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_oracle_custom_offset_and_base_address_config ... ok
test cnft_tests::test_mint_redeem ... ok
test cnft_tests::test_cnft_scenario_mint_update_metadata ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_data_section_with_binary_data ... ok
test cnft_tests::test_mint_transfer_transfer ... ok
test mpl_core_tests::test_mpl_core_get_asset ... ok
test cnft_tests::test_cnft_scenario_mint_update_metadata_remove_creators ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_app_data_with_binary_data_and_owner_is_data_authority ... ok
test cnft_tests::test_asset_decompress ... ok
test cnft_tests::test_mint_transfer_noop ... ok
test cnft_tests::test_mint_transfer_burn ... ok
test general_scenario_tests::test_asset_parsing ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_oracle_external_plugin_on_collection ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_edition ... ok
test mpl_core_tests::test_mpl_core_autograph_plugin_with_signature ... ok
test cnft_tests::test_cnft_owners_table has been running for over 60 seconds
test cnft_tests::test_mint_no_json_uri has been running for over 60 seconds
test mpl_core_tests::test_mpl_core_get_asset_with_oracle_no_offset has been running for over 60 seconds
test cnft_tests::test_mint_no_json_uri ... ok
test mpl_core_tests::test_mpl_core_get_asset_with_oracle_no_offset ... ok
test cnft_tests::test_cnft_owners_table ... ok
test mpl_core_tests::test_mpl_core_get_collection_with_linked_app_data_with_binary_data_and_address_is_data_authority ... ok

test result: ok. 48 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 63.03s

[kespinola]

This is going to get harry fast through. Other SVM based networks are going to want to deploy metaplex and if we have to adjust the code each time to support its going stifle adoption.

I think we need to look at this more at the validator level and having adjustments made for ALT-SVM to have overrides that they can manage for deploying and upgrading programs from mainnet without having to change the address every time.

[danenbm]

This is going to get harry fast through. Other SVM based networks are going to want to deploy metaplex and if we have to adjust the code each time to support its going stifle adoption.

I think we need to look at this more at the validator level and having adjustments made for ALT-SVM to have overrides that they can manage for deploying and upgrading programs from mainnet without having to change the address every time.

If Metaplex is deploying Bubblegum on another SVM, we would deploy mpl-account-compression and mpl-noop as well, and to the same addresses that we plan to use for Eclipse. I agree this would be a headache if we had to keep using new addresses for other SVM networks. But I don't see why that would be the case after this deployment.

Also, if SVM teams can deploy and upgrade programs without having to change the address, it means they can deploy without needing the private key for program address. Wouldn't that break trust assumptions of what the SVM provides in terms of account security and decentralization?

I think immutable SPL programs like spl-token might be able to be considered a required base layer, but for mutable programs (SPL or otherwise), someone has to be on the hook for security and maintenance in general, and it should be the team that owns the program. But they should not be expected to maintain a program on an SVM they didn't agree to support.

[kespinola]

Is it necessary to make this change now, given that the noop program is referenced during transaction execution? As long as the instruction data allows, SVM chains could deploy the noop program once, and as long as it’s not upgraded, it should remain usable.

In the future, if an upgrade to the program becomes necessary and mpl-noop is in use, the transaction program address would update, allowing us to handle both scenarios as needed on a per-transaction basis.

For now, I’d like to test if spl-noop can work as it is on mainnet, deployed under a single address without access to update authority. If the goal is to establish flows for protocol teams to deploy to multiple networks under a consistent address, then Metaplex could manage bubblegum while Solana Labs manages spl-noop.

[kespinola]

Also, if SVM teams can deploy and upgrade programs without having to change the address, it means they can deploy without needing the private key for program address. Wouldn't that break trust assumptions of what the SVM provides in terms of account security and decentralization?

It depends if the network is running under an authority for say SPE or closed network alt-svm where a mod to upgrade program is deployed to only let a specific address deploy.

However, overall agree that a public key should represent a private key and if they are able to be overriden then trust assumption start to fall apart.

As I said in my previous comment I think its more on improving program deployment workflows to accommodate launching to multiple networks.

[blockiosaurus]

I think it would be pretty onerous to have to ask Solana Labs to deploy Noop on every SVM we would have to deploy to. They don't have a strong incentive to support other SVMs. In general they have gone the direction of self-CPI for all logging, and spl account compression doesn't see much non-bubblegum use so I think it makes sense to consolidate these under the same authority umbrella.

[danenbm]

Is it necessary to make this change now, given that the noop program is referenced during transaction execution? As long as the instruction data allows, SVM chains could deploy the noop program once, and as long as it’s not upgraded, it should remain usable.

In the future, if an upgrade to the program becomes necessary and mpl-noop is in use, the transaction program address would update, allowing us to handle both scenarios as needed on a per-transaction basis.

For now, I’d like to test if spl-noop can work as it is on mainnet, deployed under a single address without access to update authority. If the goal is to establish flows for protocol teams to deploy to multiple networks under a consistent address, then Metaplex could manage bubblegum while Solana Labs manages spl-noop.

We also need spl-account-compression for Bubblegum to work, and that is the one that is more worrisome than spl-noop, as we do expect that to need periodic updates including potential security hotfixes. If an upgrade to either noop or account-compression is needed, and at that point we pivot to the MPL programs, we would also still need to fix the SPL versions for compatibility with existing trees.

Solana has firmly said to us they do not want to deploy and manage spl-noop and spl-account-compression on other SVMs, and they recommended we fork the programs. And Metaplex does want be able to deploy uniform updates across SVMs, maintaining ownership for the quality and security of Bubblegum everywhere it is deployed.