fix: remove decompress ix handling and use db transactions #156

NicolasPennie · 2024-01-11T07:47:14Z

Overview

Remove decompress ix handling for cNFTs. It's redundant. Instead we should fully rely on the indexing of the regular NFT via accounts (new source of truth)
Fix out-of-order issue between txns and accounts. When indexing the metadata account, we allow the update for an asset if slot_updated is null. This happens when there is already a corresponding row in the asset table from cNFT events.
Properly wipe the data/creator hash to enforce consistency for decompressed NFTs.
Group all DB operations for a single cNFT ix within one DB transaction. Cleaner to read / maintain, but also marginally faster on the DB and enforces consistent reads for customers. Without a DB transaction a customer might see a newly minted cNFT without a collection.

Testing

Running in Helius prod

nft_ingester/src/program_transformers/bubblegum/mint_v1.rs

nft_ingester/src/program_transformers/bubblegum/update_metadata.rs

danenbm · 2024-01-11T08:17:58Z

nft_ingester/src/program_transformers/token_metadata/v1_asset.rs

                    asset::Column::Compressed,
                    asset::Column::Compressible,
+                    asset::Column::DataHash,
+                    asset::Column::CreatorHash,


Good catch on missed columns.

Nitpick: I would put these in same order as they are in the model above (hashes first then compressed/compressible) just for consistency.

danenbm · 2024-01-11T08:23:57Z

nft_ingester/src/program_transformers/token_metadata/v1_asset.rs

@@ -234,7 +236,7 @@ pub async fn save_v1_asset<T: ConnectionTrait + TransactionTrait>(
        )
        .build(DbBackend::Postgres);
    query.sql = format!(
-        "{} WHERE excluded.slot_updated > asset.slot_updated",
+        "{} WHERE excluded.slot_updated > asset.slot_updated OR asset.slot_updated IS NULL",


Good catch here too!

* Update raw name and raw symbol for existing NFTs (#139) Update the raw name and raw symbol from the onchain data for existing NFTs when reingesting. This allows correction of incorrect values during reprocessing on an existing index. * Upstream Helius features (#133) * Bubblegum Update Metadata Version 1 (#134) * Add code to index Bubblegum Update Metadata * Update rust toolchain file * Fix moved variable after merge * Add code from mintV1 that allows for empty URI * Ordering using asset.seq initially applied to update_metadata * Add simple check for whether asset was decompressed to Bubblegum transformers * Don't prevent sequence number update when already decompressed * Add sequence number to downloading metadata background task * Add sequence number migration (Sea ORM not regenerated yet) * Regenerate Sea-ORM types * Use new sequence numbers for Bubblegum Update Metadata * Extra condition to protect out of order creator verification * Remove base_info_seq for each creator and add creators_added_seq to asset table * Regenerate Sea-ORM types * Change creator metadata updates to use new creators_added_seq * Factor out common creator update code to helper function * Update to latest blockbuster beta * Use less than or equal for download metadata seq check * Index verified for token metadata collection * Add slot_updated to initial asset upsert, and removed duplicate items * Remove asset_was_decompressed Replaced with WHERE clauses on each upsert. Move remaining upserts from mint_v1 to db.rs. Remove upsert to asset_v1_account_attachments from mint_V1. Combine upserts for asset base info and royalty amount. * Rename royalty_amount_seq to base_info_seq * Fix typo in WHERE clause * Do not delete existing creators in mint_v1 * Update comments around database txns * Use transaction in mint_V1 and update_metadata * Use transaction for other Bubblegum instructions asset table updates * Fix tree_id key index in update_metadata * Remove use of was_decompressed flag on asset table * Add migration to remove was_decompressed and regenerate SeaORM types * Combine upsert_asset_base_info and upsert_creators and add lock * Remove unneeded creators_added_seq * Switch to EXCLUSIVE mode lock * Add NULL condition check on asset.seq * Refactored creator indexing * Use new Blockbuster that always updates all creators and verification status. * Remove deleting creators with lower sequence numbers as it would not work due to race conditions. * Add concept of "empty" creator value to support Bubblegum empty creator arrays. * Add filtering out of old creators or having no creators to DAS code. * Also get authority and tree_id accounts from Bubblegum during mint and update_metadata. * Add conditions to creator upsert, add another check at DAS API level * Rename asset_creators.verified_seq back to just regular seq * Remove unneeded condition on asset_authority upsert * Apply stale creator filtering to all DAS API queries * Use latest blockbuster beta release * Remove download_metadata_seq and add URI match check instead * Fix task URI initial query * Regenerate Sea ORM types without download_metadata_seq * asset_grouping.verified option remove * Fix filtering for getAssetsByCreator * Update to blockbuster 0.9.0-beta.5 and mpl-bubblegum 1.0.1-beta.4 * Configurable account streams (#148) * Make workers configurable Make workers fully configurab le and remove reference to the plerkle plugin. * fix lifetime --------- Co-authored-by: Kirill Fomichev <[email protected]> * fix: splt tokens with no token stanard are incorrectly categorized as single. Logic in token account updates would change owner when any token account had amount > 0 would triggers spam updates of the owner of the asset with any transfer. (#151) * Improve workspace usage (#141) * fix: remove decompress ix handling and use db transactions (#156) * Improvements for consistency NFT indexing and query consistency (#157) * Danenbm/bubblegum sequence tests 2 (#160) * Add script to forward transactions an check database results * Fix ordering and add debug info * Add remaining non-creator/non-collection tests * Require asset and cl_items files to exist * Add asset_creators and asset_grouping tests * Add verify_creator and verify_collection tests * Add more collection verification tests * Move test data to subirectory * Move repeated code to functions * Add support for running sequences in reverse * Add instructions to README for running test script * Minor README update * feat: `getSignaturesForAsset` endpoint on top of new `cl_audits_v2` table (#155) * add migration files for cl_audits_v2 * add types * ingester * add getSignaturesForAsset endpoint * refactor to resolve merge conflict related bugs * address clippy error * rename to get_asset_signatures * add instruction type update_metadata * add error log if instruction is unknown * add sort order changes --------- Co-authored-by: Nicolas Pennie <[email protected]> * fix ORM circular dependency (#161) * fix ORM circular dependency * PR comments --------- Co-authored-by: Linus Kendall <[email protected]> Co-authored-by: Tahsin Tunan <[email protected]> Co-authored-by: Michael Danenberg <[email protected]> Co-authored-by: Kirill Fomichev <[email protected]> Co-authored-by: Nicolas Pennie <[email protected]>

fix: remove decompress ix handling and use db transactions

7b5d22e

danenbm requested changes Jan 11, 2024

View reviewed changes

danenbm reviewed Jan 11, 2024

View reviewed changes

adjust for pr comments

8505332

danenbm approved these changes Jan 11, 2024

View reviewed changes

NicolasPennie merged commit 6afeeb5 into metaplex-foundation:main Jan 11, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: remove decompress ix handling and use db transactions #156

fix: remove decompress ix handling and use db transactions #156

NicolasPennie commented Jan 11, 2024

danenbm Jan 11, 2024

danenbm Jan 11, 2024

fix: remove decompress ix handling and use db transactions #156

fix: remove decompress ix handling and use db transactions #156

Conversation

NicolasPennie commented Jan 11, 2024

Overview

Testing

danenbm Jan 11, 2024

Choose a reason for hiding this comment

danenbm Jan 11, 2024

Choose a reason for hiding this comment