Fix before/after cursors. (#67) #77
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
linuskendall
added a commit
that referenced
this pull request
Jul 4, 2023
* Update owner and delegate in asset table when collection or creator verification occurs.
* Modify program transformers to enable out-of-order transaction processing by adding the following changes:
* Upsert `leaf` info based on `seq` OR `was_decompressed` flag.
* Upsert `owner` and `delegate` based on `owner_delegate_seq`.
* Upsert "compression status" fields (`compressed`, `compressible`, `supply`, and `supply_mint`) based on `was_decompressed` flag.
* Add `owner_delegate_seq` and `was_decompressed` columns to asset table, and `PROGRAMMABLE_NFT` to `specification_asset_class` type.
* Regenerate Sea ORM objects.
* Needed to temporarily comment out a performance improvement migration that removed foreign keys so that SeaORM CLI would regenerate, since it builds relations based on those constraints.
* Needed to add a migration to add `PROGRAMMABLE_NFT` to `specification_asset_class` since it is needed by the API code but missing from the SQL code.
* Fix docker preparation script to build solana-program-library.
* Also remove unused 'filling' variable.
* Rustfmt/clippy fixes as I change files.
1. Made sure it builds and runs in Docker.
2. Used transaction forwarder to send out of order transactions and then observe asset table in database.
1. Ran a `CreateTree`/`Mint`, `Transfer`, and `Burn` and observed asset table. Ran it in reverse (`Burn`, `Transfer`, and then `CreateTree`/`Mint`) and observed database update properly in reverse order. Compared final state and they match. While running in reverse order I observed that could not get info for the asset via `getAsset` until `Mint` was indexed.
2. Ran a `CreateTree`/`Mint`, `Redeem`, `CancelRedeem`, second `Redeem`, and `Decompress` and observed asset table. Ran it in reverse (`Decompress`, second `Redeem`, `CancelRedeem`, first `Redeem`, and then `CreateTree`/`Mint`) and observed database update properly in reverse order. Compared final state and they match.
3. Ran a `CreateTree`/`Mint`, `Transfer`, and second `Transfer`, and observed asset table. Ran it in reverse (second `Transfer`, first `Transfer`, and then `CreateTree`/`Mint`) and observed database update properly in reverse order. Compared final state and they match.
3. Ran these same transactions in correct forward order on main branch and observed for each asset, final state of asset matched final state of asset when using my PR branch. Observed that running transactions in backwards order on main branch results in various incorrect data in asset table.
4. Ran these same transactions in correct forward order on main branch and observed state of cl_items table was the same as when using my PR branch. Observed that running transactions in backwards order on main branch results in various incorrect data in cl_items table.
linuskendall
added a commit
that referenced
this pull request
Jul 4, 2023
* Update owner and delegate in asset table when collection or creator verification occurs.
* Modify program transformers to enable out-of-order transaction processing by adding the following changes:
* Upsert `leaf` info based on `seq` OR `was_decompressed` flag.
* Upsert `owner` and `delegate` based on `owner_delegate_seq`.
* Upsert "compression status" fields (`compressed`, `compressible`, `supply`, and `supply_mint`) based on `was_decompressed` flag.
* Add `owner_delegate_seq` and `was_decompressed` columns to asset table, and `PROGRAMMABLE_NFT` to `specification_asset_class` type.
* Regenerate Sea ORM objects.
* Needed to temporarily comment out a performance improvement migration that removed foreign keys so that SeaORM CLI would regenerate, since it builds relations based on those constraints.
* Needed to add a migration to add `PROGRAMMABLE_NFT` to `specification_asset_class` since it is needed by the API code but missing from the SQL code.
* Fix docker preparation script to build solana-program-library.
* Also remove unused 'filling' variable.
* Rustfmt/clippy fixes as I change files.
1. Made sure it builds and runs in Docker.
2. Used transaction forwarder to send out of order transactions and then observe asset table in database.
1. Ran a `CreateTree`/`Mint`, `Transfer`, and `Burn` and observed asset table. Ran it in reverse (`Burn`, `Transfer`, and then `CreateTree`/`Mint`) and observed database update properly in reverse order. Compared final state and they match. While running in reverse order I observed that could not get info for the asset via `getAsset` until `Mint` was indexed.
2. Ran a `CreateTree`/`Mint`, `Redeem`, `CancelRedeem`, second `Redeem`, and `Decompress` and observed asset table. Ran it in reverse (`Decompress`, second `Redeem`, `CancelRedeem`, first `Redeem`, and then `CreateTree`/`Mint`) and observed database update properly in reverse order. Compared final state and they match.
3. Ran a `CreateTree`/`Mint`, `Transfer`, and second `Transfer`, and observed asset table. Ran it in reverse (second `Transfer`, first `Transfer`, and then `CreateTree`/`Mint`) and observed database update properly in reverse order. Compared final state and they match.
3. Ran these same transactions in correct forward order on main branch and observed for each asset, final state of asset matched final state of asset when using my PR branch. Observed that running transactions in backwards order on main branch results in various incorrect data in asset table.
4. Ran these same transactions in correct forward order on main branch and observed state of cl_items table was the same as when using my PR branch. Observed that running transactions in backwards order on main branch results in various incorrect data in cl_items table.
linuskendall
pushed a commit
that referenced
this pull request
Jul 26, 2023
linuskendall
pushed a commit
that referenced
this pull request
Jul 26, 2023
fanatid
pushed a commit
that referenced
this pull request
Jul 27, 2023
* Fix docker preparation script to build SPL * Update owner and delegate in asset table when collection or creator verification occurs * Modify program transformers to upsert in asset table * This allows out-of-order Bubblegum transactions to create and update the asset table. * Upsert leaf schema, owner, delegate, and seq separately since those are updated by all instructions and gated by sequence number to ensure freshest value. * Mint, burn, and decompress happen without regard to sequence number because they operate on unique fields. * Mint and burn have been updated but Decompress still needs to be fixed to handle out of order transactions. * Also remove unused 'filling' variable. * Update mint and decompress to be able to upsert asset info out of order * Add second sequence number for compression status fields * Reduce logging in docker * Comment out compressed_seq before regenerating Sea ORM objects * Add migration for asset specification * Update README * Rename PNFT and regenerate Sea ORM types * Apply usage of compressed_seq after regenerating Sea ORM types * Add owner delegate sequence number for owner and delegate fields. Also remove not null constraints for asset fields without defaults. * Regenerating database types * Update handling for non null constrained asset table * Update tests to use new Sea ORM types * Use owner_and_delegate_seq to separate upserts Also update redeem and decompress to not use leaf schema events. * Adding was_decompressed flag to replace compressed_seq compressed_seq won't work because decompression doesn't create a cl_event. * Regenerating Sea ORM types * Update code to use was_decompressed flag * Fix new boolean SQL conditions * Update comment * Remove column updates in asset table during mint for items not in model * Clippy fixes in ingester main * Cleanup debug comment * Allow for sequence number to be NULL (needed after decompress now) * Add leaf specific sequence number to protect that field in asset table * Revert "Allow for sequence number to be NULL (needed after decompress now)" This reverts commit 2713a18. * Update nft_ingester/src/program_transformers/bubblegum/redeem.rs Co-authored-by: Nicolas Pennie <[email protected]> --------- Co-authored-by: Nicolas Pennie <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix before/after cursors