Danenbm/bubblegum sequence tests 2

README.md

@@ -96,17 +96,16 @@ Developing with Docker is much easier, but has some nuances to it. This test doc
 * Token 2022
 * Latest version of the Associated token program
 
-You need to run the following script (which takes a long time) in order to get all those .so files.
+You need to run the following script in order to get the .so files.
 
 ```bash
-chmod +x ./prepare-local-docker-env.sh
 ./prepare-local-docker-env.sh
 ```
-This script grabs all the code for these programs and compiles it, and chucks it into your programs folder. Go grab some coffe because this will take a while/
-If you get some permissions errors, just sudo delete the programs directory and start again.
+This script downloads these programs from mainnet and puts them in the `programs/` folder.
 
 #### Authentication with Docker and AWS
 
+_This step is not normally needed for basic local docker usage._
 ```aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin {your aws container registry}```
 
 #### Running the application
@@ -129,6 +128,48 @@ Also when mucking about with the docker file if your gut tells you that somethin
 
 Sometimes you will want to delete the db do so with `sudo rm -rf db-data`.  You can also delete the ledger with `sudo rm -rf ledger`.
 
+#### Running Bubblegum Test Sequences
+
+While running the multi-container Docker application locally, you can run a script located in `tools/txn_forwarder/bubblegum_tests` that will send sequences of bubblegum transactions via the `txn_forwarder`, and then use `psql` to read and verify the indexing results in the local Postgres database.
+
+```bash
+sudo rm -rf db-data/
+sudo rm -rf ledger/
+docker compose up --force-recreate --build
+```
+_In another terminal:_
+```bash
+cd tools/txn_forwarder/bubblegum_tests/
+./run-bubblegum-sequences.sh
+```
+
+You should see it log something like:
+```
+Running 10 scenarios forwards
+mint_transfer_burn.scenario initial asset table state passed
+mint_transfer_burn.scenario initial asset_creators table state passed
+mint_transfer_burn.scenario initial asset_grouping table state passed
+mint_transfer_burn.scenario initial cl_items table state passed
+...
+mint_to_collection_unverify_collection.scenario asset table passed
+mint_to_collection_unverify_collection.scenario asset_creators table passed
+mint_to_collection_unverify_collection.scenario asset_grouping table passed
+mint_to_collection_unverify_collection.scenario cl_items table passed
+
+ALL TESTS PASSED FORWARDS!
+```
+
+You can also run the sequences in reverse:
+```bash
+./run-bubblegum-sequences.sh reverse
+```
+And after it runs you should see `ALL TESTS PASSED IN REVERSE!`
+
+A few detailed notes about this test script:
+* This script is not all-encompassing.  It is only meant to automate some normal basic tests that were previously done manually.  The reason this test is not added to CI is because requires a more powerful system to run the Docker application, which contains the no-vote Solana validator.
+* The test sequences are in `.scenario` files, but instead of sending those files to the `txn_forwarder` directly (which supports the file format), we parse them out and send them individually using the `single` parameter.  This is because using the `.scenario` file directly results in random ordering of the transactions and we are explicity trying to test them going forwards and in reverse.
+* In general the expected database results are the same when running the transactions forwards and backwards.  However, for assets that are decompressed, this is not true because we don't index some of the asset information from Bubblegum mint indexing if we already know the asset has been decompressed.  We instead let Token Metadata account based indexing fill in that information.  This is not reflected by this test script so the results differ when running these sequences in reverse.  The differing results are reflected in test files with the `_reverse` suffix.
+
 #### Logs
 To get a reasonable amount of logs while running Docker, direct grafana logs to a file:
 ```
@@ -138,7 +179,7 @@ grafana:
       ...
       - GF_LOG_MODE=file
 ```
-and set Solana Rust logs to error level:
+and set Solana Rust logs to error level (it is already set to error level now in the current docker compose file):
 ```
   solana:
     ...

tools/txn_forwarder/bubblegum_tests/run-bubblegum-sequences.sh

@@ -0,0 +1,179 @@
+#!/bin/bash
+
+# Pass `reverse` to run scenarios in reverse.
+if [ "$1" = "reverse" ]; then
+    REVERSE="true"
+else
+    REVERSE="false"
+fi
+
+SCENARIOS=("mint_transfer_burn.scenario" \
+"mint_redeem_decompress.scenario"
+"mint_redeem_cancel_redeem_redeem_decompress.scenario" \
+"mint_transfer_transfer.scenario" \
+"mint_delegate_transfer.scenario" \
+"mint_verify_creator.scenario" \
+"mint_verify_collection.scenario" \
+"mint_verify_collection_unverify_collection.scenario" \
+"mint_set_and_verify_collection.scenario" \
+"mint_to_collection_unverify_collection.scenario"
+)
+
+TEST_SCENARIO_DATA_DIR="test_scenario_data"
+
+# Output text in colors.
+# $1 is output text.
+RED() { echo $'\e[1;31m'$1$'\e[0m'; }
+GRN() { echo $'\e[1;32m'$1$'\e[0m'; }
+
+# Read from database using psql and compare to expected value.
+# $1 is SQL command.
+# $2 extra CLI args to send to psql.
+# $3 is expected value.
+# $4 is topic for the pass/fail message.
+# Returns 0 if database value matches expected value, otherwise returns 1.
+CHECK_DATABASE() {
+    local DATABASE_VAL=$(PGPASSWORD=solana psql -h localhost -U solana "$2" --command="$1")
+    # Remove `created_at` since the date changes for `asset` table entries.
+    local DATABASE_VAL=$(sed '/^created_at/d' <<< "$DATABASE_VAL")
+    if [ "$3" = "$DATABASE_VAL" ]; then
+        echo $(GRN "${SCENARIOS[$i]} $4 passed") >&2
+        return 0
+    else
+        echo $(RED "${SCENARIOS[$i]} $4 failed") >&2
+        echo "Asset ID: $ASSET_ID" >&2
+        echo "Expected:" >&2
+        echo "$3" >&2
+        echo ""
+        echo "Actual:" >&2
+        echo "$DATABASE_VAL" >&2
+        return 1
+    fi
+}
+
+# Read in expected data from test data file.  If the $REVERSE flag is set to "true" then first
+# look for a file with the `_reverse` suffix. If one does not exist, use the default filename
+# for that suffix.
+# $1 is scenario file to use as a base name.
+# $2 is the suffix for the type of test data, i.e. "asset", "cl_items", etc.
+# If successful, prints contents of file on stdout and returns 0, otherwise returns 1.
+READ_IN_EXPECTED_DATA() {
+    local BASE_NAME=$(basename "$1" .scenario)
+    if [ "$REVERSE" = "true" ]; then
+        local EXPECTED_DATA_FILE_BW="$TEST_SCENARIO_DATA_DIR/${BASE_NAME}_"$2"_reverse.txt"
+        if [ -f "$EXPECTED_DATA_FILE_BW" ]; then
+            cat "$EXPECTED_DATA_FILE_BW"
+            return 0
+        fi
+    fi
+
+    local EXPECTED_DATA_FILE="$TEST_SCENARIO_DATA_DIR/${BASE_NAME}_"$2".txt"
+    if [ -f "$EXPECTED_DATA_FILE" ]; then
+        cat "$EXPECTED_DATA_FILE"
+        return 0
+    else
+        echo $(RED "$1 missing $2 file") >&2
+        return 1
+    fi
+}
+
+if [ "$REVERSE" = "true" ]; then
+    echo "Running ${#SCENARIOS[@]} scenarios in reverse"
+else
+    echo "Running ${#SCENARIOS[@]} scenarios forwards"
+fi
+
+# 0 is pass, 1 is fail.
+STATUS=0
+
+# Run each scenario and check for expected database result.
+for i in ${!SCENARIOS[@]}; do
+    # Read in the expected database data for this scenario.
+    EXPECTED_ASSET_VALUE=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "asset") || { STATUS=1; continue; }
+    EXPECTED_ASSET_CREATORS=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "asset_creators") || { STATUS=1; continue; }
+    EXPECTED_ASSET_GROUPING=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "asset_grouping") || { STATUS=1; continue; }
+    EXPECTED_CL_ITEMS=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "cl_items") || { STATUS=1; continue; }
+
+    # Parse out the asset ID.
+    ASSET_ID=$(echo "$EXPECTED_ASSET_VALUE" | grep -oP '^(?!tree_id).*id\s+\|\s+\K[^ ]+')
+    if [ ${#ASSET_ID} -ne 66 ]; then
+        echo $(RED "${SCENARIOS[$i]} incorrect asset ID parsing")
+        echo "Asset ID: $ASSET_ID"
+        STATUS=1
+        continue
+    fi
+
+    # Parse out the tree ID.
+    TREE_ID=$(echo "$EXPECTED_CL_ITEMS" | grep -oP '^\s*\K\\x[0-9a-f]+' | head -n 1)
+    if [ ${#TREE_ID} -ne 66 ]; then
+        echo $(RED "${SCENARIOS[$i]} incorrect asset ID parsing")
+        echo "Tree ID: $TREE_ID"
+        STATUS=1
+        continue
+    fi
+
+    # Initially this asset should not be in any database tables.
+    ASSET_SQL="SELECT * FROM asset WHERE id = '$ASSET_ID';"
+    CHECK_DATABASE "$ASSET_SQL" "-x" "(0 rows)" "initial asset table state" || STATUS=1
+
+    ASSET_CREATORS_SQL="SELECT asset_id, creator, share, verified, seq, slot_updated, position \
+        FROM asset_creators \
+        WHERE asset_id = '$ASSET_ID' \
+        ORDER BY position;"
+    CHECK_DATABASE "$ASSET_CREATORS_SQL" "-x" "(0 rows)" "initial asset_creators table state" || STATUS=1
+
+    ASSET_GROUPING_SQL="SELECT asset_id, group_key, group_value, seq, slot_updated, verified, group_info_seq \
+        FROM asset_grouping \
+        WHERE asset_id = '$ASSET_ID';"
+    CHECK_DATABASE "$ASSET_GROUPING_SQL" "-x" "(0 rows)" "initial asset_grouping table state" || STATUS=1
+
+    CL_ITEMS_SQL="select tree, node_idx, leaf_idx, seq, level, hash from cl_items where tree = '$TREE_ID' order by level;"
+    CHECK_DATABASE "$CL_ITEMS_SQL" "-x" "(0 rows)" "initial cl_items table state" || STATUS=1
+
+    # Run the scenario file that indexes the asset.  These are done with separate calls to the `txn_forwarder`
+    # in order to enforce order.  Just calling the `txn_forwarder` with the file results in random ordering.
+    readarray -t TXS < "$TEST_SCENARIO_DATA_DIR/${SCENARIOS[$i]}"
+
+    # Reverse transactions if necessary.
+    if [ "$REVERSE" = "true" ]; then
+        REVERSED_TXS=()
+        for ((j = ${#TXS[@]} - 1; j >= 0; j--)); do
+            REVERSED_TXS+=("${TXS[j]}")
+        done
+        TXS=("${REVERSED_TXS[@]}")
+    fi
+
+    for TX in ${TXS[@]}; do
+        (cd .. && \
+        cargo run -- \
+        --redis-url 'redis://localhost/' \
+        --rpc-url 'https://api.devnet.solana.com' \
+        single \
+        --txn  "$TX" \
+        2>&1 | grep -v "Group already exists: BUSYGROUP: Consumer Group name already exists")
+    done
+
+    sleep 2
+
+    # Asset should now be in the database and all fields match (except `created_at` in `asset`` table).
+    CHECK_DATABASE "$ASSET_SQL" "-x" "$EXPECTED_ASSET_VALUE" "asset table" || STATUS=1
+    CHECK_DATABASE "$ASSET_CREATORS_SQL" "-x" "$EXPECTED_ASSET_CREATORS" "asset_creators table" || STATUS=1
+    CHECK_DATABASE "$ASSET_GROUPING_SQL" "-x" "$EXPECTED_ASSET_GROUPING" "asset_grouping table" || STATUS=1
+    CHECK_DATABASE "$CL_ITEMS_SQL" "" "$EXPECTED_CL_ITEMS" "cl_items table" || STATUS=1
+
+    echo ""
+done
+
+if [ "$REVERSE" = "true" ]; then
+    SUFFIX="IN REVERSE"
+else
+    SUFFIX="FORWARDS"
+fi
+
+if [ $STATUS -eq 1 ]; then
+    echo $(RED "SOME TESTS FAILED $SUFFIX!")
+else
+    echo $(GRN "ALL TESTS PASSED $SUFFIX!")
+fi
+
+exit $STATUS

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer.scenario

@@ -0,0 +1,3 @@
+KNWsAYPo3mm1HuFxRyEwBBMUZ2hqTnFXjoPVFo7WxGTfmfRwz6K8eERc4dnJpHyuoDkAZu1czK55iB1SbtCsdW2
+3B1sASkuToCWuGFRG47axQDm1SpgLi8qDDGnRFeR7LB6oa5C3ZmkEuX98373gdMTBXED44FkwT227kBBAGSw7e8M
+5Q8TAMMkMTHEM2BHyD2fp2sVdYKByFeATzM2mHF6Xbbar33WaeuygPKGYCWiDEt3MZU1mUrq1ePnT9o4Pa318p8w

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_asset.txt

@@ -0,0 +1,28 @@
+-[ RECORD 1 ]-------------+-------------------------------------------------------------------
+id                        | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
+alt_id                    | 
+specification_version     | v1
+specification_asset_class | NFT
+owner                     | \xbd44aea35a84cb4ef1a03d3f66896abab67932a7fb812d6f583ef4099f59843a
+owner_type                | single
+delegate                  | 
+frozen                    | f
+supply                    | 1
+supply_mint               | 
+compressed                | t
+compressible              | f
+seq                       | 3
+tree_id                   | \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c
+leaf                      | \xa23d2bea8c65587f7f9fc627e3799ad8ffd0e222ccfda60e4327e07c51b8be3e
+nonce                     | 0
+royalty_target_type       | creators
+royalty_target            | 
+royalty_amount            | 0
+asset_data                | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
+burnt                     | f
+slot_updated              | 226274127
+data_hash                 | F5iDDHxd2DVZa5eZCqE2a91QLadea4ygJwM18UUut6dj      
+creator_hash              | EF57j46BT5Cynwija675rq59iDN1ZapYsJDnMHqta463      
+owner_delegate_seq        | 3
+leaf_seq                  | 3
+base_info_seq             | 1

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_asset_creators.txt

@@ -0,0 +1,16 @@
+-[ RECORD 1 ]+-------------------------------------------------------------------
+asset_id     | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
+creator      | \xdc53343c46e5d7a1bd08f4780285621202f66b596f984e940a5eb423ac560fed
+share        | 55
+verified     | f
+seq          | 1
+slot_updated | 226274127
+position     | 0
+-[ RECORD 2 ]+-------------------------------------------------------------------
+asset_id     | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
+creator      | \xe3dc3480d88f714c38827aada3064e8d212a91e3fdb67f22438c1ed89318a0e5
+share        | 45
+verified     | f
+seq          | 1
+slot_updated | 226274127
+position     | 1

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_asset_grouping.txt

@@ -0,0 +1,8 @@
+-[ RECORD 1 ]--+-------------------------------------------------------------------
+asset_id       | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
+group_key      | collection
+group_value    | 
+seq            | 
+slot_updated   | 226274127
+verified       | f
+group_info_seq | 1

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_cl_items.txt

@@ -0,0 +1,18 @@
+                                tree                                | node_idx | leaf_idx | seq | level |                                hash                                
+--------------------------------------------------------------------+----------+----------+-----+-------+--------------------------------------------------------------------
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |    16384 |        0 |   3 |     0 | \xa23d2bea8c65587f7f9fc627e3799ad8ffd0e222ccfda60e4327e07c51b8be3e
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |     8192 |          |   3 |     1 | \xe9152e451cbd119ea3bab38afbf6b4711199c73fdb385e44b06c2e707f8ca243
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |     4096 |          |   3 |     2 | \x111573f71f36e611641839e7e560bb73b680309841360c514965d168a4389a51
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |     2048 |          |   3 |     3 | \x4f64ec2a0d08484ec567cc0b4060dbfbceb9fa665d8a17dc8af9ebece7f01d9b
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |     1024 |          |   3 |     4 | \x4989debb3a21bd71c16a74d766abbd699650ee1aeae315fdda79a1728746da95
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |      512 |          |   3 |     5 | \x418f8841080e45b9b7cf09a3caaefd42976708bf0257c6c0806bb16ead54cf53
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |      256 |          |   3 |     6 | \xd4935ac4cb16da1b52198a0dac6d2609c14430ef5212809e294652ec25d9ef59
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |      128 |          |   3 |     7 | \xd26ec1beefbb64d05cc4c976c617485d82c240033e94e747129975b40638e111
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |       64 |          |   3 |     8 | \x3542d0fba80dd3681e3933f0ecd2ebd344a0ad96f23c8f2574099c7318c533e7
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |       32 |          |   3 |     9 | \x9d39b564f68ed8b88e6357f64e5cd0c82ca6d534aa2dab975d9666c8f56725d5
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |       16 |          |   3 |    10 | \x36d934448197c94e7d45f0863811edc196ed987620c41227e9f6ef1035c5934b
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |        8 |          |   3 |    11 | \x4637695dc219619cf5226f28ab04cb718f1140377ad01a5fff4fcd0fe2623be5
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |        4 |          |   3 |    12 | \x12374d301a2a513f3b23b3645c1c1c7789bf8c9c15ef80caf9cf46351dc53dcb
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |        2 |          |   3 |    13 | \x4e9fe09b7aa626a863a95c6b9808a2c63ec6cc685993bd43a0b1a347c18d920f
+ \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c |        1 |          |   3 |    14 | \x5f6a8668430a4bc94b280cdbba44efeb8847d9654ff034ffc1f537a66c120fb1
+(15 rows)

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_redeem_cancel_redeem_redeem_decompress.scenario

@@ -0,0 +1,5 @@
+3uzWoVgLGVd9cGXaF3JW7znpWgKse3obCa2Vvdoe59kaziX84mEXTwecUoZ49PkJDjReRMSXksKzyfj7pf3ekAGR
+49bJ8U3cK9htmLvA1mhXXcjKdpV2YN5JQBrb3Quh7wxENz1BP9F8fE9CKsje41aMbZwzgomnkXirKx2Xpdvprtak
+32FpSe6r9jnFNjjvbx2PPQdZqs5KpMoF6yawiRW1F6ctu1kmx2B4sLDBGjsthVQtmnhaJVrqdtmUP893FwXCbqY5
+3HqVaL9xgroAVYehnrk98dju4Ck3v8TqB4nfwnFCaNWKueaAVZcB3841jv5km1KpnubbUDoJAGv4ZMeLfAmdN18f
+3s3EN8gTthjjSXq5aV6imtBB58DQ1UGLch9dyzaACid4tAEDvqcEXoE3ibiAqxTaR5XSAArzPHgXtFM6FWvN7oDr

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_redeem_cancel_redeem_redeem_decompress_asset.txt

@@ -0,0 +1,28 @@
+-[ RECORD 1 ]-------------+-------------------------------------------------------------------
+id                        | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
+alt_id                    | 
+specification_version     | v1
+specification_asset_class | NFT
+owner                     | \xdc53343c46e5d7a1bd08f4780285621202f66b596f984e940a5eb423ac560fed
+owner_type                | single
+delegate                  | 
+frozen                    | f
+supply                    | 1
+supply_mint               | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
+compressed                | f
+compressible              | f
+seq                       | 0
+tree_id                   | 
+leaf                      | 
+nonce                     | 0
+royalty_target_type       | creators
+royalty_target            | 
+royalty_amount            | 0
+asset_data                | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
+burnt                     | f
+slot_updated              | 224494461
+data_hash                 | 
+creator_hash              | 
+owner_delegate_seq        | 3
+leaf_seq                  | 
+base_info_seq             | 1

tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_redeem_cancel_redeem_redeem_decompress_asset_creators.txt

@@ -0,0 +1,16 @@
+-[ RECORD 1 ]+-------------------------------------------------------------------
+asset_id     | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
+creator      | \xdc53343c46e5d7a1bd08f4780285621202f66b596f984e940a5eb423ac560fed
+share        | 55
+verified     | f
+seq          | 1
+slot_updated | 224494461
+position     | 0
+-[ RECORD 2 ]+-------------------------------------------------------------------
+asset_id     | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
+creator      | \xe6cd5c15dc19a877bc4bdf888a32145bd007f3a6ad52243937ffe41e3c2e7c3e
+share        | 45
+verified     | f
+seq          | 1
+slot_updated | 224494461
+position     | 1