Skip to content

Commit

Permalink
Danenbm/bubblegum sequence tests 2 (#160)
Browse files Browse the repository at this point in the history 
* Add script to forward transactions an check database results

* Fix ordering and add debug info

* Add remaining non-creator/non-collection tests

* Require asset and cl_items files to exist

* Add asset_creators and asset_grouping tests

* Add verify_creator and verify_collection tests

* Add more collection verification tests

* Move test data to subirectory

* Move repeated code to functions

* Add support for running sequences in reverse

* Add instructions to README for running test script

* Minor README update
  • Loading branch information
@danenbm
danenbm authored Jan 18, 2024
1 parent 024efaa commit 82a8d2a
Show file tree
Hide file tree
Showing 54 changed files with 1,009 additions and 5 deletions.
51 changes: 46 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,17 +96,16 @@ Developing with Docker is much easier, but has some nuances to it. This test doc
* Token 2022
* Latest version of the Associated token program

You need to run the following script (which takes a long time) in order to get all those .so files.
You need to run the following script in order to get the .so files.

```bash
chmod +x ./prepare-local-docker-env.sh
./prepare-local-docker-env.sh
```
This script grabs all the code for these programs and compiles it, and chucks it into your programs folder. Go grab some coffe because this will take a while/
If you get some permissions errors, just sudo delete the programs directory and start again.
This script downloads these programs from mainnet and puts them in the `programs/` folder.

#### Authentication with Docker and AWS

_This step is not normally needed for basic local docker usage._
```aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin {your aws container registry}```

#### Running the application
Expand All @@ -129,6 +128,48 @@ Also when mucking about with the docker file if your gut tells you that somethin

Sometimes you will want to delete the db do so with `sudo rm -rf db-data`. You can also delete the ledger with `sudo rm -rf ledger`.

#### Running Bubblegum Test Sequences

While running the multi-container Docker application locally, you can run a script located in `tools/txn_forwarder/bubblegum_tests` that will send sequences of bubblegum transactions via the `txn_forwarder`, and then use `psql` to read and verify the indexing results in the local Postgres database.

```bash
sudo rm -rf db-data/
sudo rm -rf ledger/
docker compose up --force-recreate --build
```
_In another terminal:_
```bash
cd tools/txn_forwarder/bubblegum_tests/
./run-bubblegum-sequences.sh
```

You should see it log something like:
```
Running 10 scenarios forwards
mint_transfer_burn.scenario initial asset table state passed
mint_transfer_burn.scenario initial asset_creators table state passed
mint_transfer_burn.scenario initial asset_grouping table state passed
mint_transfer_burn.scenario initial cl_items table state passed
...
mint_to_collection_unverify_collection.scenario asset table passed
mint_to_collection_unverify_collection.scenario asset_creators table passed
mint_to_collection_unverify_collection.scenario asset_grouping table passed
mint_to_collection_unverify_collection.scenario cl_items table passed
ALL TESTS PASSED FORWARDS!
```

You can also run the sequences in reverse:
```bash
./run-bubblegum-sequences.sh reverse
```
And after it runs you should see `ALL TESTS PASSED IN REVERSE!`

A few detailed notes about this test script:
* This script is not all-encompassing. It is only meant to automate some normal basic tests that were previously done manually. The reason this test is not added to CI is because requires a more powerful system to run the Docker application, which contains the no-vote Solana validator.
* The test sequences are in `.scenario` files, but instead of sending those files to the `txn_forwarder` directly (which supports the file format), we parse them out and send them individually using the `single` parameter. This is because using the `.scenario` file directly results in random ordering of the transactions and we are explicity trying to test them going forwards and in reverse.
* In general the expected database results are the same when running the transactions forwards and backwards. However, for assets that are decompressed, this is not true because we don't index some of the asset information from Bubblegum mint indexing if we already know the asset has been decompressed. We instead let Token Metadata account based indexing fill in that information. This is not reflected by this test script so the results differ when running these sequences in reverse. The differing results are reflected in test files with the `_reverse` suffix.

#### Logs
To get a reasonable amount of logs while running Docker, direct grafana logs to a file:
```
Expand All @@ -138,7 +179,7 @@ grafana:
...
- GF_LOG_MODE=file
```
and set Solana Rust logs to error level:
and set Solana Rust logs to error level (it is already set to error level now in the current docker compose file):
```
solana:
...
Expand Down
179 changes: 179 additions & 0 deletions tools/txn_forwarder/bubblegum_tests/run-bubblegum-sequences.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
#!/bin/bash

# Pass `reverse` to run scenarios in reverse.
if [ "$1" = "reverse" ]; then
REVERSE="true"
else
REVERSE="false"
fi

SCENARIOS=("mint_transfer_burn.scenario" \
"mint_redeem_decompress.scenario"
"mint_redeem_cancel_redeem_redeem_decompress.scenario" \
"mint_transfer_transfer.scenario" \
"mint_delegate_transfer.scenario" \
"mint_verify_creator.scenario" \
"mint_verify_collection.scenario" \
"mint_verify_collection_unverify_collection.scenario" \
"mint_set_and_verify_collection.scenario" \
"mint_to_collection_unverify_collection.scenario"
)

TEST_SCENARIO_DATA_DIR="test_scenario_data"

# Output text in colors.
# $1 is output text.
RED() { echo $'\e[1;31m'$1$'\e[0m'; }
GRN() { echo $'\e[1;32m'$1$'\e[0m'; }

# Read from database using psql and compare to expected value.
# $1 is SQL command.
# $2 extra CLI args to send to psql.
# $3 is expected value.
# $4 is topic for the pass/fail message.
# Returns 0 if database value matches expected value, otherwise returns 1.
CHECK_DATABASE() {
local DATABASE_VAL=$(PGPASSWORD=solana psql -h localhost -U solana "$2" --command="$1")
# Remove `created_at` since the date changes for `asset` table entries.
local DATABASE_VAL=$(sed '/^created_at/d' <<< "$DATABASE_VAL")
if [ "$3" = "$DATABASE_VAL" ]; then
echo $(GRN "${SCENARIOS[$i]} $4 passed") >&2
return 0
else
echo $(RED "${SCENARIOS[$i]} $4 failed") >&2
echo "Asset ID: $ASSET_ID" >&2
echo "Expected:" >&2
echo "$3" >&2
echo ""
echo "Actual:" >&2
echo "$DATABASE_VAL" >&2
return 1
fi
}

# Read in expected data from test data file. If the $REVERSE flag is set to "true" then first
# look for a file with the `_reverse` suffix. If one does not exist, use the default filename
# for that suffix.
# $1 is scenario file to use as a base name.
# $2 is the suffix for the type of test data, i.e. "asset", "cl_items", etc.
# If successful, prints contents of file on stdout and returns 0, otherwise returns 1.
READ_IN_EXPECTED_DATA() {
local BASE_NAME=$(basename "$1" .scenario)
if [ "$REVERSE" = "true" ]; then
local EXPECTED_DATA_FILE_BW="$TEST_SCENARIO_DATA_DIR/${BASE_NAME}_"$2"_reverse.txt"
if [ -f "$EXPECTED_DATA_FILE_BW" ]; then
cat "$EXPECTED_DATA_FILE_BW"
return 0
fi
fi

local EXPECTED_DATA_FILE="$TEST_SCENARIO_DATA_DIR/${BASE_NAME}_"$2".txt"
if [ -f "$EXPECTED_DATA_FILE" ]; then
cat "$EXPECTED_DATA_FILE"
return 0
else
echo $(RED "$1 missing $2 file") >&2
return 1
fi
}

if [ "$REVERSE" = "true" ]; then
echo "Running ${#SCENARIOS[@]} scenarios in reverse"
else
echo "Running ${#SCENARIOS[@]} scenarios forwards"
fi

# 0 is pass, 1 is fail.
STATUS=0

# Run each scenario and check for expected database result.
for i in ${!SCENARIOS[@]}; do
# Read in the expected database data for this scenario.
EXPECTED_ASSET_VALUE=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "asset") || { STATUS=1; continue; }
EXPECTED_ASSET_CREATORS=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "asset_creators") || { STATUS=1; continue; }
EXPECTED_ASSET_GROUPING=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "asset_grouping") || { STATUS=1; continue; }
EXPECTED_CL_ITEMS=$(READ_IN_EXPECTED_DATA "${SCENARIOS[$i]}" "cl_items") || { STATUS=1; continue; }

# Parse out the asset ID.
ASSET_ID=$(echo "$EXPECTED_ASSET_VALUE" | grep -oP '^(?!tree_id).*id\s+\|\s+\K[^ ]+')
if [ ${#ASSET_ID} -ne 66 ]; then
echo $(RED "${SCENARIOS[$i]} incorrect asset ID parsing")
echo "Asset ID: $ASSET_ID"
STATUS=1
continue
fi

# Parse out the tree ID.
TREE_ID=$(echo "$EXPECTED_CL_ITEMS" | grep -oP '^\s*\K\\x[0-9a-f]+' | head -n 1)
if [ ${#TREE_ID} -ne 66 ]; then
echo $(RED "${SCENARIOS[$i]} incorrect asset ID parsing")
echo "Tree ID: $TREE_ID"
STATUS=1
continue
fi

# Initially this asset should not be in any database tables.
ASSET_SQL="SELECT * FROM asset WHERE id = '$ASSET_ID';"
CHECK_DATABASE "$ASSET_SQL" "-x" "(0 rows)" "initial asset table state" || STATUS=1

ASSET_CREATORS_SQL="SELECT asset_id, creator, share, verified, seq, slot_updated, position \
FROM asset_creators \
WHERE asset_id = '$ASSET_ID' \
ORDER BY position;"
CHECK_DATABASE "$ASSET_CREATORS_SQL" "-x" "(0 rows)" "initial asset_creators table state" || STATUS=1

ASSET_GROUPING_SQL="SELECT asset_id, group_key, group_value, seq, slot_updated, verified, group_info_seq \
FROM asset_grouping \
WHERE asset_id = '$ASSET_ID';"
CHECK_DATABASE "$ASSET_GROUPING_SQL" "-x" "(0 rows)" "initial asset_grouping table state" || STATUS=1

CL_ITEMS_SQL="select tree, node_idx, leaf_idx, seq, level, hash from cl_items where tree = '$TREE_ID' order by level;"
CHECK_DATABASE "$CL_ITEMS_SQL" "-x" "(0 rows)" "initial cl_items table state" || STATUS=1

# Run the scenario file that indexes the asset. These are done with separate calls to the `txn_forwarder`
# in order to enforce order. Just calling the `txn_forwarder` with the file results in random ordering.
readarray -t TXS < "$TEST_SCENARIO_DATA_DIR/${SCENARIOS[$i]}"

# Reverse transactions if necessary.
if [ "$REVERSE" = "true" ]; then
REVERSED_TXS=()
for ((j = ${#TXS[@]} - 1; j >= 0; j--)); do
REVERSED_TXS+=("${TXS[j]}")
done
TXS=("${REVERSED_TXS[@]}")
fi

for TX in ${TXS[@]}; do
(cd .. && \
cargo run -- \
--redis-url 'redis://localhost/' \
--rpc-url 'https://api.devnet.solana.com' \
single \
--txn "$TX" \
2>&1 | grep -v "Group already exists: BUSYGROUP: Consumer Group name already exists")
done

sleep 2

# Asset should now be in the database and all fields match (except `created_at` in `asset`` table).
CHECK_DATABASE "$ASSET_SQL" "-x" "$EXPECTED_ASSET_VALUE" "asset table" || STATUS=1
CHECK_DATABASE "$ASSET_CREATORS_SQL" "-x" "$EXPECTED_ASSET_CREATORS" "asset_creators table" || STATUS=1
CHECK_DATABASE "$ASSET_GROUPING_SQL" "-x" "$EXPECTED_ASSET_GROUPING" "asset_grouping table" || STATUS=1
CHECK_DATABASE "$CL_ITEMS_SQL" "" "$EXPECTED_CL_ITEMS" "cl_items table" || STATUS=1

echo ""
done

if [ "$REVERSE" = "true" ]; then
SUFFIX="IN REVERSE"
else
SUFFIX="FORWARDS"
fi

if [ $STATUS -eq 1 ]; then
echo $(RED "SOME TESTS FAILED $SUFFIX!")
else
echo $(GRN "ALL TESTS PASSED $SUFFIX!")
fi

exit $STATUS
3 changes: 3 additions & 0 deletions tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer.scenario
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
KNWsAYPo3mm1HuFxRyEwBBMUZ2hqTnFXjoPVFo7WxGTfmfRwz6K8eERc4dnJpHyuoDkAZu1czK55iB1SbtCsdW2
3B1sASkuToCWuGFRG47axQDm1SpgLi8qDDGnRFeR7LB6oa5C3ZmkEuX98373gdMTBXED44FkwT227kBBAGSw7e8M
5Q8TAMMkMTHEM2BHyD2fp2sVdYKByFeATzM2mHF6Xbbar33WaeuygPKGYCWiDEt3MZU1mUrq1ePnT9o4Pa318p8w
28 changes: 28 additions & 0 deletions tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_asset.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-[ RECORD 1 ]-------------+-------------------------------------------------------------------
id | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
alt_id |
specification_version | v1
specification_asset_class | NFT
owner | \xbd44aea35a84cb4ef1a03d3f66896abab67932a7fb812d6f583ef4099f59843a
owner_type | single
delegate |
frozen | f
supply | 1
supply_mint |
compressed | t
compressible | f
seq | 3
tree_id | \x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c
leaf | \xa23d2bea8c65587f7f9fc627e3799ad8ffd0e222ccfda60e4327e07c51b8be3e
nonce | 0
royalty_target_type | creators
royalty_target |
royalty_amount | 0
asset_data | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
burnt | f
slot_updated | 226274127
data_hash | F5iDDHxd2DVZa5eZCqE2a91QLadea4ygJwM18UUut6dj
creator_hash | EF57j46BT5Cynwija675rq59iDN1ZapYsJDnMHqta463
owner_delegate_seq | 3
leaf_seq | 3
base_info_seq | 1
16 changes: 16 additions & 0 deletions ...xn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_asset_creators.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
-[ RECORD 1 ]+-------------------------------------------------------------------
asset_id | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
creator | \xdc53343c46e5d7a1bd08f4780285621202f66b596f984e940a5eb423ac560fed
share | 55
verified | f
seq | 1
slot_updated | 226274127
position | 0
-[ RECORD 2 ]+-------------------------------------------------------------------
asset_id | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
creator | \xe3dc3480d88f714c38827aada3064e8d212a91e3fdb67f22438c1ed89318a0e5
share | 45
verified | f
seq | 1
slot_updated | 226274127
position | 1
8 changes: 8 additions & 0 deletions ...xn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_asset_grouping.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-[ RECORD 1 ]--+-------------------------------------------------------------------
asset_id | \x5aed3f1d3faf9f5f4664f98055d25627d59d0351ee8f0802ebbb33ccd8220d67
group_key | collection
group_value |
seq |
slot_updated | 226274127
verified | f
group_info_seq | 1
18 changes: 18 additions & 0 deletions tools/txn_forwarder/bubblegum_tests/test_scenario_data/mint_delegate_transfer_cl_items.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
tree | node_idx | leaf_idx | seq | level | hash
--------------------------------------------------------------------+----------+----------+-----+-------+--------------------------------------------------------------------
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 16384 | 0 | 3 | 0 | \xa23d2bea8c65587f7f9fc627e3799ad8ffd0e222ccfda60e4327e07c51b8be3e
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 8192 | | 3 | 1 | \xe9152e451cbd119ea3bab38afbf6b4711199c73fdb385e44b06c2e707f8ca243
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 4096 | | 3 | 2 | \x111573f71f36e611641839e7e560bb73b680309841360c514965d168a4389a51
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 2048 | | 3 | 3 | \x4f64ec2a0d08484ec567cc0b4060dbfbceb9fa665d8a17dc8af9ebece7f01d9b
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 1024 | | 3 | 4 | \x4989debb3a21bd71c16a74d766abbd699650ee1aeae315fdda79a1728746da95
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 512 | | 3 | 5 | \x418f8841080e45b9b7cf09a3caaefd42976708bf0257c6c0806bb16ead54cf53
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 256 | | 3 | 6 | \xd4935ac4cb16da1b52198a0dac6d2609c14430ef5212809e294652ec25d9ef59
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 128 | | 3 | 7 | \xd26ec1beefbb64d05cc4c976c617485d82c240033e94e747129975b40638e111
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 64 | | 3 | 8 | \x3542d0fba80dd3681e3933f0ecd2ebd344a0ad96f23c8f2574099c7318c533e7
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 32 | | 3 | 9 | \x9d39b564f68ed8b88e6357f64e5cd0c82ca6d534aa2dab975d9666c8f56725d5
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 16 | | 3 | 10 | \x36d934448197c94e7d45f0863811edc196ed987620c41227e9f6ef1035c5934b
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 8 | | 3 | 11 | \x4637695dc219619cf5226f28ab04cb718f1140377ad01a5fff4fcd0fe2623be5
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 4 | | 3 | 12 | \x12374d301a2a513f3b23b3645c1c1c7789bf8c9c15ef80caf9cf46351dc53dcb
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 2 | | 3 | 13 | \x4e9fe09b7aa626a863a95c6b9808a2c63ec6cc685993bd43a0b1a347c18d920f
\x1163033de45ffcf58694abca011d79e5c3682c00c65caf710490650c151b7c4c | 1 | | 3 | 14 | \x5f6a8668430a4bc94b280cdbba44efeb8847d9654ff034ffc1f537a66c120fb1
(15 rows)
5 changes: 5 additions & 0 deletions ...r/bubblegum_tests/test_scenario_data/mint_redeem_cancel_redeem_redeem_decompress.scenario
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
3uzWoVgLGVd9cGXaF3JW7znpWgKse3obCa2Vvdoe59kaziX84mEXTwecUoZ49PkJDjReRMSXksKzyfj7pf3ekAGR
49bJ8U3cK9htmLvA1mhXXcjKdpV2YN5JQBrb3Quh7wxENz1BP9F8fE9CKsje41aMbZwzgomnkXirKx2Xpdvprtak
32FpSe6r9jnFNjjvbx2PPQdZqs5KpMoF6yawiRW1F6ctu1kmx2B4sLDBGjsthVQtmnhaJVrqdtmUP893FwXCbqY5
3HqVaL9xgroAVYehnrk98dju4Ck3v8TqB4nfwnFCaNWKueaAVZcB3841jv5km1KpnubbUDoJAGv4ZMeLfAmdN18f
3s3EN8gTthjjSXq5aV6imtBB58DQ1UGLch9dyzaACid4tAEDvqcEXoE3ibiAqxTaR5XSAArzPHgXtFM6FWvN7oDr
28 changes: 28 additions & 0 deletions .../bubblegum_tests/test_scenario_data/mint_redeem_cancel_redeem_redeem_decompress_asset.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-[ RECORD 1 ]-------------+-------------------------------------------------------------------
id | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
alt_id |
specification_version | v1
specification_asset_class | NFT
owner | \xdc53343c46e5d7a1bd08f4780285621202f66b596f984e940a5eb423ac560fed
owner_type | single
delegate |
frozen | f
supply | 1
supply_mint | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
compressed | f
compressible | f
seq | 0
tree_id |
leaf |
nonce | 0
royalty_target_type | creators
royalty_target |
royalty_amount | 0
asset_data | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
burnt | f
slot_updated | 224494461
data_hash |
creator_hash |
owner_delegate_seq | 3
leaf_seq |
base_info_seq | 1
16 changes: 16 additions & 0 deletions ...m_tests/test_scenario_data/mint_redeem_cancel_redeem_redeem_decompress_asset_creators.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
-[ RECORD 1 ]+-------------------------------------------------------------------
asset_id | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
creator | \xdc53343c46e5d7a1bd08f4780285621202f66b596f984e940a5eb423ac560fed
share | 55
verified | f
seq | 1
slot_updated | 224494461
position | 0
-[ RECORD 2 ]+-------------------------------------------------------------------
asset_id | \x4302561a2799244936a3b020569bf90f58cd9b406af4cd084a82a2cf6415a8f0
creator | \xe6cd5c15dc19a877bc4bdf888a32145bd007f3a6ad52243937ffe41e3c2e7c3e
share | 45
verified | f
seq | 1
slot_updated | 224494461
position | 1
Loading

0 comments on commit 82a8d2a

Please sign in to comment.