Align triton-build with upstream 1/26/24 (#127)

* Update raw name and raw symbol for existing NFTs (#139) Update the raw name and raw symbol from the onchain data for existing NFTs when reingesting. This allows correction of incorrect values during reprocessing on an existing index. * Upstream Helius features (#133) * Bubblegum Update Metadata Version 1 (#134) * Add code to index Bubblegum Update Metadata * Update rust toolchain file * Fix moved variable after merge * Add code from mintV1 that allows for empty URI * Ordering using asset.seq initially applied to update_metadata * Add simple check for whether asset was decompressed to Bubblegum transformers * Don't prevent sequence number update when already decompressed * Add sequence number to downloading metadata background task * Add sequence number migration (Sea ORM not regenerated yet) * Regenerate Sea-ORM types * Use new sequence numbers for Bubblegum Update Metadata * Extra condition to protect out of order creator verification * Remove base_info_seq for each creator and add creators_added_seq to asset table * Regenerate Sea-ORM types * Change creator metadata updates to use new creators_added_seq * Factor out common creator update code to helper function * Update to latest blockbuster beta * Use less than or equal for download metadata seq check * Index verified for token metadata collection * Add slot_updated to initial asset upsert, and removed duplicate items * Remove asset_was_decompressed Replaced with WHERE clauses on each upsert. Move remaining upserts from mint_v1 to db.rs. Remove upsert to asset_v1_account_attachments from mint_V1. Combine upserts for asset base info and royalty amount. * Rename royalty_amount_seq to base_info_seq * Fix typo in WHERE clause * Do not delete existing creators in mint_v1 * Update comments around database txns * Use transaction in mint_V1 and update_metadata * Use transaction for other Bubblegum instructions asset table updates * Fix tree_id key index in update_metadata * Remove use of was_decompressed flag on asset table * Add migration to remove was_decompressed and regenerate SeaORM types * Combine upsert_asset_base_info and upsert_creators and add lock * Remove unneeded creators_added_seq * Switch to EXCLUSIVE mode lock * Add NULL condition check on asset.seq * Refactored creator indexing * Use new Blockbuster that always updates all creators and verification status. * Remove deleting creators with lower sequence numbers as it would not work due to race conditions. * Add concept of "empty" creator value to support Bubblegum empty creator arrays. * Add filtering out of old creators or having no creators to DAS code. * Also get authority and tree_id accounts from Bubblegum during mint and update_metadata. * Add conditions to creator upsert, add another check at DAS API level * Rename asset_creators.verified_seq back to just regular seq * Remove unneeded condition on asset_authority upsert * Apply stale creator filtering to all DAS API queries * Use latest blockbuster beta release * Remove download_metadata_seq and add URI match check instead * Fix task URI initial query * Regenerate Sea ORM types without download_metadata_seq * asset_grouping.verified option remove * Fix filtering for getAssetsByCreator * Update to blockbuster 0.9.0-beta.5 and mpl-bubblegum 1.0.1-beta.4 * Configurable account streams (#148) * Make workers configurable Make workers fully configurab le and remove reference to the plerkle plugin. * fix lifetime --------- Co-authored-by: Kirill Fomichev <[email protected]> * fix: splt tokens with no token stanard are incorrectly categorized as single. Logic in token account updates would change owner when any token account had amount > 0 would triggers spam updates of the owner of the asset with any transfer. (#151) * Improve workspace usage (#141) * fix: remove decompress ix handling and use db transactions (#156) * Improvements for consistency NFT indexing and query consistency (#157) * Danenbm/bubblegum sequence tests 2 (#160) * Add script to forward transactions an check database results * Fix ordering and add debug info * Add remaining non-creator/non-collection tests * Require asset and cl_items files to exist * Add asset_creators and asset_grouping tests * Add verify_creator and verify_collection tests * Add more collection verification tests * Move test data to subirectory * Move repeated code to functions * Add support for running sequences in reverse * Add instructions to README for running test script * Minor README update * feat: `getSignaturesForAsset` endpoint on top of new `cl_audits_v2` table (#155) * add migration files for cl_audits_v2 * add types * ingester * add getSignaturesForAsset endpoint * refactor to resolve merge conflict related bugs * address clippy error * rename to get_asset_signatures * add instruction type update_metadata * add error log if instruction is unknown * add sort order changes --------- Co-authored-by: Nicolas Pennie <[email protected]> * fix ORM circular dependency (#161) * fix ORM circular dependency * PR comments --------- Co-authored-by: Linus Kendall <[email protected]> Co-authored-by: Tahsin Tunan <[email protected]> Co-authored-by: Michael Danenberg <[email protected]> Co-authored-by: Kirill Fomichev <[email protected]> Co-authored-by: Nicolas Pennie <[email protected]>
rpcpool · Jan 26, 2024 · 2ffedab · 2ffedab
1 parent 090897a
commit 2ffedab
Show file tree

Hide file tree

Showing 196 changed files with 5,175 additions and 14,412 deletions.
diff --git a/.github/workflows/build-api.yml b/.github/workflows/build-api.yml
@@ -0,0 +1,113 @@
+name: Build das api components
+# This workflow uses github runners.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+# This may be adjusted to whatever suits best your runners config.
+# Current config will build on manual trigger or pull-request (each push)
+on:
+  # pull_request can be removed, to save minutes on github runners
+  pull_request:
+  workflow_dispatch:
+  push:
+    branches:
+      - 'main'
+
+env:
+   CARGO_TERM_COLOR: always
+
+jobs:
+  build-api:
+    strategy:
+      matrix:
+        os: [ubuntu-20.04, ubuntu-22.04]
+    # This can be also be runned on self-hosted github runners
+    runs-on: ["${{ matrix.os }}"]
+
+    steps:
+      - name: checkout repo
+        uses: actions/checkout@v3
+      # This step can be omited, to save storage space on the organization account
+      # Build process will take longer
+      - name: set build cache
+        uses: actions/cache@v3
+        with:
+            path: |
+                ~/.cargo/bin/
+                ~/.cargo/registry/index/
+                ~/.cargo/registry/cache/
+                ~/.cargo/git/db/
+                digital-asset-rpc-infrastructure/target/
+            key: ${{ matrix.os }}_digital-asset-rpc-infrastructure_${{ hashFiles('digital-asset-rpc-infrastructure/Cargo.lock') }}
+            restore-keys: |
+              ${{ matrix.os }}_digital-asset-rpc-infrastructure
+
+      - name: build digital asset rpc infra
+        run: cargo build --verbose --release
+
+      - name: rename binaries for ubuntu22 release
+        if: matrix.os == 'ubuntu-22.04'
+        run: |
+          mv target/release/nft_ingester target/release/nft_ingester22
+          mv target/release/fetch_trees target/release/fetch_trees22 
+          mv target/release/migration target/release/migration22
+          mv target/release/das_api target/release/das_api22
+
+      # This steps can be omited to save space, are mostly in place to validate binaries (manually) and path to them 
+      # Omiting this will save on storage consumption on the account
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-22.04'
+        uses: actions/[email protected]
+        with:
+          name: nft_ingester22
+          path: target/release/nft_ingester22
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-22.04'
+        uses: actions/[email protected]
+        with:
+          name: das_api22
+          path: target/release/das_api22
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-22.04'
+        uses: actions/[email protected]
+        with:
+          name: migration22
+          path: target/release/migration22
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-22.04'
+        uses: actions/[email protected]
+        with:
+          name: fetch-trees22
+          path: target/release/fetch_trees22
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-20.04'
+        uses: actions/[email protected]
+        with:
+          name: nft_ingester
+          path: target/release/nft_ingester
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-20.04'
+        uses: actions/[email protected]
+        with:
+          name: das_api
+          path: target/release/das_api
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-20.04'
+        uses: actions/[email protected]
+        with:
+          name: migration
+          path: target/release/migration
+
+      - name: Publish artifact
+        if: matrix.os == 'ubuntu-20.04'
+        uses: actions/[email protected]
+        with:
+          name: fetch-trees
+          path: target/release/fetch_trees
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -1,35 +1,27 @@
-name: Build das api components
-# This workflow uses github runners.
+name: Check lock file, fmt, clippy
+
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
-# This may be adjusted to whatever suits best your runners config.
-# Current config will build on manual trigger or pull-request (each push)
 on:
-  # pull_request can be removed, to save minutes on github runners
   pull_request:
-  workflow_dispatch:
   push:
     branches:
       - 'main'
+  workflow_dispatch:
 
 env:
    CARGO_TERM_COLOR: always
 
 jobs:
-  build-api:
-    strategy:
-      matrix:
-        os: [ubuntu-20.04, ubuntu-22.04]
-    # This can be also be runned on self-hosted github runners
-    runs-on: ["${{ matrix.os }}"]
+  test:
+    runs-on: ubuntu-22.04
 
     steps:
       - name: checkout repo
         uses: actions/checkout@v3
-      # This step can be omited, to save storage space on the organization account
-      # Build process will take longer
+
       - name: set build cache
         uses: actions/cache@v3
         with:
@@ -39,83 +31,19 @@ jobs:
                 ~/.cargo/registry/cache/
                 ~/.cargo/git/db/
                 digital-asset-rpc-infrastructure/target/
-            key: ${{ matrix.os }}_digital-asset-rpc-infrastructure_${{ hashFiles('digital-asset-rpc-infrastructure/Cargo.lock') }}
-            restore-keys: |
-              ${{ matrix.os }}_digital-asset-rpc-infrastructure
-
-      - name: build digital asset rpc infra
-        run: cargo build --verbose --release
-
-      - name: build das_api
-        working-directory: das_api
-        run: cargo build --verbose --release
+            key: cargo-${{ hashFiles('**/Cargo.lock') }}-0001
 
-      - name: build migration
-        working-directory: migration
-        run: cargo build --verbose --release
-
-      - name: rename binaries for ubuntu22 release
-        if: matrix.os == 'ubuntu-22.04'
+      # Cargo.lock
+      - name: Check lock file
         run: |
-          mv target/release/nft_ingester target/release/nft_ingester22
-          mv target/release/fetch_trees target/release/fetch_trees22 
-          mv das_api/target/release/das_api das_api/target/release/das_api22 
-          mv migration/target/release/migration migration/target/release/migration22
+          cargo tree
+          git checkout Cargo.lock
+          cargo tree --frozen
 
-      # This steps can be omited to save space, are mostly in place to validate binaries (manually) and path to them 
-      # Omiting this will save on storage consumption on the account
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-22.04'
-        uses: actions/[email protected]
-        with:
-          name: nft_ingester22
-          path: target/release/nft_ingester22
+      # fmt
+      - name: Check fmt
+        run: cargo fmt --all -- --check
 
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-22.04'
-        uses: actions/[email protected]
-        with:
-          name: das_api22
-          path: das_api/target/release/das_api22
-
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-22.04'
-        uses: actions/[email protected]
-        with:
-          name: migration22
-          path: migration/target/release/migration22
-
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-22.04'
-        uses: actions/[email protected]
-        with:
-          name: fetch-trees22
-          path: migration/target/release/fetch_trees22
-
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-20.04'
-        uses: actions/[email protected]
-        with:
-          name: nft_ingester
-          path: target/release/nft_ingester
-
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-20.04'
-        uses: actions/[email protected]
-        with:
-          name: das_api
-          path: das_api/target/release/das_api
-
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-20.04'
-        uses: actions/[email protected]
-        with:
-          name: migration
-          path: migration/target/release/migration
-
-      - name: Publish artifact
-        if: matrix.os == 'ubuntu-20.04'
-        uses: actions/[email protected]
-        with:
-          name: fetch-trees
-          path: target/release/fetch_trees
+      # clippy
+      - name: Check clippy
+        run: cargo clippy --all-targets --tests
diff --git a/.gitignore b/.gitignore
@@ -23,10 +23,3 @@ programs
 *.iml
 skaffold-state.json
 test-programs
-
-# Rust build dirs
-/das_api/target
-/migration/target
-/tests/*/target
-/*/target
-/target