jito patch

only reroute if relayer connected (solana-labs#123) feat: add client tls config (solana-labs#121) remove extra val (solana-labs#129) fix clippy (solana-labs#130) copy all binaries to docker-output (solana-labs#131) Ledger tool halts at slot passed to create-snapshot (solana-labs#118) update program submodule (solana-labs#133) quick fix for tips and clearing old bundles (solana-labs#135) update submodule to new program (solana-labs#136) Improve stake-meta-generator usability (solana-labs#134) pinning submodule head (solana-labs#140) Use BundleAccountLocker when handling tip txs (solana-labs#147) Add metrics for relayer + block engine proxy (solana-labs#149) Build claim-mev in docker (solana-labs#141) Rework bundle receiving and add metrics (solana-labs#152) (solana-labs#154) update submodule + dev files (solana-labs#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (solana-labs#159) update jito-programs submodule (solana-labs#160) Separate MEV tip related workflow (solana-labs#161) Add block builder fee protos (solana-labs#162) fix jito programs (solana-labs#163) update submodule so autosnapshot exits out of ledger tool early (solana-labs#164) Pipe through block builder fee (solana-labs#167) pull in new snapshot code (solana-labs#171) block builder bug (solana-labs#172) Pull in new slack autosnapshot submodule (solana-labs#174) sort stake meta json and use int math (solana-labs#176) add accountsdb conn submod (solana-labs#169) Update tip distribution parameters (solana-labs#177) new submodules (solana-labs#180) Add buildkite link for jito CI (solana-labs#183) Fixed broken links to repositories (solana-labs#184) Changed from ssh to https transfer for clone Seg/update submods (solana-labs#187) fix tests (solana-labs#190) rm geyser submod (solana-labs#192) rm dangling geyser references (solana-labs#193) fix syntax err (solana-labs#195) use deterministic req ids in batch calls (solana-labs#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (solana-labs#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (solana-labs#203) Add logging for sol balance desired (solana-labs#205) * add logging * add logging * update msg * tweak vars update submodule (solana-labs#204) use efficient data structures when calling batch_simulate_bundles (solana-labs#206) [JIT-504] Add low balance check in uploading merkle roots (solana-labs#209) add config to simulate on top of working bank (solana-labs#211) rm frozen bank check simulate_bundle rpc bugfixes (solana-labs#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (solana-labs#210) * add files * switch to include bump update submodule (solana-labs#217) add amount filter (solana-labs#218) update autosnapshot (solana-labs#222) Print TX error in Bundles (solana-labs#223) add new args to support single relayer and block-engine endpoints (solana-labs#224) point to new jito-programs submod and invoke updated init tda instruction (solana-labs#228) fix clippy errors (solana-labs#230) fix validator start scripts (solana-labs#232) Point README to gitbook (solana-labs#237) use packaged cargo bin to build (solana-labs#239) Add validator identity pubkey to StakeMeta (solana-labs#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (solana-labs#240) Seg/tip distributor master (solana-labs#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (solana-labs#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (solana-labs#247) Improve docker caching delay constructing claim mev txs (solana-labs#253) fix stake meta tests from bb fee (solana-labs#254) fix tests Buffer bundles that exceed cost model (solana-labs#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (solana-labs#260) update Cargo.lock to correct solana versions in jito-programs submodule (solana-labs#265) fix simulate_bundle client and better error handling (solana-labs#267) update submod (solana-labs#272) Preallocate Bundle Cost (solana-labs#238) fix Dockerfile (solana-labs#278) Fix Tests (solana-labs#279) Fix Tests (solana-labs#281) * fix tests update jito-programs submod (solana-labs#282) add reclaim rent workflow (solana-labs#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (solana-labs#292) Remove security.md (solana-labs#293) demote frequent relayer_stage-stream_error to warn (solana-labs#275) account for case where TDA exists but not allocated (solana-labs#295) implement better retries for tip-distributor workflows (solana-labs#297) limit number of concurrent rpc calls (solana-labs#298) Discard Empty Packet Batches (solana-labs#299) Identity Hotswap (solana-labs#290) small fixes (solana-labs#305) Set backend config from admin rpc (solana-labs#304) Admin Shred Receiver Change (solana-labs#306) Seg/rm bundle UUID (solana-labs#309) Fix github workflow to recursively clone (solana-labs#327) Add recursive checkout for downstream-project-spl.yaml (solana-labs#341) Use cluster info functions for tpu (solana-labs#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (solana-labs#374) Updates bootstrap and start scripts needed for local dev. (solana-labs#384) Remove Deprecated Cli Args (solana-labs#387) Master Rebase improve simulate_bundle errors and response (solana-labs#404) derive Clone on accountoverrides (solana-labs#416) Add upsert to AccountOverrides (solana-labs#419) update jito-programs (solana-labs#430) [JIT-1661] Faster Autosnapshot (solana-labs#436) Reverts simulate_transaction result calls to upstream (solana-labs#446) Don't unlock accounts in TransactionBatches used during simulation (solana-labs#449) first pass at wiring up jito-plugin (solana-labs#428) [JIT-1713] Fix bundle's blockspace preallocation (solana-labs#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (solana-labs#491) [JIT-1710] - Optimize Bundle Consumer Checks (solana-labs#490) Add Blockhash Metrics to Bundle Committer (solana-labs#500) add priority fee ix to mev-claim (solana-labs#520) Update Autosnapshot (solana-labs#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (solana-labs#582) Update CI (solana-labs#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (solana-labs#585) Fix more release urls (solana-labs#588) [JIT-1812] Fix blocking mutexs (solana-labs#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (solana-labs#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase solana-labs#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (solana-labs#597) Sleep longer waiting for buildkite to start (solana-labs#598) correctly initialize account overrides (solana-labs#595) Fix: Ensure set contact info to UDP port instead of QUIC (solana-labs#603) Add fast replay branch to daily rebase (solana-labs#607) take a snapshot of all bundle accounts before sim (#13) (solana-labs#615) update jito-programs submodule
ChorusOne · Jul 21, 2024 · 025f00e · 025f00e
1 parent c6b0570
commit 025f00e
Show file tree

Hide file tree

Showing 182 changed files with 17,799 additions and 871 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,9 @@
+.dockerignore
+.git/
+.github/
+.gitignore
+.idea/
+README.md
+Dockerfile
+f
+target/
diff --git a/.github/workflows/cargo.yml b/.github/workflows/cargo.yml
@@ -35,6 +35,8 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - uses: mozilla-actions/[email protected]
         with:

diff --git a/.github/workflows/changelog-label.yml b/.github/workflows/changelog-label.yml
@@ -13,6 +13,7 @@ jobs:
     - uses: actions/checkout@v4
       with:
         fetch-depth: 0
+        submodules: 'recursive'
     - name: Check if changes to CHANGELOG.md
       shell: bash
       env:

diff --git a/.github/workflows/client-targets.yml b/.github/workflows/client-targets.yml
@@ -32,6 +32,8 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - run: cargo install [email protected]
 
@@ -56,6 +58,8 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - name: Setup Rust
         run: |

diff --git a/.github/workflows/crate-check.yml b/.github/workflows/crate-check.yml
@@ -19,6 +19,7 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          submodules: 'recursive'
 
       - name: Get commit range (push)
         if: ${{ github.event_name == 'push' }}

diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -22,6 +22,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          submodules: 'recursive'
 
       - name: Get commit range (push)
         if: ${{ github.event_name == 'push' }}
@@ -77,6 +78,8 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - name: Setup Node
         uses: actions/setup-node@v4

diff --git a/.github/workflows/downstream-project-anchor.yml b/.github/workflows/downstream-project-anchor.yml
@@ -43,9 +43,11 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        version: ["v0.29.0", "v0.30.0"]
+        version: [ "v0.29.0", "v0.30.0" ]
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |

diff --git a/.github/workflows/downstream-project-spl.yml b/.github/workflows/downstream-project-spl.yml
@@ -42,6 +42,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |
@@ -68,7 +70,7 @@ jobs:
         arrays:
           [
             {
-              test_paths: ["token/cli"],
+              test_paths: [ "token/cli" ],
               required_programs:
                 [
                   "token/program",
@@ -78,14 +80,14 @@ jobs:
                 ],
             },
             {
-              test_paths: ["single-pool/cli"],
+              test_paths: [ "single-pool/cli" ],
               required_programs:
                 [
                   "single-pool/program",
                 ],
             },
             {
-              test_paths: ["token-upgrade/cli"],
+              test_paths: [ "token-upgrade/cli" ],
               required_programs:
                 [
                   "token-upgrade/program",
@@ -94,6 +96,8 @@ jobs:
           ]
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |
@@ -128,26 +132,28 @@ jobs:
     strategy:
       matrix:
         programs:
-          - [token/program]
+          - [ token/program ]
           - [
-              instruction-padding/program,
-              token/program-2022,
-              token/program-2022-test,
-            ]
+            instruction-padding/program,
+            token/program-2022,
+            token/program-2022-test,
+          ]
           - [
-              associated-token-account/program,
-              associated-token-account/program-test,
-            ]
-          - [token-upgrade/program]
-          - [feature-proposal/program]
-          - [governance/addin-mock/program, governance/program]
-          - [memo/program]
-          - [name-service/program]
-          - [stake-pool/program]
-          - [single-pool/program]
+            associated-token-account/program,
+            associated-token-account/program-test,
+          ]
+          - [ token-upgrade/program ]
+          - [ feature-proposal/program ]
+          - [ governance/addin-mock/program, governance/program ]
+          - [ memo/program ]
+          - [ name-service/program ]
+          - [ stake-pool/program ]
+          - [ single-pool/program ]
 
     steps:
       - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
 
       - shell: bash
         run: |

diff --git a/.github/workflows/rebase.yaml b/.github/workflows/rebase.yaml
@@ -0,0 +1,181 @@
+# This workflow runs a periodic rebase process, pulling in updates from an upstream repository
+# The workflow for rebasing a jito-solana branch to a solana labs branch locally is typically:
+# $ git checkout v1.17
+# $ git pull --rebase # --rebase needed locally
+# $ git branch -D lb/v1.17_rebase # deletes branch from last v1.17 rebase
+# $ git checkout -b lb/v1.17_rebase
+# $ git fetch upstream
+# $ git rebase upstream/v1.17 # rebase + fix merge conflicts
+# $ git rebase --continue
+# $ git push origin +lb/v1.17_rebase # force needed to overwrite remote. wait for CI, fix if any issues
+# $ git checkout v1.17
+# $ git reset --hard lb/v1.17_rebase
+# $ git push origin +v1.17
+#
+# This workflow automates this process, with periodic status updates over slack.
+# It will also run CI and wait for it to pass before performing the force push to v1.17.
+# In the event there's a failure in the process, it's reported to slack and the job stops.
+
+name: "Rebase jito-solana from upstream anza-xyz/agave"
+
+on:
+  #  push:
+  schedule:
+    - cron: "30 18 * * 1-5"
+
+jobs:
+  rebase:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - branch: master
+            rebase: upstream/master
+          - branch: v1.18
+            rebase: upstream/v1.18
+          - branch: v1.17
+            rebase: upstream/v1.17
+            # note: this will always be a day behind because we're rebasing from the previous day's rebase
+            # and NOT upstream
+          - branch: v1.17-fast-replay
+            rebase: origin/v1.17
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ matrix.branch }}
+          submodules: recursive
+          fetch-depth: 0
+          token: ${{ secrets.JITO_SOLANA_RELEASE_TOKEN }}
+      - name: Add upstream
+        run: git remote add upstream https://github.com/anza-xyz/agave.git
+      - name: Fetch upstream
+        run: git fetch upstream
+      - name: Fetch origin
+        run: git fetch origin
+      - name: Set REBASE_BRANCH
+        run: echo "REBASE_BRANCH=ci/nightly/${{ matrix.branch }}/$(date +'%Y-%m-%d-%H-%M')" >> $GITHUB_ENV
+      - name: echo $REBASE_BRANCH
+        run: echo $REBASE_BRANCH
+      - name: Create rebase branch
+        run: git checkout -b $REBASE_BRANCH
+      - name: Setup email
+        run: |
+          git config --global user.email "[email protected]"
+          git config --global user.name "Jito Infrastructure"
+      - name: Rebase
+        id: rebase
+        run: git rebase ${{ matrix.rebase }}
+      - name: Send warning for rebase error
+        if: failure() && steps.rebase.outcome == 'failure'
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: Rebase failed to apply cleanly"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      - name: Check if rebase applied
+        id: check_rebase_applied
+        run: |
+          PRE_REBASE_SHA=$(git rev-parse ${{ matrix.branch }})
+          POST_REBASE_SHA=$(git rev-parse HEAD)
+          if [ "$PRE_REBASE_SHA" = "$POST_REBASE_SHA" ]; then
+            echo "No rebase was applied, exiting..."
+            exit 1
+          else
+            echo "Rebase applied successfully."
+          fi
+      - name: Send warning for rebase error
+        if: failure() && steps.check_rebase_applied.outcome == 'failure'
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: Rebase not needed"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      - name: Set REBASE_SHA
+        run: echo "REBASE_SHA=$(git rev-parse HEAD)" >> $GITHUB_ENV
+      - name: Push changes
+        uses: ad-m/github-push-action@master
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          branch: ${{ env.REBASE_BRANCH }}
+      - name: Wait for buildkite to start build
+        run: sleep 300
+      - name: Wait for buildkite to finish
+        id: wait_for_buildkite
+        timeout-minutes: 300
+        run: |
+          while true; do
+            response=$(curl -s -f -H "Authorization: Bearer ${{ secrets.BUILDKITE_TOKEN }}" "https://api.buildkite.com/v2/organizations/jito/pipelines/jito-solana/builds?commit=${{ env.REBASE_SHA }}")
+            if [ $? -ne 0 ]; then
+              echo "Curl request failed."
+              exit 1
+            fi
+
+            state=$(echo $response | jq --exit-status -r '.[0].state')
+            echo "Current build state: $state"
+
+            # Check if the state is one of the finished states
+            case $state in
+              "passed"|"finished")
+                echo "Build finished successfully."
+                exit 0
+              ;;
+              "canceled"|"canceling"|"not_run")
+                # ignoring "failing"|"failed" because flaky CI, can restart and hope it finishes or times out
+                echo "Build failed or was cancelled."
+                exit 2
+              ;;
+            esac
+
+          sleep 30
+          done
+      - name: Send failure update
+        uses: slackapi/[email protected]
+        if: failure() && steps.wait_for_buildkite.outcome == 'failure'
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: CI failed\nBranch: ${{ env.REBASE_BRANCH}}\nBuild: https://buildkite.com/jito/jito-solana/builds?commit=${{ env.REBASE_SHA }}"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      # check to see if different branch since CI build can take awhile and these steps are not atomic
+      - name: Fetch the latest remote changes
+        run: git fetch origin ${{ matrix.branch }}
+      - name: Check if origin HEAD has changed from the beginning of the workflow
+        run: |
+          LOCAL_SHA=$(git rev-parse ${{ matrix.branch }})
+          ORIGIN_SHA=$(git rev-parse origin/${{ matrix.branch }})
+          if [ "$ORIGIN_SHA" != "$LOCAL_SHA" ]; then
+            echo "The remote HEAD of ${{ matrix.branch }} does not match the local HEAD of ${{ matrix.branch }} at the beginning of CI."
+            echo "origin sha: $ORIGIN_SHA"
+            echo "local sha: $LOCAL_SHA"
+            exit 1
+          else
+            echo "The remote HEAD matches the local REBASE_SHA at the beginning of CI. Proceeding."
+          fi
+      - name: Reset ${{ matrix.branch }} to ${{ env.REBASE_BRANCH }}
+        run: |
+          git checkout ${{ matrix.branch }}
+          git reset --hard ${{ env.REBASE_BRANCH }}
+      - name: Push rebased %{{ matrix.branch }}
+        uses: ad-m/github-push-action@master
+        with:
+          github_token: ${{ secrets.JITO_SOLANA_RELEASE_TOKEN }}
+          branch: ${{ matrix.branch }}
+          force: true
+      - name: Send success update
+        uses: slackapi/[email protected]
+        with:
+          payload: |
+            {
+              "text": "Nightly rebase on branch ${{ matrix.branch }}\nStatus: CI success, rebased, and pushed\nBranch: ${{ env.REBASE_BRANCH}}\nBuild: https://buildkite.com/jito/jito-solana/builds?commit=${{ env.REBASE_SHA }}"
+            }
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
diff --git a/.github/workflows/release-artifacts.yml b/.github/workflows/release-artifacts.yml
@@ -22,6 +22,7 @@ jobs:
         with:
           ref: master
           fetch-depth: 0
+          submodules: 'recursive'
 
       - name: Setup Rust
         shell: bash

diff --git a/.gitignore b/.gitignore
@@ -4,6 +4,7 @@ target/
 /solana-release.tar.bz2
 /solana-metrics/
 /solana-metrics.tar.bz2
+**/target/
 /test-ledger/
 
 **/*.rs.bk
@@ -27,7 +28,11 @@ log-*/
 # fetch-spl.sh artifacts
 /spl-genesis-args.sh
 /spl_*.so
+/jito_*.so
 
 .DS_Store
 # scripts that may be generated by cargo *-bpf commands
 **/cargo-*-bpf-child-script-*.sh
+
+.env
+docker-output/
diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,9 @@
+[submodule "anchor"]
+	path = anchor
+	url = https://github.com/jito-foundation/anchor.git
+[submodule "jito-programs"]
+	path = jito-programs
+	url = https://github.com/jito-foundation/jito-programs.git
+[submodule "jito-protos/protos"]
+	path = jito-protos/protos
+	url = https://github.com/jito-labs/mev-protos.git