get rid of const in tests

Co-authored-by: HaoranYi <[email protected]>

* rpc: add tpu_forwards/tpu_forwards_quic to getClusterNodes

* add tvu

* order

* add tpu_vote

* add serve_repair

* fix tests

* fix name in tests

* remove tvu / repair quic

…yz#1746)

* remove unused build dep `cc` from solana-program

* remove cc from workspace

* Remove redundant Option return type from accountinfo_from_shared_account_data

* Remove redundant Option return type from accountinfo_from_stored_account_meta

There are a handful of commands that specify AccountsDb config. To
create consistency between all these commands and remove repeated code,
build all of the arguments in a helper function that is passed to the
relevant commands.

Add lamports burned to error log

Add log to unreachable macro

…-xyz#1654)


Co-authored-by: Jon C <[email protected]>

…s for partitions (anza-xyz#1769)

Simplify assertion that epoch contains enough slots for partitions

…a-xyz#1758)

* docs: update ZK Token Proof program to ZK ElGamal Proof program

* remove blank line at eof

* local program cache: add `modified_entries` field

* use `modified_entries` for modified program cache

* invoke context: make `program_cache_for_tx_batch` mutable

* invoke context: unify local program cache instances

* remove `find_program_in_cache` alias

… one (anza-xyz#1736)

* Add num_partitions field to Rewards proto definition

* Add type to hold rewards plus num_partitions

* Add Bank method to get rewards plus num_partitions for recording

* Update Blockstore::write_rewards to use num_partitions

* Update RewardsRecorderService to handle num_partitions

* Populate num_partitions in ReplayStage::record_rewards

* Write num_partitions to Bigtable

* Reword KeyedRewardsAndNumPartitions method

* Clone immediately

* Determine epoch boundary by checking parent epoch

* Rename UiConfirmedBlock field

* nit: fix comment typo

* Add test_get_rewards_and_partitions

* Add pre-activation test

* Add should_record unit test

)

* send actual gossip ip:port when running solana-gossip rpc-url

* refactor gossip_addr out from spy and rpc methods

* add `zk-elgamal-proof-program-enabled` feature gate

* reserve account for zk elgamal proof program

* add zk elgamal proof as builtin program

* cargo lock

* cargo sort

* replace feature gate key to start with `zk..`

* add simd number

* update `elgamal_program` to `zk_elgamal_proof_program`

There are several arguments to control snapshot configuration in the
various ledger-tool commands. The inclusion of args in each command
is inconsistent, especially for commands outside of main.rs

This change consolidates the snapshot related arguments into a single
function to help create consistency and reduce duplicate code

…z#1673)

AccountsBackgroundService performs several operations that can take a
long time to complete and do not check the exit flag mid-operation.
Thus, ledger-tool can get hung up for a while waiting for ABS to
finish. However, many ledger-tool command do not ABS to have finished.

So, return a handle to the ABS thread and allow the caller to decide
whether to join ABS or not. As of right now, create-snapshot is the
only command that requires ABS to have finished before continuing.

* extract curve25519 crate

* remove obsolete comment

* fix Cargo.toml files

* fix imports

* update lock file

* remove unused deps from zk-token-sdk

* fmt

* add solana-curve25519 patch

* add missing override to programs/sbf/Cargo.toml

* copy over an allow()

* move new crate to curves dir

* use workspace version

* add back missing dev dep

* add missing dependencies to programs/sbf

* fmt

* move dep to the correct dependency table

* remove #[cfg(not(target_os = "solana"))] above errors mod

Consider this scenario:

 - Program increases length of an account
 - Program start CPI and adds this account as a read-only account
 - In fn update_callee_account() we resize account, which may change
   the pointer
 - Once CPI finishes, the program continues and may read/write from
   the account. The mapping must be up-to-date else we use stale
   pointers.

Note that we always call callee_account.set_data_length(), which
may change the pointer. In testing I found that resizing a vector
from 10240 down to 127 sometimes changes its pointer. So, always
update the pointer.

* remove `ElGamalError` from curve25519 crate

* add `ElGamalError` to zk-token-sdk

…1658)

* move wasm-bindgen dep under cfg(target_arch = "wasm32") in sdk and program

* remove wasm_bindgen_stub (we don't need it where we're going)

* put wasm_bindgen usage behind #[cfg(target_arch = "wasm32")]

* remove doc comments from skippeed fields

* add missing attribute

* another missing attribute

* add doc comments explaining duplicated structs

* fmt

* fix wasm comments

* Update sdk/program/src/instruction.rs

---------

Co-authored-by: Jon C <[email protected]>

…ToUiAmount` (anza-xyz#1737)

* transaction-status: Use string instead of int for `amount`

* Add a changelog entry

* Update CHANGELOG.md

Co-authored-by: Tyera <[email protected]>

---------

Co-authored-by: Tyera <[email protected]>

* SVM: add `fee_structure` to environment arg

* runtime: add `fee_structure` to bank

* SVM: drop `fee_structure` from global configs

* always pack a few newest ancient slots

* pr feedback

* remove extra ()

* adds high slot tests

---------

Co-authored-by: brooks <[email protected]>

…nza-xyz#1774)

The argument is currently declared in multiple places. So, delcare the
argument in one central place.

* add ancient pack metrics

* add missing params

---------

Co-authored-by: brooks <[email protected]>

* Removes ProgramTest from simulation tests.

* Removes ProgramTest from sysvar syscall tests.

* Workaround for rustc crash caused by 16 byte aligned memcpy.

* Deduplicates test_program_sbf_sanity.

* Moves mem and remaining_compute_units into test_program_sbf_sanity().

* Removes unused dev-dependencies in Cargo.toml.

* Removes crate-type = lib from Cargo.tomls.

* Adds SBF_OUT_DIR env to CI script.

* Adds "sysvar" to build.rs.

ci: ignore curve25519-dalek audit

* build(deps): bump bytemuck from 1.16.0 to 1.16.1

Bumps [bytemuck](https://github.com/Lokathor/bytemuck) from 1.16.0 to 1.16.1.
- [Changelog](https://github.com/Lokathor/bytemuck/blob/main/changelog.md)
- [Commits](Lokathor/bytemuck@v1.16.0...v1.16.1)

---
updated-dependencies:
- dependency-name: bytemuck
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump url from 2.5.1 to 2.5.2

Bumps [url](https://github.com/servo/rust-url) from 2.5.1 to 2.5.2.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](servo/rust-url@v2.5.1...v2.5.2)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* port join from itertools and use it in program_stubs.rs

* move itertools to dev-dependencies of solana-program

* add comment to join fn

* more concise replacement for join fn

Co-authored-by: Jon C <[email protected]>

* remove join fn

---------

Co-authored-by: Jon C <[email protected]>

* Adjust replay-related metrics for unified schduler

* Fix grammar

* Don't compute slowest for unified scheduler

* Rename to is_unified_scheduler_enabled

* Hoist uses to top of file

* Conditionally disable replay-slot-end-to-end-stats

* Remove the misleading fairly balanced text

…ry (anza-xyz#1370)

* Rename ComputeBudget::max_invoke_stack_height to max_instruction_stack_depth

The new name is consistent with the existing
ComputeBudget::max_instruction_trace_length.

Also expose compute_budget:MAX_INSTRUCTION_DEPTH.

* bpf_loader: use an explicit thread-local pool for stack and heap memory

Use a fixed thread-local pool to hold stack and heap memory. This
mitigates the long standing issue of jemalloc causing TLB shootdowns to
serve such frequent large allocations.

Because we need 1 stack and 1 heap region per instruction, and the
current max instruction nesting is hardcoded to 5, the pre-allocated
size is (MAX_STACK + MAX_HEAP) * 5 * NUM_THREADS. With the current
limits that's about 2.5MB per thread. Note that this is memory that
would eventually get allocated anyway, we're just pre-allocating it now.

* programs/sbf: add test for stack/heap zeroing

Add TEST_STACK_HEAP_ZEROED which tests that stack and heap regions are
zeroed across reuse from the memory pool.

The bank-hash command in ledger-tool was recently deprecated. However,
the command is used by some of the scripts that coordinate starting up
a fresh cluster. So, the deprecation of bank-hash broke those scripts.

This change fixes the scripts by doing the following:
- Makes --print-bank-hash support --output json
- Updates scripts to install jq on provisioned nodes
- Update remote-node.sh to parse the bank hash from json using jq

* use bytemuck_derive 1.7.0 explicitly in sdk and program

* explicitly activate the derive feature of bytemuck in zk-sdk

* Replaces the rust build script by a makefile

…ocation (anza-xyz#1759)

…#1636)

* Fix SyscallLogPubkey doc comment

* Update logging.rs

* refactor: reuse compute budget limits

* fix tests

There is often a desire to examine/replay/etc older blocks. If the
blocks are recent enough, they can be pulled from an actively running
node. Otherwise, the blocks must be pulled down from warehouse node
archives. These archives are uploaded on a per-epoch basis so they are
quite large, and can take multiple hours to download and decompress.

With the addition of Entry data to BigTable, blocks can be recreated
from BigTable data. Namely, we can recreate the Entries with proper PoH
and transaction data. We can then shred them such that they are the
same format as blocks that are produced from the cluster.

This change introduces a new command that will read BigTable data and
insert shreds into a local Blockstore. The new command is:
  $ agave-ledger-tool bigtable shreds ...

Several important notes about the change:
- Shred for some slot S will not be signed by the actual leader for
  slot S. Instead, shreds will be signed with a "dummy" keypair. The
  shred signatures does not affect the ability to replay the block.
- Entry PoH data does not go back to genesis in BigTable. This data
  could be extracted and uploaded from the existing rocksdb archives;
  however, that work is not planned as far as I know. --allow-mock-poh
  can be passed to generate filler PoH data. Blocks created with this
  flag are replayable by passing --skip-poh-verify to ledger-tool.
- A snapshot will be unpacked to determine items such as the shred
  version, tick hash rate and ticks per slot. This snapshot must be in
  the same epoch as the requested slots

* docs: add ciphertext validity proof docs

* update docs link to anza's

* remove extra space at the end of line

…ytemuck (anza-xyz#1799)

* deps: Use bytemuck_derive explicitly

* Missed a couple in zk-token-sdk

* Fix last few bits

* Fixup a re-export

* Use re-exports properly

* Improve SchedulerStatus code and test as follow-up

* Don't use wait_timeout_while with magic number

* SVM: hoist `program_modification_slot` up from bank

* SVM: add `check_program_modification_slot` to processing config

* SVM: hoist `program_match_criteria` up from bank

* SVM: drop `get_program_match_critera` from callbacks

Verify elf locally for write buffer command

* add stats for ancient bytes_from_smallest_storages

* fix else if

* remove totally unrelated demo code. ugh.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

) (anza-xyz#1843)

chore: add dcou to apply_votes_to_tower (anza-xyz#1831)

* add dcou to apply_votes_to_tower

* cargo sort

* fix fmt

(cherry picked from commit 66bdefd)

Co-authored-by: Yihau Chen <[email protected]>

…rgo.toml (backport of anza-xyz#1828) (anza-xyz#1844)

chore: remove publish=false from transaction-metrics-tracker/Cargo.toml (anza-xyz#1828)

(cherry picked from commit ac63c0a)

Co-authored-by: Yihau Chen <[email protected]>

…nza-xyz#1846)

chore: publish solana-tps-client (anza-xyz#1845)

(cherry picked from commit 7584773)

Co-authored-by: Yihau Chen <[email protected]>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…anza-xyz#1864) (anza-xyz#1868)

Add partitioned epoch rewards to changelog for v2.0 (anza-xyz#1864)

Add partitioned epoch rewards to changelog

(cherry picked from commit f0641ef)

Co-authored-by: Tyera <[email protected]>

…yz#1872) (anza-xyz#1880)

spl: Upgrade all crates to v2-only versions (anza-xyz#1872)

(cherry picked from commit 6aee845)

Co-authored-by: Jon C <[email protected]>

… (backport of anza-xyz#1677) (anza-xyz#1881)

Use num_partitions to find specific stake rewards in partitions (anza-xyz#1677)

* Add helper to find and filter rewards from a slot

* Check feature enabled for desired epoch

* Refactor existing rewards code to support vote-rewards after activation

* Append stake rewards from partitions

* Remove feature deactivation from TestValidator

* Improve comments

* Add comment about retaining feature activation slot logic

* Add custom error and use in getInflationReward

* Review nit

(cherry picked from commit 0496b06)

Co-authored-by: Tyera <[email protected]>

anza-xyz#1668) (anza-xyz#1874)

Make unified scheduler opt-in for block verification (anza-xyz#1668)

* Enable unified scheduler for block verification

* Revert making unified scheduler enabled by default

(cherry picked from commit 40508cd)

Co-authored-by: Ryo Onodera <[email protected]>

…yz#1809) (anza-xyz#1886)

* Remove support for deprecated rpc endpoints (anza-xyz#1809)

* Remove rpc_obsolete_v1_7

* Remove rpc_deprecated_v1_7

* Remove rpc_deprecated_v1_9

* Add CHANGELOG entry

* Add another CHANGELOG entry

(cherry picked from commit 83527d9)

# Conflicts:
#	CHANGELOG.md

* Fix conflict

---------

Co-authored-by: Tyera <[email protected]>

…gelog (backport of anza-xyz#1878) (anza-xyz#1879)

Add --block-verification-method=unified-scheduler for v2.0 changelog (anza-xyz#1878)

(cherry picked from commit 2b88299)

# Conflicts:
#	CHANGELOG.md

Co-authored-by: Ryo Onodera <[email protected]>

…anza-xyz#1861) (anza-xyz#1854)

* Avoid unneeded start_session() when spawning (anza-xyz#1815)

* Avoid unneeded start_session() when spawning

* Add comments

(cherry picked from commit 40a9851)

* Apply cosmetic changes to unified scheduler (anza-xyz#1861)

* Apply cosmetic changes to unified scheduler

* Use first instead of old-fashioned firstly

Co-authored-by: Andrew Fitzgerald <[email protected]>

---------

Co-authored-by: Andrew Fitzgerald <[email protected]>

---------

Co-authored-by: Ryo Onodera <[email protected]>
Co-authored-by: Andrew Fitzgerald <[email protected]>

…anza-xyz#1910)

Add since field to deprecation note (anza-xyz#1905)

Add  to deprecation note

(cherry picked from commit 70254b1)

Co-authored-by: Tyera <[email protected]>

…yz#1917)

vote: remove deprecated ixs (anza-xyz#1906)

(cherry picked from commit e42e7fd)

Co-authored-by: Ashwin Sekar <[email protected]>

…#1882) (anza-xyz#1907)

Deprecate --rocksdb-shred-compaction fifo (anza-xyz#1882)

The fifo compaction option was originally added to mitigate write
stalls that were occurring with level compaction. Since fifo was
introduced, the level compaction implementation has been optimized
to reduce I/O amplification. With these improvements, level and fifo
are comparable and sticking with level only simplifies things.

For now, only a deprecation warning will be printed. In the next
release branch (2.1.0), specifying fifo will be an error

(cherry picked from commit e15e235)

Co-authored-by: steviez <[email protected]>

…nza-xyz#1888) (anza-xyz#1900)

* Refactor and additional metrics for cost tracking (anza-xyz#1888)

* Refactor and add metrics:
- Combine remove_* and update_* functions to reduce locking on cost-tracker and iteration.
- Add method to calculate executed transaction cost by directly using actual execution cost and loaded accounts size;
- Wireup histogram to report loaded accounts size;
- Report time of block limits checking;
- Move account counters from ExecuteDetailsTimings to ExecuteAccountsDetails;

* Move committed transactions adjustment into its own function

(cherry picked from commit c3fadac)

* rename cost_tracker.account_data_size to better describe its purpose is to tracker per-block new account allocation

---------

Co-authored-by: Tao Zhu <[email protected]>
Co-authored-by: Tao Zhu <[email protected]>

anza-xyz#1909)

Remove deprecated SyncClient methods (anza-xyz#1902)

* Remove deprecated methods from SyncClient trait

* Add line to changelog

(cherry picked from commit 3d3faf5)

Co-authored-by: Tyera <[email protected]>

…port of anza-xyz#1888) (anza-xyz#1900) (anza-xyz#1937)

Revert "v2.0: Refactor and additional metrics for cost tracking (backport of anza-xyz#1888) (anza-xyz#1900)"

This reverts commit 0aef62e.

…anza-xyz#1922)

Remove deprecated RpcClient methods (anza-xyz#1899)

* Remove deprecated RpcClient methods corresponding to anza-xyz#1809

* Remove internal handling for pre-v1.9.0 servers

* Remove unused import of deprecated_config module

* Remove request mapping (pre-v1.7.0)

* Remove dangling allow-deprecated tag

* Add line to changelog

* Remove deprecated RpcRequest variants

* Remove reprecated rpc-client-nonce-utils methods

* Remove deprecated TestValidator method

* Remove deprecated solana fees command

(cherry picked from commit 51af772)

Co-authored-by: Tyera <[email protected]>

…#1895) (anza-xyz#1925)

Deprecate RpcClient::get_stake_activation (anza-xyz#1895)

* Deprecate RpcClient::get_stake_activation

* Fixup redelegation test

(cherry picked from commit 114041c)

Co-authored-by: Tyera <[email protected]>

) (anza-xyz#1926)

Remove rpc endpoint deprecated in v1.18 (anza-xyz#1897)

* Remove rpc_deprecated_v1_18

* Update changelog

* Fixup redelegation test

(cherry picked from commit 99b2d5a)

Co-authored-by: Tyera <[email protected]>

…za-xyz#1957)

Remove deprecated_config module (anza-xyz#1951)

(cherry picked from commit 00ee0ac)

Co-authored-by: Tyera <[email protected]>

…ons (backport of anza-xyz#1931) (anza-xyz#1941)

gossip: do not allow duplicate proofs for incorrect shred versions (anza-xyz#1931)

* gossip: do not allow duplicate proofs for incorrect shred versions

* pr feedback: refactor test function to take shred_version

(cherry picked from commit 69ea21e)

Co-authored-by: Ashwin Sekar <[email protected]>

…z#1945) (anza-xyz#1961)

ci: skip spl test when version is too high (anza-xyz#1945)

(cherry picked from commit 4d9d27f)

Co-authored-by: Yihau Chen <[email protected]>