Cost Model to limit transactions which are not parallelizeable #16694

tao-stones · 2021-04-20T23:18:48Z

Problem

Needed by #5395 to limit number of transaction in a block.

Summary of Changes

added cost model that calculates each transaction's "executing cost"; if a system_transfer instruction is cost of '1', a BPF program maybe configured to have cost of 1_000.
pack transactions to comply with two limits:
2.1 the total cost of un-parallelizeable transactions (eg transactions need to lock and write to same account ) must be less than configured chain_max_cost;
2.2 total cost of a blob of transaction should be less than configured block_max_cost.

Fixes #

core/src/banking_stage.rs

tao-stones · 2021-04-20T23:22:58Z

core/src/cost_model.rs

+};
+use std::{collections::HashMap, str::FromStr};
+
+// TODO  revisit these hardcoded numbers, better get from mainnet log


below are just some made-up numbers, will be replace with stats collected from mainnet later

core/src/cost_model.rs

mvines · 2021-04-21T01:07:56Z

"[WIP] initial review"

Please give your PRs a helpful title for easier scanning 🙏🏼

tao-stones · 2021-04-21T02:29:42Z

"[WIP] initial review"

Please give your PRs a helpful title for easier scanning 🙏🏼

Oops, meant for a private review. In any case, always a good practice to supply a meaningful title and description. Thanks for pointing out

tao-stones · 2021-04-26T23:14:23Z

@sakridge Implemented your suggestion - making cost_model a filter when banking_stage gets transactions from packages. It dramatically simplified the code since cost model no longer needs to catch and organize transactions, instead it only needs to keep tracking cost for one block.
What is in right now is the final version of draft :D Can you review when you have chance?
next steps:
[] ran bench_tps to check if any performance change;
[] using mainnet data to refine the cost metrics.

codecov · 2021-04-27T01:06:30Z

Codecov Report

Merging #16694 (c64c529) into master (53c91ee) will increase coverage by 0.0%.
The diff coverage is 95.6%.

@@           Coverage Diff            @@
##           master   #16694    +/-   ##
========================================
  Coverage    82.6%    82.7%            
========================================
  Files         427      429     +2     
  Lines      119184   119731   +547     
========================================
+ Hits        98559    99049   +490     
- Misses      20625    20682    +57

core/src/cost_model.rs

tao-stones · 2021-05-10T04:10:13Z

@sakridge, this PR is force pushed again :D This time to include changes:

move cost_model into Bank to be its member, shareble between threads;
Bank engage cost_model during loand_and_execute_transaction, which is called by both banking_stage and replay_stage.
It returns a new error CostExceedsLimit when failed to add transaction due to cost model, and marks that transaction as retryable to be reprocess next round.
When Bank::freeze() is called, I added code to debug log cost model stats. However, running it as validator in mainnet for few hours during weekend, the freeze() function never seem to be called. Need to investigate that on Monday

tao-stones · 2021-05-14T15:37:13Z

Hardcoded cost model config based on data collected from mainnet ledger using ledger-tool. The idea is to set block cost limit much higher than what's in current ledger to allow rapid growth, while setting chain cost limit (eg cost per writable account) to close to current average to encourage adding parallelizable transactions in to bank.

core/src/banking_stage.rs

core/src/cost_model.rs

core/src/cost_tracker.rs

sakridge

lgtm

…a-labs#16694) * * Add following to banking_stage: 1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions. 2. CostTracker which is shared between threads, tracks transaction costs for each block. * replace hard coded program ID with id() calls * Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed. * Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table. * add test for cost_tracker atomically try_add operation, serves as safety guard for future changes * check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker; * bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations

solana-labs#16694)" This reverts commit b000d49.

* * Add following to banking_stage: 1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions. 2. CostTracker which is shared between threads, tracks transaction costs for each block. * replace hard coded program ID with id() calls * Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed. * Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table. * add test for cost_tracker atomically try_add operation, serves as safety guard for future changes * check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker; * bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations (cherry picked from commit b000d49) # Conflicts: # Cargo.lock # core/Cargo.toml # core/benches/banking_stage.rs # core/src/banking_stage.rs

…a-labs#16694) * * Add following to banking_stage: 1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions. 2. CostTracker which is shared between threads, tracks transaction costs for each block. * replace hard coded program ID with id() calls * Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed. * Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table. * add test for cost_tracker atomically try_add operation, serves as safety guard for future changes * check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker; * bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations

* Cost Model to limit transactions which are not parallelizeable (#16694) * * Add following to banking_stage: 1. CostModel as immutable ref shared between threads, to provide estimated cost for transactions. 2. CostTracker which is shared between threads, tracks transaction costs for each block. * replace hard coded program ID with id() calls * Add Account Access Cost as part of TransactionCost. Account Access cost are weighted differently between read and write, signed and non-signed. * Establish instruction_execution_cost_table, add function to update or insert instruction cost, unit tested. It is read-only for now; it allows Replay to insert realtime instruction execution costs to the table. * add test for cost_tracker atomically try_add operation, serves as safety guard for future changes * check cost against local copy of cost_tracker, return transactions that would exceed limit as unprocessed transaction to be buffered; only apply bank processed transactions cost to tracker; * bencher to new banking_stage with max cost limit to allow cost model being hit consistently during bench iterations * replay stage feed back program cost (#17731) * replay stage feeds back realtime per-program execution cost to cost model; * program cost execution table is initialized into empty table, no longer populated with hardcoded numbers; * changed cost unit to microsecond, using value collected from mainnet; * add ExecuteCostTable with fixed capacity for security concern, when its limit is reached, programs with old age AND less occurrence will be pushed out to make room for new programs. * investigate system performance test degradation (#17919) * Add stats and counter around cost model ops, mainly: - calculate transaction cost - check transaction can fit in a block - update block cost tracker after transactions are added to block - replay_stage to update/insert execution cost to table * Change mutex on cost_tracker to RwLock * removed cloning cost_tracker for local use, as the metrics show clone is very expensive. * acquire and hold locks for block of TXs, instead of acquire and release per transaction; * remove redundant would_fit check from cost_tracker update execution path * refactor cost checking with less frequent lock acquiring * avoid many Transaction_cost heap allocation when calculate cost, which is in the hot path - executed per transaction. * create hashmap with new_capacity to reduce runtime heap realloc. * code review changes: categorize stats, replace explicit drop calls, concisely initiate to default * address potential deadlock by acquiring locks one at time * Persist cost table to blockstore (#18123) * Add `ProgramCosts` Column Family to blockstore, implement LedgerColumn; add `delete_cf` to Rocks * Add ProgramCosts to compaction excluding list alone side with TransactionStatusIndex in one place: `excludes_from_compaction()` * Write cost table to blockstore after `replay_stage` replayed active banks; add stats to measure persist time * Deletes program from `ProgramCosts` in blockstore when they are removed from cost_table in memory * Only try to persist to blockstore when cost_table is changed. * Restore cost table during validator startup * Offload `cost_model` related operations from replay main thread to dedicated service thread, add channel to send execute_timings between these threads; * Move `cost_update_service` to its own module; replay_stage is now decoupled from cost_model. * log warning when channel send fails (#18391) * Aggregate cost_model into cost_tracker (#18374) * * aggregate cost_model into cost_tracker, decouple it from banking_stage to prevent accidental deadlock. * Simplified code, removed unused functions * review fixes * update ledger tool to restore cost table from blockstore (#18489) * update ledger tool to restore cost model from blockstore when compute-slot-cost * Move initialize_cost_table into cost_model, so the function can be tested and shared between validator and ledger-tool * refactor and simplify a test * manually fix merge conflicts * Per-program id timings (#17554) * more manual fixing * solve a merge conflict * featurize cost model * more merge fix * cost model uses compute_unit to replace microsecond as cost unit (#18934) * Reject blocks for costs above the max block cost (#18994) * Update block max cost limit to fix performance regession (#19276) * replace function with const var for better readability (#19285) * Add few more metrics data points (#19624) * periodically report sigverify_stage stats (#19674) * manual merge * cost model nits (#18528) * Accumulate consumed units (#18714) * tx wide compute budget (#18631) * more manual merge * ignore zerorize drop security * - update const cost values with data collected by #19627 - update cost calculation to closely proposed fee schedule #16984 * add transaction cost histogram metrics (#20350) * rebase to 1.7.15 * add tx count and thread id to stats (#20451) each stat reports and resets when slot changes * remove cost_model feature_set * ignore vote transactions from cost model Co-authored-by: sakridge <[email protected]> Co-authored-by: Jeff Biseda <[email protected]> Co-authored-by: Jack May <[email protected]>

tao-stones commented Apr 20, 2021

View reviewed changes

core/src/banking_stage.rs Outdated Show resolved Hide resolved

tao-stones commented Apr 20, 2021

View reviewed changes

core/src/cost_model.rs Show resolved Hide resolved

tao-stones commented Apr 20, 2021

View reviewed changes

core/src/cost_model.rs Outdated Show resolved Hide resolved

tao-stones commented Apr 20, 2021

View reviewed changes

core/src/cost_model.rs Outdated Show resolved Hide resolved

tao-stones changed the title ~~[WIP] initial review~~ Cost Model to limit transactions which are not parallelizeable [wip, for initial review] Apr 21, 2021

tao-stones requested a review from sakridge April 21, 2021 02:31

tao-stones self-assigned this Apr 21, 2021

tao-stones linked an issue Apr 22, 2021 that may be closed by this pull request

Transaction cost model for bankless leader #16641

Closed

tao-stones requested a review from jeffwashington April 26, 2021 23:14

tao-stones marked this pull request as ready for review April 27, 2021 21:30

tao-stones marked this pull request as draft April 30, 2021 15:28

sakridge reviewed May 3, 2021

View reviewed changes

core/src/cost_model.rs Outdated Show resolved Hide resolved

This was referenced May 3, 2021

Comprehensive fees proposal #16984

Closed

Add proposal for transactions v2 and address map program #17103

Merged

tao-stones force-pushed the 16641-cost-model branch from fa1af83 to 63a2339 Compare May 10, 2021 03:54

tao-stones force-pushed the 16641-cost-model branch 3 times, most recently from 3ee2296 to 8a3dae3 Compare May 10, 2021 23:00

mvines mentioned this pull request May 13, 2021

Add subcommand "compute-slot-cost" to ledger tool #17181

Merged

tao-stones force-pushed the 16641-cost-model branch from 8a3dae3 to 8048612 Compare May 14, 2021 15:31

tao-stones marked this pull request as ready for review May 14, 2021 15:37

tao-stones requested a review from sakridge May 14, 2021 15:37

tao-stones changed the title ~~Cost Model to limit transactions which are not parallelizeable [wip, for initial review]~~ Cost Model to limit transactions which are not parallelizeable May 16, 2021

tao-stones force-pushed the 16641-cost-model branch from bf3e06b to 1e44c90 Compare May 28, 2021 05:29