TransactionSource: specify maximum number of transactions to be fetched #2182

acerone85 · 2024-09-09T18:49:07Z

Linked Issues/PRs

Part of #2114

Description

Add a new argument transactions_limit: u16 to the function TxSource::next().
Changes the logic of the block production to never exceed u16::MAX transactions (included FTI transactions and mint transactions) when selecting L2 transacitons. The number of transactions can still exceed u16::MAX when selecting transactions from L1. This is addressed in Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189.
Changes the implementation of all TxSource adapters to fetch a number of transactions below or equal to the transaction_limit.
Introduces a new WASM host function to peek the next transactions size for a number of transactions below a specified limit.

Breaks forward compatibility of the genesis transition function. See https://github.com/FuelLabs/fuel-core/blob/8eeae5d6730d34ae06390b7797478bb98fd07987/version-compatibility/forkless-upgrade/README.md

## TODO:

Check that requirements are met
Implementation for the WasmTxSource is missing (only the argument has been added)
Integration Tests

Checklist

[ x] Breaking changes are clearly marked as such in the PR description and changelog
New behavior is reflected in tests
The specification matches the implemented behavior (link update PR if changes are needed)

Before requesting review

I have reviewed the code myself
I have created follow-up issues caused by this PR and linked them here

After merging, notify other teams

[Add or remove entries as needed]

Rust SDK
Sway compiler
Platform documentation (for out-of-organization contributors, the person merging the PR will do this)
Someone else?

crates/services/executor/src/ports.rs

crates/services/txpool/src/txpool.rs

crates/services/upgradable-executor/wasm-executor/src/ext.rs

crates/services/upgradable-executor/src/instance.rs

xgreenx

LGTM!=)

xgreenx · 2024-09-18T07:59:27Z

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - [2151](https://github.com/FuelLabs/fuel-core/pull/2151): Added limitations on gas used during dry_run in API.
 - [2188](https://github.com/FuelLabs/fuel-core/pull/2188): Added the new variant `V2` for the `ConsensusParameters` which contains the new `block_transaction_size_limit` parameter.
 - [2163](https://github.com/FuelLabs/fuel-core/pull/2163): Added runnable task for fetching block committer data.
+- [2182](https://github.com/FuelLabs/fuel-core/pull/2151): Limit number of transactions that can be fetched via TxSource::next


After release we need to move it up=)

xgreenx · 2024-09-18T13:33:32Z

CHANGELOG.md

@@ -26,6 +27,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - [2158](https://github.com/FuelLabs/fuel-core/pull/2158): Log the public address of the signing key, if it is specified
 - [2188](https://github.com/FuelLabs/fuel-core/pull/2188): Upgraded the `fuel-vm` to `0.57.0`. More information in the [release](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.57.0).

+


Suggested change

xgreenx · 2024-09-18T13:35:15Z

crates/services/executor/src/executor.rs

+        // When processing l2 transactions, we must take into account transactions from the l1
+        // that have been included in the block already (stored in `data.tx_count`), as well
+        // as the final mint transaction.
+        let max_tx_count = u16::MAX.saturating_sub(1);


We can store this value in some constant=)

xgreenx · 2024-09-18T13:37:48Z

tests/tests/blocks.rs

+        assert_eq!(second_last_block.transactions.len(), 65_535);
+        assert_eq!(
+            last_block.transactions.len(),
+            (tx_count as usize - 65_534) + 1


Suggested change

(tx_count as usize - 65_534) + 1

(tx_count as usize - u16::MAX) + 2 /* mint transaction from two blocks */

xgreenx · 2024-09-18T13:37:55Z

tests/tests/blocks.rs

+            .await
+            .unwrap()
+            .expect("Last Block should be defined");
+        assert_eq!(second_last_block.transactions.len(), 65_535);


Suggested change

assert_eq!(second_last_block.transactions.len(), 65_535);

assert_eq!(second_last_block.transactions.len(), u16::MAX);

netrome

Nice stuff. Some minor thoughts from me.

tests/tests/blocks.rs

netrome

Nice!

xgreenx · 2024-09-19T13:06:21Z

crates/services/executor/src/executor.rs

+        // that have been included in the block already (stored in `data.tx_count`), as well
+        // as the final mint transaction.
+        let mut remaining_tx_count = MAX_TX_COUNT.saturating_sub(data.tx_count);
+
        let mut regular_tx_iter = l2_tx_source


If the next returns more data than you requested, you will not catch a situation in which you have more transitions than allowed. Unit test is passing since OnceTransactionsSource handles it. But in the case of another TxSource it may not work.

Good point. All TxSources implement next to never return more than remaining_tx_count transactions, but I have added a .take to avoid this scenario:

Two remarks:

The new behaviour of the function means that if we use a "bad" transaction source, transactions over the limit will be removed and not be included in a block. If we want to be completely safe we could have a new SafeTransactionSource trait with the same signature of TransactionsSource, and implement a struct CacheableTransactionSource<TxSource> which is implemented whenever TxSource: TransactionsSource,
and keeps transactions over the limit in memory.
Given that we control the implementations of TransactionsSource, this is probably an overkill.

The new change is not tested, I'd need to implement a BadTransactionSource and modify the commit_without_execute function to run with that transaction source. I can implement that if necessary, but I'd prefer to open a dedicated PR for it.

Yeah, BadTransactionSource sounds good=D

xgreenx · 2024-09-19T17:11:11Z

crates/services/executor/src/executor.rs

+        // that have been included in the block already (stored in `data.tx_count`), as well
+        // as the final mint transaction.
+        let mut remaining_tx_count = MAX_TX_COUNT.saturating_sub(data.tx_count);
+
        let mut regular_tx_iter = l2_tx_source


Yeah, BadTransactionSource sounds good=D

…ed (#2182) ## Linked Issues/PRs  Part of #2114 ## Description - Add a new argument `transactions_limit: u16` to the function `TxSource::next()`. - Changes the logic of the block production to never exceed `u16::MAX` transactions (included FTI transactions and mint transactions) when selecting L2 transacitons. The number of transactions can still exceed `u16::MAX` when selecting transactions from L1. This is addressed in #2189. - Changes the implementation of all TxSource adapters to fetch a number of transactions below or equal to the `transaction_limit`. - Introduces a new WASM host function to peek the next transactions size for a number of transactions below a specified limit. Breaks forward compatibility of the genesis transition function. See https://github.com/FuelLabs/fuel-core/blob/8eeae5d6730d34ae06390b7797478bb98fd07987/version-compatibility/forkless-upgrade/README.md ## TODO: - [x] Check that requirements are met - [x] Implementation for the WasmTxSource is missing (only the argument has been added) - [ ] Integration Tests ## Checklist - [ x] Breaking changes are clearly marked as such in the PR description and changelog - [x] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [x] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else?

@rymnc

## Version v0.37.0 ### Added - [1609](#1609): Add DA compression support. Compressed blocks are stored in the offchain database when blocks are produced, and can be fetched using the GraphQL API. - [2290](#2290): Added a new CLI argument `--graphql-max-directives`. The default value is `10`. - [2195](#2195): Added enforcement of the limit on the size of the L2 transactions per block according to the `block_transaction_size_limit` parameter. - [2131](#2131): Add flow in TxPool in order to ask to newly connected peers to share their transaction pool - [2182](#2151): Limit number of transactions that can be fetched via TxSource::next - [2189](#2151): Select next DA height to never include more than u16::MAX -1 transactions from L1. - [2162](#2162): Pool structure with dependencies, etc.. for the next transaction pool module. Also adds insertion/verification process in PoolV2 and tests refactoring - [2265](#2265): Integrate Block Committer API for DA Block Costs. - [2280](#2280): Allow comma separated relayer addresses in cli - [2299](#2299): Support blobs in the predicates. - [2300](#2300): Added new function to `fuel-core-client` for checking whether a blob exists. ### Changed #### Breaking - [2299](#2299): Anyone who wants to participate in the transaction broadcasting via p2p must upgrade to support new predicates on the TxPool level. - [2299](#2299): Upgraded `fuel-vm` to `0.58.0`. More information in the [release](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.58.0). - [2276](#2276): Changed how complexity for blocks is calculated. The default complexity now is 80_000. All queries that somehow touch the block header now are more expensive. - [2290](#2290): Added a new GraphQL limit on number of `directives`. The default value is `10`. - [2206](#2206): Use timestamp of last block when dry running transactions. - [2153](#2153): Updated default gas costs for the local testnet configuration to match `fuel-core 0.35.0`. ## What's Changed * fix: use core-test.fuellabs.net for dnsaddr resolution by @rymnc in #2214 * Removed state transition bytecode from the local testnet by @xgreenx in #2215 * Send whole transaction pool upon subscription to gossip by @AurelienFT in #2131 * Update default gas costs based on 0.35.0 benchmarks by @xgreenx in #2153 * feat: Use timestamp of last block when dry running transactions by @netrome in #2206 * fix(dnsaddr_resolution): use fqdn separator to prevent suffixing by dns resolvers by @rymnc in #2222 * TransactionSource: specify maximum number of transactions to be fetched by @acerone85 in #2182 * Implement worst case scenario for price algorithm v1 by @rafal-ch in #2219 * chore(gas_price_service): define port for L2 data by @rymnc in #2224 * Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 by @acerone85 in #2189 * Weekly `cargo update` by @github-actions in #2236 * Use fees to calculate DA reward and avoid issues with Gwei/Wei conversions by @MitchTurner in #2229 * Protect against passing `i128::MIN` to `abs()` which causes overflow by @rafal-ch in #2241 * Acquire `da_finalization_period` from the command line by @rafal-ch in #2240 * Executor: test Tx_count limit with incorrect tx source by @acerone85 in #2242 * Minor updates to docs + a few typos fixed by @rafal-ch in #2250 * chore(gas_price_service): move algorithm_updater to fuel-core-gas-price-service by @rymnc in #2246 * Use single heavy input in the `transaction_throughput.rs` benchmarks by @xgreenx in #2205 * Enforce the block size limit by @rafal-ch in #2195 * feat: build ARM and AMD in parallel by @mchristopher in #2130 * Weekly `cargo update` by @github-actions in #2268 * chore(gas_price_service): split into v0 and v1 and squash FuelGasPriceUpdater type into GasPriceService by @rymnc in #2256 * feat(gas_price_service): update block committer da source with established contract by @rymnc in #2265 * Use bytes from `unrecorded_blocks` rather from the block from DA by @MitchTurner in #2252 * TxPool v2 General architecture by @AurelienFT in #2162 * Add value delimiter and tests args by @AurelienFT in #2280 * fix(da_block_costs): remove Arc<Mutex<>> on shared_state and expose channel by @rymnc in #2278 * fix(combined_database): syncing auxiliary databases on startup with custom behaviour by @rymnc in #2272 * fix: Manually encode Authorization header for eventsource_client by @Br1ght0ne in #2284 * Address `async-graphql` vulnerability by @MitchTurner in #2290 * Update the WASM compatibility tests for `0.36` release by @rafal-ch in #2271 * DA compression by @Dentosal in #1609 * Use different port for every version compatibility test by @rafal-ch in #2301 * Fix block query complexity by @xgreenx in #2297 * Support blobs in predicates by @Voxelot in #2299 **Full Changelog**: v0.36.0...v0.37.0

acerone85 requested review from xgreenx, Dentosal and MitchTurner as code owners September 9, 2024 18:49

acerone85 force-pushed the tx_per_block_do_not_overflow branch from 53d8b5a to c6bd7d3 Compare September 9, 2024 18:49

acerone85 marked this pull request as draft September 9, 2024 18:50

acerone85 commented Sep 10, 2024

View reviewed changes

crates/services/executor/src/ports.rs Outdated Show resolved Hide resolved

acerone85 commented Sep 10, 2024

View reviewed changes

crates/services/txpool/src/txpool.rs Outdated Show resolved Hide resolved

acerone85 commented Sep 10, 2024

View reviewed changes

crates/services/upgradable-executor/wasm-executor/src/ext.rs Outdated Show resolved Hide resolved

acerone85 force-pushed the tx_per_block_do_not_overflow branch 3 times, most recently from 70fa40d to 8075207 Compare September 12, 2024 11:01

acerone85 self-assigned this Sep 12, 2024

acerone85 commented Sep 12, 2024

View reviewed changes

crates/services/upgradable-executor/src/instance.rs Outdated Show resolved Hide resolved

acerone85 force-pushed the tx_per_block_do_not_overflow branch 4 times, most recently from f5803fa to 6842132 Compare September 15, 2024 20:10

acerone85 changed the base branch from master to feature/wasm-interface-breaking-changes September 15, 2024 20:12

acerone85 force-pushed the tx_per_block_do_not_overflow branch from 6842132 to cc331dc Compare September 15, 2024 20:14

acerone85 marked this pull request as ready for review September 16, 2024 09:32

acerone85 force-pushed the tx_per_block_do_not_overflow branch 5 times, most recently from 367ee4f to b1e887f Compare September 17, 2024 14:42

Base automatically changed from feature/wasm-interface-breaking-changes to master September 17, 2024 14:50

acerone85 force-pushed the tx_per_block_do_not_overflow branch from b1e887f to f498dce Compare September 17, 2024 15:35

xgreenx reviewed Sep 18, 2024

View reviewed changes

netrome previously approved these changes Sep 19, 2024

View reviewed changes

tests/tests/blocks.rs Outdated Show resolved Hide resolved

tests/tests/blocks.rs Show resolved Hide resolved

tests/tests/blocks.rs Outdated Show resolved Hide resolved

tests/tests/blocks.rs Show resolved Hide resolved

acerone85 dismissed netrome’s stale review via da3541a September 19, 2024 09:32

netrome previously approved these changes Sep 19, 2024

View reviewed changes

acerone85 added 4 commits September 19, 2024 12:21

Modify TxSource::next to handle the maximum number of transactions

a2cb79a

Integration test - 66K transactions are split in two blocks

7895dcd

Address code reviews

494d792

Cosmetic changes

bd67a39

acerone85 dismissed netrome’s stale review via bd67a39 September 19, 2024 11:21

acerone85 force-pushed the tx_per_block_do_not_overflow branch from 7fc88a5 to bd67a39 Compare September 19, 2024 11:21

netrome previously approved these changes Sep 19, 2024

View reviewed changes

Merge branch 'master' into tx_per_block_do_not_overflow

1caa9bc

xgreenx reviewed Sep 19, 2024

View reviewed changes

Do not trust TxSource::next

592dfc6

acerone85 dismissed netrome’s stale review via 592dfc6 September 19, 2024 16:07

xgreenx approved these changes Sep 19, 2024

View reviewed changes

acerone85 mentioned this pull request Sep 19, 2024

Test that executor is resilient against bad transaction sources #2225

Closed

netrome approved these changes Sep 19, 2024

View reviewed changes

acerone85 merged commit 79ca0d0 into master Sep 19, 2024
36 checks passed

acerone85 deleted the tx_per_block_do_not_overflow branch September 19, 2024 17:35

rafal-ch mentioned this pull request Sep 20, 2024

Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189

Merged

9 tasks

acerone85 mentioned this pull request Sep 20, 2024

Introduce Limiter abstraction for Executor parameters #2234

Closed

xgreenx mentioned this pull request Oct 5, 2024

Release v0.37.0 #2302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TransactionSource: specify maximum number of transactions to be fetched #2182

TransactionSource: specify maximum number of transactions to be fetched #2182

acerone85 commented Sep 9, 2024 •

edited

Loading

xgreenx left a comment

xgreenx Sep 18, 2024

xgreenx Sep 18, 2024

xgreenx Sep 18, 2024

xgreenx Sep 18, 2024

xgreenx Sep 18, 2024

netrome left a comment

netrome left a comment

xgreenx Sep 19, 2024

acerone85 Sep 19, 2024

xgreenx Sep 19, 2024

xgreenx Sep 19, 2024

		@@ -26,6 +27,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
		- [2158](https://github.com/FuelLabs/fuel-core/pull/2158): Log the public address of the signing key, if it is specified
		- [2188](https://github.com/FuelLabs/fuel-core/pull/2188): Upgraded the `fuel-vm` to `0.57.0`. More information in the [release](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.57.0).

	(tx_count as usize - 65_534) + 1
	(tx_count as usize - u16::MAX) + 2 /* mint transaction from two blocks */

	assert_eq!(second_last_block.transactions.len(), 65_535);
	assert_eq!(second_last_block.transactions.len(), u16::MAX);

TransactionSource: specify maximum number of transactions to be fetched #2182

TransactionSource: specify maximum number of transactions to be fetched #2182

Conversation

acerone85 commented Sep 9, 2024 • edited Loading

Linked Issues/PRs

Description

Checklist

Before requesting review

After merging, notify other teams

xgreenx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netrome left a comment

Choose a reason for hiding this comment

netrome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acerone85 commented Sep 9, 2024 •

edited

Loading