Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc v2: backpressure chainHead_v1_storage #5741

Merged
merged 37 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
7f32fdf
rpc v2: rely backpressure `Storage::query_iter`
niklasad1 Sep 9, 2024
c7d9a50
Update substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs
niklasad1 Sep 17, 2024
540daa6
Update substrate/client/rpc-spec-v2/src/chain_head/chain_head_storage.rs
niklasad1 Sep 17, 2024
423bc30
Update substrate/client/rpc-spec-v2/src/chain_head/tests.rs
niklasad1 Sep 17, 2024
ef62794
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Sep 17, 2024
7351206
cleanup
niklasad1 Sep 18, 2024
b963c8b
revert archive static limits
niklasad1 Sep 18, 2024
1cb17f3
cargo fmt
niklasad1 Sep 18, 2024
0c94a6b
add trait bound RawIter: Send
niklasad1 Sep 19, 2024
1612813
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Sep 19, 2024
583ea58
cargo fmt
niklasad1 Sep 19, 2024
486b246
remove unused import
niklasad1 Sep 19, 2024
67a37ce
Update substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs
niklasad1 Sep 19, 2024
05386e8
Update substrate/client/rpc-spec-v2/src/chain_head/subscription/inner.rs
niklasad1 Sep 19, 2024
dc17cc4
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Sep 24, 2024
ab88c05
address grumbles
niklasad1 Sep 24, 2024
accb056
add prdoc
niklasad1 Sep 24, 2024
7a7a75e
Update prdoc/pr_5741.prdoc
niklasad1 Sep 24, 2024
6fa42e4
prdoc fixes
niklasad1 Sep 24, 2024
08699f6
".git/.scripts/commands/fmt/fmt.sh"
Sep 24, 2024
9766eb3
bump tokio for permit api
niklasad1 Sep 25, 2024
af28077
simplify the code
niklasad1 Sep 26, 2024
8ccd391
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Sep 26, 2024
737e858
fix tests
niklasad1 Sep 27, 2024
797f1da
cargo fmt
niklasad1 Sep 27, 2024
2ceadfc
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Sep 27, 2024
381fcc0
".git/.scripts/commands/fmt/fmt.sh"
Sep 27, 2024
6263c83
Revert "".git/.scripts/commands/fmt/fmt.sh""
niklasad1 Sep 27, 2024
097a50e
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Sep 27, 2024
34393db
Update substrate/client/rpc-spec-v2/Cargo.toml
niklasad1 Sep 27, 2024
25051cd
Update prdoc/pr_5741.prdoc
niklasad1 Oct 1, 2024
4a8c5c3
Merge branch 'master' into na-fix-rpc-storage-iter
niklasad1 Oct 1, 2024
9152f9e
remove needless trait bounds
niklasad1 Oct 2, 2024
ec25f00
Merge remote-tracking branch 'origin/master' into na-fix-rpc-storage-…
niklasad1 Oct 2, 2024
bb251b0
remove more needless trait bounds
niklasad1 Oct 2, 2024
49e013c
remove more needless trait bounds again
niklasad1 Oct 2, 2024
4a70b0e
Merge branch 'master' into na-fix-rpc-storage-iter
niklasad1 Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -1322,7 +1322,7 @@ tikv-jemalloc-ctl = { version = "0.5.0" }
tikv-jemallocator = { version = "0.5.0" }
time = { version = "0.3" }
tiny-keccak = { version = "2.0.2" }
tokio = { version = "1.37.0", default-features = false }
tokio = { version = "1.40.0", default-features = false }
tokio-retry = { version = "0.3.0" }
tokio-stream = { version = "0.1.14" }
tokio-test = { version = "0.4.2" }
Expand Down
24 changes: 24 additions & 0 deletions prdoc/pr_5741.prdoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Schema: Polkadot SDK PRDoc Schema (prdoc) v1.0.0
# See doc at https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/prdoc/schema_user.json

title: make RPC endpoint `chainHead_v1_storage` faster

doc:
- audience: Node Operator
description: |
The RPC endpoint `chainHead_v1_storage` has removed the static limit for the number
of storage queries instead it has removed all static limits and relies on backpressure.
This should improve the throughput for bigger storage queries significantly.
niklasad1 marked this conversation as resolved.
Show resolved Hide resolved

Benchmarks using subxt on localhost:
- Iterate over 10 accounts on westend-dev -> ~2-3x faster
- Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster
- Fetch 1024 descendant values -> ~500x faster

crates:
- name: sc-rpc-spec-v2
bump: major
- name: sc-rpc-server
bump: patch
niklasad1 marked this conversation as resolved.
Show resolved Hide resolved
- name: sc-service
bump: major
5 changes: 3 additions & 2 deletions substrate/client/rpc-servers/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -255,8 +255,9 @@ where
),
};

let rpc_middleware =
RpcServiceBuilder::new().option_layer(middleware_layer.clone());
let rpc_middleware = RpcServiceBuilder::new()
.rpc_logger(1024)
.option_layer(middleware_layer.clone());
let mut svc = service_builder
.set_rpc_middleware(rpc_middleware)
.build(methods, stop_handle);
Expand Down
3 changes: 2 additions & 1 deletion substrate/client/rpc-spec-v2/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ sp-rpc = { workspace = true, default-features = true }
sp-blockchain = { workspace = true, default-features = true }
sp-version = { workspace = true, default-features = true }
sc-client-api = { workspace = true, default-features = true }
sc-utils = { workspace = true, default-features = true }
sc-rpc = { workspace = true, default-features = true }
codec = { workspace = true, default-features = true }
thiserror = { workspace = true }
Expand Down Expand Up @@ -56,6 +55,8 @@ sp-externalities = { workspace = true, default-features = true }
sp-maybe-compressed-blob = { workspace = true, default-features = true }
sc-block-builder = { workspace = true, default-features = true }
sc-service = { features = ["test-helpers"], workspace = true, default-features = true }
sc-rpc = { workspace = true, default-features = true, features = ["test-helpers"] }
assert_matches = { workspace = true }
pretty_assertions = { workspace = true }
sc-transaction-pool = { workspace = true, default-features = true }
sc-utils = { workspace = true, default-features = true }
1 change: 1 addition & 0 deletions substrate/client/rpc-spec-v2/src/archive/archive.rs
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ where
self.storage_max_descendant_responses,
self.storage_max_queried_items,
);

Ok(storage_client.handle_query(hash, items, child_trie))
}
}
6 changes: 3 additions & 3 deletions substrate/client/rpc-spec-v2/src/archive/archive_storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ impl<Client, Block, BE> ArchiveStorage<Client, Block, BE> {

impl<Client, Block, BE> ArchiveStorage<Client, Block, BE>
where
Block: BlockT + 'static,
BE: Backend<Block> + 'static,
Client: StorageProvider<Block, BE> + 'static,
Block: BlockT + Send + 'static,
BE: Backend<Block> + Send + 'static,
Client: StorageProvider<Block, BE> + Send + Sync + 'static,
{
/// Generate the response of the `archive_storage` method.
pub fn handle_query(
Expand Down
Loading
Loading