chainHead: Allow methods to be called from within a single connection context and limit connections #3481
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
lexnv
added
A1-insubstantial
niklasad1
reviewed
Feb 26, 2024
| let _ = tx.send(result); | ||
| }; | ||
|
|
||
| executor.spawn_blocking("substrate-rpc-subscription", Some("rpc"), blocking_fut.boxed()); |
There was a problem hiding this comment.
yeah right, spawn_blocking doesn't return a handle for the future that's why you need the extra oneshot channel
niklasad1
reviewed
Feb 26, 2024
| /// Limit the RPC functionality to a single connection. | ||
| #[derive(Default, Clone)] | ||
| pub struct RpcConnections { | ||
| data: Arc<Mutex<HashMap<ConnectionId, HashSet<String>>>>, |
There was a problem hiding this comment.
could also be Arc<Mutex<HashSet<(ConnectionId, SubscriptionId)>>> I guess but perhaps you have plan for it to more generic
There was a problem hiding this comment.
mmm since ConnectionId is just a small u64, I'd just combine to Arc<Mutex<HashSet<(ConnectionId, SubscriptionId)>>> and that would be more efficient & slightly simpler code below, and probably small or no real extra space cost :)
niklasad1
reviewed
Feb 26, 2024
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
jsdw
reviewed
Mar 15, 2024
niklasad1
reviewed
Mar 18, 2024
niklasad1
reviewed
Mar 18, 2024
| @@ -110,6 +110,8 @@ pub struct ChainHead<BE: Backend<Block>, Block: BlockT, Client> { | |||
| /// The maximum number of items reported by the `chainHead_storage` before | |||
| /// pagination is required. | |||
| operation_max_storage_items: usize, | |||
| /// Limit the RPC functionality to a single connection. | |||
There was a problem hiding this comment.
I would re-phrase this, I think you are trying to say that each connection has its own state and not that the RPC functionality is limited to a single connection?
This connection state is then used to determine whether a certain call is permitted on a particular connection such as "unstable_storage calls" or something like that...
There was a problem hiding this comment.
I've rephrased the documentation a bit, let me know if it makes more sense now 🙏
niklasad1
reviewed
Mar 18, 2024
niklasad1
reviewed
Mar 18, 2024
niklasad1
reviewed
Mar 18, 2024
niklasad1
reviewed
Mar 18, 2024
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
jsdw
reviewed
Mar 26, 2024
substrate/client/rpc-spec-v2/src/chain_head/subscription/mod.rs
Outdated
Show resolved
Hide resolved
jsdw
reviewed
Mar 26, 2024
jsdw
reviewed
Mar 26, 2024
jsdw
reviewed
Mar 26, 2024
| /// For transactionBroadcast, this represents the operation ID. | ||
| /// For transaction, this is empty and the number of active calls is tracked by | ||
| /// [`Self::num_identifiers`]. | ||
| identifiers: HashSet<String>, |
There was a problem hiding this comment.
Am I right to assume that we are not worries about subscription IDs and operation IDs colliding at all in our implementation here?
There was a problem hiding this comment.
Yep, IIRC, we'll either use this for chainHead with subscriptions ID only; or for the tx with operation ID only
jsdw
reviewed
Mar 26, 2024
| let mut data = self.data.lock(); | ||
|
|
||
| let entry = data.entry(connection_id).or_insert_with(ConnectionData::default); | ||
| entry.num_identifiers = entry.num_identifiers.saturating_sub(1); |
There was a problem hiding this comment.
Do we ever clear out entries completely when the number goes to 0? Else we might eventually leak memory
jsdw
reviewed
Mar 26, 2024
| connection_data.num_identifiers = connection_data.num_identifiers.saturating_sub(1); | ||
|
|
||
| if connection_data.num_identifiers == 0 { | ||
| data.remove(&connection_id); |
There was a problem hiding this comment.
Ah, I guess this is whewre we clean up :)
jsdw
approved these changes
Mar 26, 2024
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Ank4n
pushed a commit
that referenced
this pull request
Apr 9, 2024
… context and limit connections (#3481) This PR ensures that the chainHead RPC class can be called only from within the same connection context. The chainHead methods are now registered as raw methods. - paritytech/jsonrpsee#1297 The concept of raw methods is introduced in jsonrpsee, which is an async method that exposes the connection ID: The raw method doesn't have the concept of a blocking method. Previously blocking methods are now spawning a blocking task to handle their blocking (ie DB) access. We spawn the same number of tasks as before, however we do that explicitly. Another approach would be implementing a RPC middleware that captures and decodes the method parameters: - #3343 However, that approach is prone to errors since the methods are hardcoded by name. Performace is affected by the double deserialization that needs to happen to extract the subscription ID we'd like to limit. Once from the middleware, and once from the methods itself. This PR paves the way to implement the chainHead connection limiter: - #1505 Registering tokens (subscription ID / operation ID) on the `RpcConnections` could be extended to return an error when the maximum number of operations is reached. While at it, have added an integration-test to ensure that chainHead methods can be called from within the same connection context. Before this is merged, a new JsonRPC release should be made to expose the `raw-methods`: - [x] Use jsonrpsee from crates io (blocked by: paritytech/jsonrpsee#1297) Closes: #3207 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Niklas Adolfsson <[email protected]>
dharjeezy
pushed a commit
to dharjeezy/polkadot-sdk
that referenced
this pull request
Apr 9, 2024
… context and limit connections (paritytech#3481) This PR ensures that the chainHead RPC class can be called only from within the same connection context. The chainHead methods are now registered as raw methods. - paritytech/jsonrpsee#1297 The concept of raw methods is introduced in jsonrpsee, which is an async method that exposes the connection ID: The raw method doesn't have the concept of a blocking method. Previously blocking methods are now spawning a blocking task to handle their blocking (ie DB) access. We spawn the same number of tasks as before, however we do that explicitly. Another approach would be implementing a RPC middleware that captures and decodes the method parameters: - paritytech#3343 However, that approach is prone to errors since the methods are hardcoded by name. Performace is affected by the double deserialization that needs to happen to extract the subscription ID we'd like to limit. Once from the middleware, and once from the methods itself. This PR paves the way to implement the chainHead connection limiter: - paritytech#1505 Registering tokens (subscription ID / operation ID) on the `RpcConnections` could be extended to return an error when the maximum number of operations is reached. While at it, have added an integration-test to ensure that chainHead methods can be called from within the same connection context. Before this is merged, a new JsonRPC release should be made to expose the `raw-methods`: - [x] Use jsonrpsee from crates io (blocked by: paritytech/jsonrpsee#1297) Closes: paritytech#3207 cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: Niklas Adolfsson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR ensures that the chainHead RPC class can be called only from within the same connection context.
The chainHead methods are now registered as raw methods.
The concept of raw methods is introduced in jsonrpsee, which is an async method that exposes the connection ID:
The raw method doesn't have the concept of a blocking method. Previously blocking methods are now spawning a blocking task to handle their blocking (ie DB) access. We spawn the same number of tasks as before, however we do that explicitly.
Another approach would be implementing a RPC middleware that captures and decodes the method parameters:
However, that approach is prone to errors since the methods are hardcoded by name. Performace is affected by the double deserialization that needs to happen to extract the subscription ID we'd like to limit. Once from the middleware, and once from the methods itself.
This PR paves the way to implement the chainHead connection limiter:
Registering tokens (subscription ID / operation ID) on the
RpcConnectionscould be extended to return an error when the maximum number of operations is reached.While at it, have added an integration-test to ensure that chainHead methods can be called from within the same connection context.
Before this is merged, a new JsonRPC release should be made to expose the
raw-methods:Closes: #3207
cc @paritytech/subxt-team