fix(websocket client): drop subscriptions that can't keep up with the internal buffer size #166

niklasad1 · 2020-11-27T12:50:20Z

Closes #143

The major change in this PR it's using Sender::try_send for the internal message passing channel for each subscription within the client which will drop the channel if the buffer gets full.

Thus, I had to change the API a little bit for Subscriptions to behave as Streams in Rust that now returns None if the channel is full or is disconnected instead of waiting for successful responses.

The capacity for the internal buffers are now user-configurable

NOTE

This PR doesn't use Sender::try_send for the tasks that depend on remote responses (method calls, notifications, and register "new" subscriptions). That's pointless because the sender is "cloned" because it takes immutable ref, (~~this we could maybe change to &mut self~~). Thus Sender::clone increases the buffer size by one and it may diverge significantly if the many requests, notifications are performed "concurrently".

TLDR; It shouldn't deadlock for those but it might allow more buffer size than "configured"

This commit changes the behavior in the `WebSocket Client` where each subscription channel is used in a non-blocking matter until it is determined as full or disconnected. When that occurs the channel is simply dropped and when the user `poll` the subscription it will return all sent subscriptions before it was and terminate (return None) once it's polled one last time. Similarly as `Streams` works in Rust. It also adds configuration for the `WebSocket Client` to configure capacity for the different internal channels to avoid filling the buffers when it's not expected.

src/client/ws/client.rs

niklasad1 · 2020-11-27T12:54:18Z

src/client/ws/client.rs

 			Either::Right(Err(e)) => {
 				// TODO: https://github.com/paritytech/jsonrpsee/issues/67
-				log::error!("Client Error: {:?}", e);
+				log::error!("Client Error: {:?} terminating connection", e);
+				break;


Unrelated, but it closes the event loop once an error is received.

Are the sending ends of the channels going to shut down gracefully when we drop the receivers here?

Technically, the senders are not gracefully terminated but once the receiver is dropped here all senders will receive None -> Error::Internal once any of the operations (method call, notification, subscription) is invoked.

Thus, not possible for the user to know the exact failure reason without another channel or message (but we have logs lol)

EDIT:
I can revert this change if you want and handle it in a separate PR, I think it would make sense to incorporate this with https://github.com/paritytech/jsonrpsee/pull/134/files and fix better error messages for it.

Then add some similar tests that we have for the HTTP such as an invalid request ID and so on.

Up to you; it's fine to keep here too.

Make the repo structure more understable w.r.t testing.

…riptions

src/client/ws/client.rs

…ch/jsonrpsee into v2-ws-client-drop-subscriptions

Ensure that if more than the requested channel buffer capacity is exceeded it should not deadlock. Such as spawning alot of concurrent requests, notifications or new subscriptions.

src/client/ws/client.rs

niklasad1 · 2020-11-29T18:29:06Z

src/client/ws/client.rs

 	///
 	/// Ignores any malformed packet.
-	pub async fn next(&mut self) -> Notif {
+	pub async fn next(&mut self) -> Option<Notif> {


This is the major change in this PR.

niklasad1 · 2020-11-29T18:31:06Z

src/client/ws/client.rs

+/// Configuration.
+pub struct Config {
+	/// Backend channel for serving requests and notifications.
+	pub request_channel_capacity: usize,


this becomes useless in practice because we clone the sender every time, see PR description for more info.

src/client/ws/client.rs

dvdplm · 2020-11-30T22:56:20Z

src/client/ws/client.rs

 			Either::Right(Err(e)) => {
 				// TODO: https://github.com/paritytech/jsonrpsee/issues/67
-				log::error!("Client Error: {:?}", e);
+				log::error!("Client Error: {:?} terminating connection", e);
+				break;


Up to you; it's fine to keep here too.

dvdplm · 2020-11-30T22:57:58Z

src/client/ws/tests.rs

@@ -0,0 +1 @@
+#![cfg(test)]


Is there code missing here perhaps? There was a test removed in ws/tests.rs – was it meant to be moved here? Or is it all in integration_tests?

dvdplm · 2020-11-30T22:58:54Z

src/ws/tests.rs

-		}
-	}
-}
-


Can you comment on why these tests are removed?

Sorry, some of the tests here were not really "unit tests" so I moved them to integration tests instead. Thus, needed both the "non-mocked" server and client.

dvdplm · 2020-11-30T23:03:11Z

src/client/ws/client.rs

+				let notifs_tx = match active_subscriptions.get_mut(&request_id) {
+					None => {
+						log::debug!("Invalid subscription response: {:?}", request_id);
+						continue;
+					}
+					Some((notifs_tx, _)) => notifs_tx,
+				};


This is nice.

dvdplm

I have a few questions and nitpicks, but overall lgtm. :)

src/client/ws/client.rs

Co-authored-by: David <[email protected]>

* update http example * ungeneric crate * update dependencies * [client]: add WebSocket client again. * [deps]: remove needless dev dependencies * nits: forgot to commit new files * [ws client]: `send_text` instead of `send_binary` This is temporary fix to work with the `server` which assumes that `WebSocket` resonses are `text` * chore: add a bunch of more logging * [ws]: port tests but some are failing. * chore: fmt * [server API]: expose `fn local_addr` as public API. * [tests]: make them less ugly by using `127.0.0.1:0` * fix: a bunch of compiler warnings. * [api]: uniform naming, `bind -> new` in transport. * [websocket server]: reply when deserial fails When the server receives an request with invalid JSON `-32700, Parse error shall be returned` * chore: fmt * [tests]: fix remaining tests * [websocket server]: support `binary` and `text` * [ws server]: fix bug in subscription response. `.await` was missing in RegisteredSubscription::send() and no responses were actually sent which this commit fixes. * [client API]: export `WsSubscription` * [examples]: use `localhost` instead of `127.0.0.1` Hostname is required when using `wss` and `127.0.0.1` is not valid hostname. * [examples]: add subscription example. * chore: fmt * [ws server]: fix bug register new subscription. Fixes newly introduced bug that causes `register_subscription` to have side-effects even if the subscription fails. * fmt * more uniform logs * [ws server]: simple subscription test. * [ws server]: subscription tests improved. * [tests]: extract test helpers to a separate crate (#125) * [ws server]: don't close connection when `deserialization` fails (#131) * [ws server]: don't close conn. when `deser` fails * Update src/ws/transport.rs * grumbles: prefer matching of if else. * chore: CI warn `intra_doc_link_resolution_failure` (#139) Since we have not updated the documentation properly it's annoying that the entire job fails. * chore: rustfmt.toml (#138) * chore: add `rustfmt.toml` for formatting * style: `cargo fmt --all` with new config * [server raw params]: fix debug implementation (#137) * [server]: simply raw params impl Use debug implementation of `common::Params` instead of doing something similar that doesn't work properly. * [raw params]: derive `Debug` impl. * [ws server]: parse subscription ID for unsubscription instead of hardcoding `JsonValue::Null` (#136) * [ws server]: fix broken unsubscribe. Try to parse the subscription ID as the first element of an Array or the `subscription` field of an Object/Map. If both of those fails then regard it as a error. * fmt * fix grumbles: remove space indentation * fix(ws server): sub/unsubscribe to same method should generate an error (#140) * fix(ws server): sub/unsubscribe to same method err Subscribe and unsubscribe to the same method should generate an error, which this commit fixed. This bug was introduced by myself in fc87889 * Update src/ws/server.rs Co-authored-by: David <[email protected]> Co-authored-by: David <[email protected]> * chore: add naive benches for request/response (#142) Co-authored-by: Niklas <[email protected]> * fix(ws server): remove faulty debug_assert (#145) The code assumed that `subscription id` is still in `active_subscriptions` when the connection was dropped. The list of subscriptions (kept in raw server) are not notified when a client dropped its subscription/unsubscribed thus it's possible that the actual subscriptions are closed before the entire client was dropped. * ci(benches): cargo check on benches. (#146) * fix(http client): implement `clone` uniform API. (#147) * chore(deps): update `futures v0.3.7` (#148) * chore(deps): update remaining crates (#149) * chore(deps): update `futures v0.3.7` * chore(deps): bump the rest of deps * Improve HTTP client background thread (#150) * refactor: resultify API + some crate reorg (#144) * [ws client]: resultify API and fix subscribe. * The commit changes the API to return `Err` when it's possible and to not ignore underlying errors. * Fix that `fn subscribe` doesn't accept the subscription and unsubscription to be same which causes errors in the server. * nits: Err::SubscriptionMetod -> Err::Subscription * refactor(client): common error type * refactor(http client): resultify * refactor(common): rename common -> types.. This commit renames the `common module` to `types` and tries to distinguish the types that is directly related to the `JSON-RPC v2 specification` from others. Somethings are a little big sloppy named as naming is hard. Also, as bonus a removed a bunch of needless stuff in http server related to subscription. * Update src/ws/tests.rs * style: cargo fmt * fix(grumble): matches -> assert(matches) * fix(grumbles): `jsonrpc_v2` -> `jsonrpc` * fix(nit): remove unused code. * fix(benches): make it compile again. * style: cargo fmt * fix nits (#151) * fix(ws client): send binary (1 byte less payload) * docs(ws server): fix bad comment. * chore: add `editorconfig` (#152) * chore: make `debug log` less verbose. (#153) * chore: make `debug log` less verbose. The debug logging was just too verbose and this commit simplies it as follows: ``` DEBUG recv: {"jsonrpc":"2.0","method":"<METHOD>","params":<PARAMS>,"id":<ID>} DEBUG send: {"jsonrpc":"2.0","result":"<RESULT>","id":<ID>} ``` * style: cargo fmt * fix: missed logs * [jsonrpc types]: implement Display for Request/Response (#160) * feat(jsonrpc response/request): impl `Display` * refactor(logging): use display impl * use serde_json for verbosity * [http client]: refactor with "syncronous-like" design (#156) * experimental * ci(benches): sync and concurrent roundtrips Improve benchmarks to take concurrent requests into account. * ci(benches): sync and concurrent roundtrips Improve benchmarks to take concurrent requests into account. * fix(nits) * feat(http client): limit max request body size * test(http transport): request limit test * test(http client): add tests. * fix typo * fix(benches): make it compile again. * fix(ws example): revert unintentional change. * test(http client): subscription response on call. * fix(cleanup) * fix(benches): make it compile again. * Update src/client/http/transport.rs * fix(http client): `&str` -> `AsRef<str>` * docs(client types): better docs for Mismatch type. * style: `Default::default` -> `HttpConfig::default` * fix(http client): read body size from header. Expermential to read number of bytes from `HTTP Content Length` to pre-allocate the number of bytes and bail early if the length is bigger than the `max_request_body size` Need to be benched with bigger requests. * test(raw http): enable tests to works again. * style: cargo fmt * benches: address grumbles * feat(jsonrpc response/request): impl `Display` * refactor(logging): use display impl * fix(http client): nits. * Update benches/benches.rs * fix bad merge. * chore(deps): update dependencies. (#164) * feat(http server): configurable request body limit (#162) * feat(http server): configurable request body limit * refactor(crate reorg): to have shared http helpers. * Merge client and server errors. * Move `http_server_utils` to `utils/http` * Minor cleanup * fix nits * fix(hyper helper): u64 -> u32 * Update src/utils/http/hyper_helpers.rs Co-authored-by: David <[email protected]> * Update src/utils/http/hyper_helpers.rs Co-authored-by: David <[email protected]> * fix: grumbles * Update src/utils/http/hyper_helpers.rs Co-authored-by: David <[email protected]> * Update src/http/server.rs Co-authored-by: David <[email protected]> Co-authored-by: David <[email protected]> * ci: remove nightly (#167) Use stabilized `broken_intra_doc_links` instead of `intra_doc_link_resolution_failure` * fix(websocket client): drop subscriptions that can't keep up with the internal buffer size (#166) * fix(ws client): drop subscriptions when full. This commit changes the behavior in the `WebSocket Client` where each subscription channel is used in a non-blocking matter until it is determined as full or disconnected. When that occurs the channel is simply dropped and when the user `poll` the subscription it will return all sent subscriptions before it was and terminate (return None) once it's polled one last time. Similarly as `Streams` works in Rust. It also adds configuration for the `WebSocket Client` to configure capacity for the different internal channels to avoid filling the buffers when it's not expected. * tests(ws client): simple subscription test. * fix: nits * Update src/client/ws/client.rs * refactor(tests): introduce integration_tests Make the repo structure more understable w.r.t testing. * chore(license): add missing license headers * Update src/client/ws/client.rs * Update src/client/ws/client.rs * style: remove unintended spaces. * tests: add concurrent deadlock test Ensure that if more than the requested channel buffer capacity is exceeded it should not deadlock. Such as spawning alot of concurrent requests, notifications or new subscriptions. * Update src/client/ws/client.rs * fix: review grumbles * fix nits: `remove needless closure` * fix: cargo fmt * Update src/client/ws/client.rs Co-authored-by: David <[email protected]> * fix more nits Co-authored-by: David <[email protected]> * fix(ws client): embed request id in `SubscriptionClosed` (#170) * fix(ws client): embed request id SubscriptClosed Fixes #169 * Update src/client/ws/client.rs * Update src/client/ws/client.rs Co-authored-by: David <[email protected]> * Update src/client/ws/client.rs Co-authored-by: David <[email protected]> Co-authored-by: David <[email protected]> * chore(deps): bump dependencies (#172) * [ws client]: add tests (#134) * [test utils]: add `internal_err` and consts [errors]: unify client/server errors [test utils]: fake WebSocket jsonrpc server [ws client]: export errors [ws client]: add some basic tests * fmt * remove log target * fix nits * [ws client]: add subscription test * revert unintendend changes. * fmt * [ws client]: fix panic in tests * cleanup * tests(ws client): test for invalid request ID. * fix nits * [ws client]: kill raw client (#171) * getting started * WIP WIP * cleanup * cleanup v2 * cleanup v3 * perf: use BufReader BufWriter * fix(request manager): resultify insert API The rationale behind this change is that the `insert_methods` takes ownership of the `send_back_oneshot` and if the operation fails it should be propagated the frontend. So returning the `Err(send_back_oneshot)` if it fails makes it possible. * fix nits * examples(ws): revert changes * Update tests/integration_tests.rs * nits: fix unwraps * Update src/client/ws/manager.rs Co-authored-by: David <[email protected]> * Update src/client/ws/transport.rs Co-authored-by: David <[email protected]> * Update src/client/ws/client.rs Co-authored-by: David <[email protected]> * fix build * refactor: simplify `Error::InvalidRequestId` It was hard to use when the expected id is not known. * fix(ws client): error handling. * fix(grumble error type): better error message. * fix(grumble): docs `JSONRPC WebSocket transport` * fix(ws manager): fix grumbles. * Add better documentation * Rename methods. * Add `proof` to unreachable! * fix(ws manager): fix nit in docs. * fix(grumbles): ws client * fix more nits * fix compile warning: export websocket transports. * Update src/client/ws/manager.rs Co-authored-by: David <[email protected]> * deps: tokio 1.0 and hyper 0.14 (#176) * deps: tokio 1.0 and hyper 0.14 * Update Cargo.toml * refactor: crate re-organization with separate crates (#177) * [ci]: feature `http` and `ws` removed. * refactor: re-org crate with smaller crates. * fmt * [ci]: remove default features Currently there are no features in the crates, so that check is not needed. * [http client]: remove unused dependency tokio * docs(http client): fix nits tokio 0.2 -> tokio 1.0 (#178) * docs(http client): tokio 0.2 -> tokio 1.0 * fix: better link * [ci]: github actions (#179) * docs(http client): tokio 0.2 -> tokio 1.0 * [ci]: remove travis * [ci]: add github actions. * [ci]: fix identation nits * [ci]: use cache for actions * [ci]: filter to clippy * [ci]: remove hacks * separate action for ci and benchmarks * [ci]: tweak to run on master branch. * examples/subscription -> examples/ws_subscription * force CI Co-authored-by: David <[email protected]> Co-authored-by: Maciej Hirsz <[email protected]> Co-authored-by: Atkins <[email protected]>

niklasad1 added 3 commits November 27, 2020 13:01

tests(ws client): simple subscription test.

15cae68

fix: nits

29beb50

niklasad1 commented Nov 27, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

Update src/client/ws/client.rs

f50becd

niklasad1 commented Nov 27, 2020

View reviewed changes

niklasad1 requested review from maciejhirsz and dvdplm November 27, 2020 13:01

niklasad1 added 3 commits November 27, 2020 18:32

refactor(tests): introduce integration_tests

213e966

Make the repo structure more understable w.r.t testing.

chore(license): add missing license headers

2f72354

Merge remote-tracking branch 'origin/v2' into v2-ws-client-drop-subsc…

0247463

…riptions

niklasad1 commented Nov 28, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

Update src/client/ws/client.rs

64e480f

niklasad1 commented Nov 28, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

niklasad1 added 4 commits November 28, 2020 12:27

Update src/client/ws/client.rs

b5392d2

style: remove unintended spaces.

4dd85a5

Merge branch 'v2-ws-client-drop-subscriptions' of github.com:parityte…

9a44b4a

…ch/jsonrpsee into v2-ws-client-drop-subscriptions

tests: add concurrent deadlock test

29ffcd8

Ensure that if more than the requested channel buffer capacity is exceeded it should not deadlock. Such as spawning alot of concurrent requests, notifications or new subscriptions.

niklasad1 marked this pull request as ready for review November 29, 2020 12:24

niklasad1 commented Nov 29, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

niklasad1 commented Nov 29, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

niklasad1 commented Nov 29, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

Update src/client/ws/client.rs

f82277f

dvdplm reviewed Nov 30, 2020

View reviewed changes

dvdplm approved these changes Nov 30, 2020

View reviewed changes

niklasad1 added 2 commits December 1, 2020 10:48

fix: review grumbles

7678ed4

fix nits: remove needless closure

2645f96

fix: cargo fmt

488d626

dvdplm reviewed Dec 1, 2020

View reviewed changes

src/client/ws/client.rs Outdated Show resolved Hide resolved

niklasad1 and others added 2 commits December 1, 2020 14:22

Update src/client/ws/client.rs

bba1594

Co-authored-by: David <[email protected]>

fix more nits

3e94f6a

niklasad1 merged commit 53eaf0e into v2 Dec 2, 2020

niklasad1 deleted the v2-ws-client-drop-subscriptions branch December 2, 2020 10:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(websocket client): drop subscriptions that can't keep up with the internal buffer size #166

fix(websocket client): drop subscriptions that can't keep up with the internal buffer size #166

niklasad1 commented Nov 27, 2020 •

edited

Loading

niklasad1 Nov 27, 2020

maciejhirsz Nov 30, 2020

niklasad1 Nov 30, 2020 •

edited

Loading

dvdplm Nov 30, 2020

niklasad1 Nov 29, 2020

niklasad1 Nov 29, 2020

dvdplm Nov 30, 2020

dvdplm Nov 30, 2020

dvdplm Nov 30, 2020

niklasad1 Dec 1, 2020 •

edited

Loading

dvdplm Nov 30, 2020

dvdplm left a comment

		@@ -0,0 +1 @@
		#![cfg(test)]

fix(websocket client): drop subscriptions that can't keep up with the internal buffer size #166

fix(websocket client): drop subscriptions that can't keep up with the internal buffer size #166

Conversation

niklasad1 commented Nov 27, 2020 • edited Loading

NOTE

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niklasad1 Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niklasad1 Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dvdplm left a comment

Choose a reason for hiding this comment

niklasad1 commented Nov 27, 2020 •

edited

Loading

niklasad1 Nov 30, 2020 •

edited

Loading

niklasad1 Dec 1, 2020 •

edited

Loading