issue with --ws-max-connections on http requests

[ehsanhajian]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

following https://substrate.stackexchange.com/questions/7485/parachain-rpc-errors-since-polkadot-0-9-36-client-upgrade post. I believe the issue is related to --ws-max-connections . it only works on WS connection, not on HTTP requests.

As you can see in attached images I have a node for both WS and HTTP requests. I set --ws-max-connections 50000 but it only works on WS requests not on HTTP requests. when I sent more than 100 http requests it started to to generate "Transport endpoint is not connected" error/message. It has nothing with SSL/TLS. I tested on both http and https protocols.

Steps to reproduce

I use https://github.com/shazow/ethspam and https://github.com/INFURA/versus to generate valid jsonrpc queries. the command that I use

ethspam --rpc http://endpointurl | versus --concurrency=100 --stop-after=10000 http://endpointurl

[bkchr]

CC @niklasad1

[niklasad1]

@ehsanhajian

You are doing the call against the old HTTP server right i.e, port --rpc-port, correct? It will use the default value by jsonrpsee which causes that to be 100 which was a breaking change I realized now.

The server running on --ws-port will use this --ws-max-connections so you could actually try your script against that one, it should work both on HTTP and WebSocket.

Thus, if we continue and merge #13384 this will be fixed. Otherwise, I suppose we need to expose another CLI flag for the "HTTP server only" or just hardcode it some huge value.

[ehsanhajian]

@niklasad1 I didn't use rpc-port in my config, I've exposed both ws and rpc on default ports 9933 and 9944

I also tried --rpc-ws-port but seems still not supported. error: unexpected argument '--rpc-ws-port' found

[niklasad1]

sorry my bad, it is --ws-port but which port are you doing the HTTP requests to?

Assuming 9933 from your description...

[ehsanhajian]

Exactly. 9933 for http and 9944 for ws connections

[niklasad1]

As I tried to explain earlier the "port 9933 endpoint" has the number of connection hardcoded to 100 but 9944 supports both HTTP and WS based on --ws-max-connections

Can you double check doing the HTTP requests against 9944 as well? 🙏

[ehsanhajian]

yes. That's correct. 9944 can support both http and ws requests.

[ehsanhajian]

but we still have this message

2023-03-09 16:20:07 HTTP serve connection failed hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })

is it only information log or an error?

[niklasad1]

I would not regard it as an error, the remote peer/client closed the connection thus "information log"

I think we can make this log debug instead as it causes more confusion than good.

I expect this log only be visible if something goes wrong when reading the socket as the error directly comes from https://docs.rs/hyper/latest/hyper/server/conn/struct.Http.html#method.serve_connection

[ehsanhajian]

well even in debug mode I couldn't find anything related to it (at least for me) we just have many connectivity error to peers node. it seems because port 30333is not reachable. I'm not sure if it's relevant to "Transport endpoint is not connected" or not.

Mar 13 07:17:50 poc-0 astar-collator[1441909]: 2023-03-13 07:17:50.112 DEBUG tokio-runtime-worker libp2p_swarm: [Parachain] Connection attempt to PeerId("12D3KooWMJTknQ69Tu46pBygus1RemovedByMe") failed with Transport([("/ip4/18.181.186.198/tcp/30333/ws/p2p/12D3KooWMJTknQ69Tu46pBygus1RemovedByMe", Other(Custom { kind: Other, error: Timeout }))]).
Mar 13 07:17:54 poc-0 astar-collator[1441909]: 2023-03-13 07:17:54.696  WARN tokio-runtime-worker jsonrpsee_server::server: HTTP serve connection failed hyper::Error(Shutdown, Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" })
Mar 13 07:17:54 poc-0 astar-collator[1441909]: 2023-03-13 07:17:54.696  WARN tokio-runtime-worker jsonrpsee_server::server: HTTP serve connection failed hyper::Error(Shutdown, Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" })
Mar 13 07:18:49 poc-0 astar-collator[1441909]: 2023-03-13 07:18:49.840 DEBUG tokio-runtime-worker libp2p_swarm: [Parachain] Connection attempt to PeerId("12D3KooWNYkYSHkjpGZi7RemovedByMe") failed with Transport([("/ip4/52.155.117.200/tcp/30333/ws/p2p/12D3KooWNYkYSHkjpGZi7RemovedByMe", Other(Custom { kind: Other, error: Timeout }))]).

[niklasad1]

libp2p and jsonrpc errors are not related in general, so for libp2p errors please open another issue.

As I said before just ignore this JSON-RPC server warning, it just that the client is not connected anymore.

[bLd75]

Thanks @niklasad1 for insights, we didn't get that both ports were merged behind --ws-port actually, it would be great if that was specified in docs and eventually merged behind --rpc-port for which terminology makes more sense for an http/ws server.

By the way, would it be possible to set the message "Accepting new connection" in debug for HTTP server? It makes the node logs very large on an http server and doesn't really makes sense since connections don't stay open on it.

[niklasad1]

By the way, would it be possible to set the message "Accepting new connection" in debug for HTTP server? It makes the node logs very large on an http server and doesn't really makes sense since connections don't stay open on it.

Yes

[niklasad1]

Closed by #13384

Now, the http server has been removed an the --rpc-max-connections will configure it for a server which supports both http and websocket