Add a JsonRpcWebSocketProvider

[peetzweg]

I use this library a lot. Way more usable than web3...

Do you guys have a feature roadmap? I wonder if there will be an implementation to use IPC and Websockets in the near future.

Thanks, keep up the good work. 👍🏼

[ricmoo]

I had totally overlooked this, as I use the default provider for the most part.

But yes, this is an incredibly useful feature which I'll add this week.

[peetzweg]

Any updates on this issue? Anything I could help with?

[florianlenz]

Hi, I would also love to see such a feature. I am willing to implement the websocket version. I would probably just need a bid of guidance.

Thank you for your great work.

[ricmoo]

Sorry, I've been busy. I will try to look at it this weekend during my downtime mentoring at the CryptoChicks Hackthon.

If I had to focus on one, which is more important to you more immediately, WebSockets or IPC?

[florianlenz]

For me personally WebSocket's. I don't mind todo it :) I guess the send function would need to be changed right? I see some polling JSON RPC provider in the code but I am not sure for what exactly it's used. Geth and Parity both have an way to subscribe to new transaction's and block's https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB

[florianlenz]

I added the websocket provider: https://github.com/florianlenz/ethers.js/blob/feature/websocket_provider/providers/web-socket-provider.js. Works quite well till now. I will test him for a few day's.

[ricmoo]

I have added the first incarnation of the IpcProvider. It is obviously only available in Node.js and not from the browser dist files.

Please try it out. :)

[MicahZoltu]

Definitely WebSocket oven IPC. The IPC protocol with Geth/Parity is both terrible and non-standard between the two. For 99% of use cases, WebSocket will be pragmatically as fast as IPC, your bottleneck when talking to a local Ethereum node is almost certainly not the networking stack.

[ricmoo]

The goal of ethers.js is to support as much of the ecosystem as is reasonable, while remaining highly compatible and compact.

The main advantage of IPC is not speed, but security.

WebScokets are certainly useful, but require a lot more work. The IpcProvider is already complete and has been included for quite some time. Both provide a very different abilities.

@MicahZoltu What incompatibilities have you found between Geth and Parity over IPC (that aren't present over http)? I also want to smooth out incompatibilities. :)

[MicahZoltu]

The one that bugged me the most is the fact that they terminate their lines differently. Both of them are delimited JSON, but one of them includes newlines between payloads while the other does not (I think it does comma or something).

In general I really detest delimited JSON because tooling for parsing it correctly is difficult to write (not impossible though).

On top of that, IIRC, named pipes on windows vs Linux had some oddities. For starters, the way they name them is different, and access controls and errors are different. I don't remember specifically if there were other differences.

I also vaguely remember Node's low level IPC code having a bug that was kind of nasty (I filed it somewhere on their GitHub if my memory is correct).

[MicahZoltu]

WebSockets on the other hand "just work" on all operating systems, they have a nice interface, they deal with message on eloping for you, and there is a ton of quality tooling to work with them easily.

What do you see as the security advantage of IPC?

[Padraic-O-Mhuiris]

@florianlenz Have you finalised the websocket implementation on your fork?

[florianlenz]

@sirromdev As far as I remember it had a bug somewhere and I didn't spend more time on it. I could check it next weekend.

[Padraic-O-Mhuiris]

Sure, I can lend a hand if you want

[ricmoo]

This is coming up on my to do list too, if you can wait. Is there a hurry for WebSocketProvider? If you do, add this please make sure it follows the shim model that other parts of the library use, so that the browser build does not contain the node ws library in its output.

[sulliwane]

websocket support in ethers.js would be awsome :) much lighter than RPC for event watching, and we could drop web3.js for All Ethers.js yeah

[ricmoo]

Coming VERY soon. Also, much better performance since it doesn’t need to poll. :)

[pgebheim]

How soon is very soon?

We are looking to port some code over to ethers.js for Augur but would need websocket support.

[marcinczenko]

Yes, we also need WebSockets provider like in web3. Some networks kick you out if you are doing extensive polling (which happens when you subscribe to events). Without WebSockets provider we still need to bundle with Web3 and that's a pain.

[hellobart]

Hey @ricmoo Any news about WS? Can't wait to test that.

[waterdrop01]

Any updates about integration of Websockets in Ethers.js?

can't wait to run npm uninstall web3!!! :)

I'd be happy to help testing...

[pstuermlinger]

I'm also switching from web3 to ethers on one of our companies products. Even though it's not a show blocker, websocket support would be awesome!

[mrwillis]

Would LOVE this to save on polling bandwidth and use eth_subscribe and stuff :)

[ricmoo]

I've made some changes to v5 (public beta soon) that should make it easier to swap out polling.

This is still coming, but my time is spread thin, especially over the next few weeks. Anyone going to ETHCapeTown? Come chat! :)

[sergejkunz]

I've made some changes to v5 (public beta soon) that should make it easier to swap out polling.

This is still coming, but my time is spread thin, especially over the next few weeks. Anyone going to ETHCapeTown? Come chat! :)

Hi Ricmoo! I was on the ETH Cape Town! Are you planning to go to ETH NEW YORK?

How are you with the Websocket implementation?

[jtakalai]

How can I check out this v5? Would be great to test it. How do things work without Websockets nowadays anyway, hasn't nigh everyone deprecated HTTP(S) in RPC?

[ricmoo]

npm install ethers@next, but there is no documentation up for it yet to chronicle the differences. The WebSocket implementation isn’t in v5 yet, just the changes to make it easier to add. :)

Everything I’ve seen (and build) still uses HTTPS JSON-RPC, the server load required to maintain WebSockets is tremendous, and it cannot be horizontally scaled easily nor load balanced using traditional infrastructure. For scaling purposes, stateless is still far simpler.

[MicahZoltu]

I agree with all of your comments with the exception of

load balanced using traditional infrastructure

There is traditional infrastructure to load balance web sockets, though it is certainly less common than load balancing one-off requests. Personally, I think the other reasons are enough to not have web sockets be a high priority, though I do think there are situations where web sockets make more sense.

[ricmoo]

I suppose I meant the “as easily” to be applied to old balancing too. ;)

I generally use AWS ELB, without the WAF and friends, so it is superbly affordable and simple to run a large collection of backends, without anything complex, and SSL termination at the load balancer further simplifies and reduces costs.

There are certainly cases where WebSockets make sense, which is why it is still on the backlog, but for what is basically a spare-time project (that eats up 30-50 hours a week), I focus on the areas I am most interested in, or need more urgently for my own random collection of projects. :)

[pgebheim]

The main use case is getting block notifications without HTTPS round trips. This is actually quite well solved with turning on http/2. Of course it would still be better to allow for WS subscriptions.

Now, that said I sure hope you're not load balancing to a cluster of eth nodes cause it really doesn't work 🤣. Blame the ETH RPC protocol for that one of course.

[cloudonshore]

+1 would love websocket support

[deacix]

What!? No Websocket? We have now a million requests on https://1inch.exchange over HTTPS provider! AAAHHHH!

[ricmoo]

WebSocket support is coming. I'm about to start receiving regular funding, which will help. :)

There are a lot of additional complications with a WebSocket provider, mostly revolving what happens when there is a disconnect; we want it to reconnect, but we also need to setup the state from before it disconnected, without replaying already-received events, and maintaining a lot of the subtleties that the standard ethers Provider provides.

For example, if an ethers provider emits a block event for blockNumber 100000, and you call getBlock(100000), due to the eventually consistent nature, the backend queried may not have that block yet, so it will stall (with exponential back-off) and respond once that block response occurs. Similar things happen with other types of events and transactions, the idea being if an event is triggered, a user should be able to reliably assume that the event happened. :)

There are a lot of important, non-obvious things to handle.

Also, keep in mind that WebSockets work great, and you are likely thrilled to use them for low-traffic services, but at scale, millions of HTTPS requests (which can be SSL terminated at the edge cache, with dedicated hardware for SSL termination) will likely greatly out-perform the same costs for the infrastructure required for WebSockets, which requires the server maintain state, meaning much beefier server architecture, and either expensive custom load-balancing hardware or complex software.

The goal of the ethers Provider is that you generally should not need to know or care what type of backend there it. That said, I'm working with some services to help give the best of both worlds, since really the only WebSocket response we really care about is getBlockNumber, since that provides a reliable boundary for when other interesting thing can happen. :)

Long story short, it's coming. :)

[deacix]

WebSocket support is coming. I'm about to start receiving regular funding, which will help. :)

There are a lot of additional complications with a WebSocket provider, mostly revolving what happens when there is a disconnect; we want it to reconnect, but we also need to setup the state from before it disconnected, without replaying already-received events, and maintaining a lot of the subtleties that the standard ethers Provider provides.

For example, if an ethers provider emits a block event for blockNumber 100000, and you call getBlock(100000), due to the eventually consistent nature, the backend queried may not have that block yet, so it will stall (with exponential back-off) and respond once that block response occurs. Similar things happen with other types of events and transactions, the idea being if an event is triggered, a user should be able to reliably assume that the event happened. :)

There are a lot of important, non-obvious things to handle.

Also, keep in mind that WebSockets work great, and you are likely thrilled to use them for low-traffic services, but at scale, millions of HTTPS requests (which can be SSL terminated at the edge cache, with dedicated hardware for SSL termination) will likely greatly out-perform the same costs for the infrastructure required for WebSockets, which requires the server maintain state, meaning much beefier server architecture, and either expensive custom load-balancing hardware or complex software.

The goal of the ethers Provider is that you generally should not need to know or care what type of backend there it. That said, I'm working with some services to help give the best of both worlds, since really the only WebSocket response we really care about is getBlockNumber, since that provides a reliable boundary for when other interesting thing can happen. :)

Long story short, it's coming. :)

Hey! Thanks for your very fast answer! We can promote your funding on twitter if you like!
Of course we don't have a million of requests, but a lot of them.
I am going to change the architecture to web worker to do it in multi thread manner!

May rxjs observables can help you with you problem?

[pgebheim]

Honestly,

Based on performance testing we've done -- simply using HTTP requests and terminating with http/2 turned on + making sure your ETH endpoints use gzip which none of them do by default -- the performance is quite good.

If events could be delivered by long-poll I would see no reason to use WS for the ethereum use case.

[cloudonshore]

Some input based on my personal experience:
Since browsers have a limit to the number of concurrent requests that can be made simultaneously (I think it's 6 in chrome), a large # of RPC requests can grind your dapp to a halt. This is pretty much always what pushes me into using a websocket provider, I'll need to make like 1000 requests to bootstrap state in a dapp and I don't want the initial page load to be slow. In Chrome, even web workers' concurrent connections are limited by that same connection pool so I don't really think there's a way around it. So if there was a way around the concurrent request limit besides a websocket I wouldn't need a websocket provider, that's pretty much my main use for it.

Oh and this is unrelated, but I think it's good to mention in this thread:
One of the hidden "gotchas" I've found using websocket providers is that different browsers have different limitations for their "max packet size", either party tries to send a packet greater than that size over the socket, the connection is dropped by the browser (without a clear indication from the browser as to why). I've done read calls from contracts that return a lot of data and the packet size itself terminates the websocket. This doesn't occur with HTTP obviously, so it's something new users of a provider like this should be aware of. .

[MicahZoltu]

@cloudonshore That is addressed almost entirely with HTTP/2. I generally recommend updating your server to HTTP/2 rather than switching to WebSockets in most cases like that. WebSockets still have advantages, but if that is all you need from it, then HTTP/2 is probably a better (easier) solution.

[marcinczenko]

@pgebheim Some corporate networks (like hospitals) are not tolerating frequent polling (frequent === dependent on the case) but still tolerate web sockets.... In one of the projects we had to throw away otherwise very nice libraries, just because of this.

[pgebheim]

@cloudonshore

Re: max frame size, this is indeed an issue not just in browsers but will all websocket clients.

For bootstrapping all eth nodes currently support websocket batching, where you can issue multiple requests in one http call.

That mixed with http/2 are much simpler operational solutions to the problem.

@marcinzenko

Well that's an odd bummer. If that were the case it seems like any page would fail since you'll be issuing large numbers of similar http requests back to back.

And sure, there are cases where websockets are a solution, so I'm glad it's being added to ethers.js, and Alchemy is now terminating WS.

For non+browser based sitatuations, eg when you have a server talking to a dedicated eth node they can be also leveraged usefully.

[mrwillis]

With the addition of websockets, would we also support eth_subscribe as in

https://web3js.readthedocs.io/en/1.0/web3-eth-subscribe.html

I am not sure if this is official spec but parity and geth do implement those endpoints.

[pgebheim]

For sure, re: eth_subscribe.

That's really the core reason for WS anyway since the eth spec didn't specify long polling for http subscriptions 🤦‍♂️

[ireply2spam]

Thanks for the work on this folks. Looking forward to v5 moving out of beta. +1 for WS support

[ricmoo]

Anyone who is interested, there is finally a WebSocketProvider. Please try it out and let me know how it works for you. :)

const provider = new ethers.providers.WebSocketProvider(connection);

:)

[ricmoo]

I'm going to close this issue now. If you have any problems with the WebSocketProvider, please feel free to open a new issue.

Thanks! :)

[sssubik]

Hey @ricmoo . I have a etheruem full node running and I am connecting it through web socket. I am trying to figure out a way so that if the connection fails due to error in full node, I want it to hit Infura end point.
I am trying to catch the error when I try to use websocket of the full node, and if there is error provider is connected to infura.

But I cant catch error.

provider = new ethers.providers.WebSocketProvider("ws://localhost:8546").on('uncaughtException', (err)=>{
        provider = new ethers.providers.WebSocketProvider("wss://mainnet.infura.io/ws/v3/" + process.env.API_KEY)
})

events.js:187
      throw er; // Unhandled 'error' event

Getting this