Websockets subscription hangs after a while

[DefiCake]

Hi, I am having issues with hardhat. Extended period of times in a subscription seem to end up with nothing being notified to the subscribed client.

I double checked that this only happens in hardhat, with both ethers.js and web3.js clients. Raw geth, Alchemy, Infura seem to be all working fine with the clients.

Repository with a very simple dockerized environment to aid with reproduction:

https://github.com/DefiCake/hardhat-ws-issue

[iscke]

this happens because ws subscriptions are implemented as filters, which have a deadline of 5 minutes and get removed after that if you haven't called eth_getFilterChanges to poll them, which you obviously don't need to do if you're receiving changes through a websocket.

i think the way to fix this is just to ignore the filter deadline for subscriptions, since polling a websocket filter to keep it active doesn't really make sense.

[iscke]

(probably relevant: #1780 (comment))

[iscke]

(regarding those comments, eth does have a filter deadline, but as far as i can tell, it's only used for the json-rpc eth_newFilter methods (see the subscription and EventSystem structs in eth/filters/filter_system in the geth codebase))

[molfar]

I confirm, there is some issue with events deadline with websocket connection.

[alcuadrado]

@iscke are you familiar with how geth handles this? do they also set a deadline? do they ignore it for websocket subscriptions? if so, don't they leak resources forever?

[iangregsondev]

Hi,

do you know when this is going to be looked at?

Websockets cease to function after around 5 minutes. So a restart of the hardhat node is required. I am running it out of process i.e "npx hardhat node"

I think my problem is related to this. Websockets initially works and events do arrive but then finally it stops delivering any events.

[shawnwang0715]

Same issue here.
My current workaround is to unsubscribe all events and re-subscribe them again every minute. So it won't hit the "5-minute deadline"

updateSubscription() {
    const subscriptions = Array.from(web3.eth._requestManager.subscriptions.values());
    for (const item of subscriptions) {
        item.subscription.unsubscribe();
    }
    // Subscribe your events again
    ...
  }

[github-actions]

This issue was marked as stale because it didn't have any activity in the last 30 days. If you think it's still relevant, please leave a comment indicating so. Otherwise, it will be closed in 7 days.

[iangregsondev]

@iscke @fvictorio Any chance we can leave this active and remove the stale label.

This is still an issue, there are some workarounds but I am not sure if they fully support all use cases and different example workarounds exist for web3 / esthers.js.

Thanks.

[timolson]

Websockets dying is 100% a critical issue preventing us from using Hardhat whatsoever. I'm pretty shocked that so many people seem to be polling for new blocks instead of subscribing to newHeads. Not sure why the priority was removed from this bug, but +1 from me.

[sertony]

In our project we've fixed it by forcing the hardhat node generating new blocks, i.e.

 const config: HardhatUserConfig = {
     solidity: '0.8.20',
     networks: {
         hardhat: {
             // See: https://hardhat.org/hardhat-network/docs/reference#mining-modes
             mining: {
                 auto: true,
                 // Produce new block every 3 minutes to resolve next issues
                 // https://github.com/NomicFoundation/hardhat/issues/2053
                 // https://github.com/ethers-io/ethers.js/issues/2338
                 // https://github.com/ethers-io/ethers.js/discussions/4116
                 interval: 3 * 60 * 1000, // should be less then 5 minutes to make event subscription work
             },
         },
     },
 };

This is important since the 5 minute deadline is getting expired due to the fact that it's not renewed when reading the filter changes which happens in ethers only once a new block is generated.

[blastrock]

If someone is searching for the new deadline, it has moved here:

hardhat/crates/edr_provider/src/filter.rs

Line 110 in 7d0f981

Instant::now() + Duration::from_secs(5 * 60)

[sim31]

In our project we've fixed it by forcing the hardhat node generating new blocks, i.e.

I'm using auto mining mode. Still am missing events after around 5 minutes. I have also setup up pinging and reconnection to keep the websocket alive so I know that websocket connection is ok, but subscription to events somehow drops.