Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

4.27.0 introduced a change that prevents the event loop from being cleaned up properly #1362

Open
evanlucas opened this issue Jun 3, 2021 · 8 comments
Labels

Comments

@evanlucas
Copy link

https://github.com/evanlucas/ioredis-sentinel-hang is how I was able to reproduce it. We rely on calling quit in all of our jobs/workers/services and if the event loop is not cleaned up, it prevents node from cleanly exiting the process. It looks like it is the new failover detection mechanism that was added. I'm seeing additional TCP sockets being opened, but never closed.

This only seems to occur when using redis sentinel.

Thanks!

@AVVS
Copy link
Contributor

AVVS commented Jun 4, 2021

confirming, it opens a bunch of new connections that are never cleaned up (see FailoverDetector)

@luin
Copy link
Collaborator

luin commented Jun 5, 2021

Hey @evanlucas, thanks for the reproducible repo! Calling .disconnect() should properly clean up everything. We should take a look for a fix for quit.

@evanlucas
Copy link
Author

Thanks for jumping on this and getting a non-breaking release published! I really appreciate y'all's time! Did y'all want to leave this issue open? or is it good to close?

@luin
Copy link
Collaborator

luin commented Jun 14, 2021

Hey @evanlucas, let's leave this open as the root issue hasn't been fixed.

@esatterwhite
Copy link

Any progress on this one. It would be fantastic to get this one resolved

@juris
Copy link

juris commented Feb 7, 2022

In a Kubernetes cluster, when one of the Sentinels restarts and gets a new IP address, client starts throwing errors like this one:

[ioredis] Unhandled error event: Error: connect EHOSTUNREACH 10.0.55.221:26379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17)

Where 10.0.55.221 is the old Sentinel IP address.

Issue disappears with failoverDetector: false

@Mall0c
Copy link

Mall0c commented Jun 8, 2022

AFAICT failoverDetector: true is a useful feature. I am testing Redis Sentinel on a local setup with three VMs, each of them with a Redis and a Redis Sentinel instance. Shutting down the VM that holds the Redis Master ungracefully with Force Off in the Virtual Machine Manager does not trigger a Reconnect. Somehow ioredis does not that Sentinel has performed a failover, unless this flag is set to true. For now, I went with calling quit() and disconnect() after that. Seems to work.

@silverwind
Copy link
Contributor

silverwind commented Sep 19, 2022

There's definitely something fishy going on when failoverDetector is enabled. After a failover, ioredis will start logging Unhandled error event to stderr, despite me having registered the error event earlier which should prevent all such logging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants