New connection created everytime

[JohnSundarraj]

Hi there, i'm trying to use RQ worker to execute background jobs in my Torando App. I'm successful sofar, but one weird thing i noticed. Redis client creates a new connection everytime a job gets executed by the RQ worker. I found this issue by using the command redis-cli info | grep 'connection'. Everytime i check this command after a job gets executed, its incremented. To solve this problem i tried various implementations, but nothing worked. Below it he wrapper class i've created. It seems to be a bug in redis client, if i'm not wrong...Need help on this issue. Thanks in advance.

# Redis controller.
class Redis(object):
  __Instance = None

  def __init__(self):
    if self.__class__.__Instance:
      return
    else:
      self.__class__.__Instance = self
    self.Conf = DBConf()['Redis']
    print "NEW REDIS"
    self.Engine = redis.Redis(connection_pool=redis.ConnectionPool(host=self.Conf['Host'],port=self.Conf['Port'],max_connections=10))

  def __new__(self):
    if self.__Instance:
      return self.__Instance
    else:
      return object.__new__(self)

[andymccurdy]

Your code above is creating a singleton out of a redis client instance. You can do the same without the need for your class. Simply create a redis client instance at the module level and have your workers import it.

For example:

# my_redis.py
import redis

client = redis.StrictRedis()


# worker.py
from my_redis import client

def worker():
    client.get("foo")

[JohnSundarraj]

@andymccurdy Thanks for quick response. I'll try your solution and comeback to you.

[JohnSundarraj]

Tried your code but still it creates a new connection. Now here is my code.

# Worker.py
import redis
from rq import Worker,Queue,Connection
from Config import DBConf
RedisConf = DBConf()['Redis']
#Redis = redis.StrictRedis(connection_pool=redis.ConnectionPool(host=RedisConf['Host'],port=RedisConf['Port'],max_connections=10))
Redis = redis.StrictRedis()

# Start RQ Worker.
if __name__=='__main__':
  listen = ['high','default','low']
  with Connection(Redis):
    Worker = Worker(map(Queue,listen))
    Worker.work()

#Async.py
from Worker import Redis

class MyClass(object):

  # Initializer.
  def __init__(self):
    self.Redis = Redis # Here is the place its creating new redis object instead of using the imported one.

  @classmethod
  def MyFunc(self,Data):
      # do some stuff here.

[andymccurdy]

What kind of job execution/worker model are you using? Are you passing the -w option to rqworker like the bottom of this page suggests? http://python-rq.org/docs/workers/

[JohnSundarraj]

No, im not passing -w option to rq worker. This is what im trying to accomplish...rq/rq#376

[andymccurdy]

I'm actually quite surprised your solution in the first post works. According to the RQ docs, the default behavior of workers is to fork, do the work, then kill the forked worker.

redis-py explicitly does not allow forked processes to reuse connections. This is because of problems with inheriting open file descriptors. See this code: https://github.com/andymccurdy/redis-py/blob/master/redis/connection.py#L819

_checkpid() is called anytime a connection is retrieved from the pool. It's purpose is to ensure that all connections within the pool belong to the current process which prevents problems like two different processes sharing the same connections.

[JohnSundarraj]

Which first post you are talking about?

So whats the workaround for this problem...I dont this i can do much with redis, i need to do something with RQ worker. Am I right?

[andymccurdy]

I was talking about the first post on this issue. While it guarantees that only a single client instance and connection pool will be constructed, it should still have the same problems when being used across multiple forked processes.

The first thing I'd try is to use the gevent worker and see if that changes anything.

[JohnSundarraj]

Yeah, you're correct, the code on the first post worked as expected. But only for Tornado, not for RQ Worker, for RQ still it created new connects. Since RQ is forking, it creates new connects everytime it executes a job(correct me if i'm wrong). So as you suggested i need to do something with gevent or multiprocessing.

Anyway thanks for quick response.

[andymccurdy]

Sure. I'd avoid multiprocessing for now. There's a known bug: #496 around it that I haven't gotten around to fixing yet.

[JohnSundarraj]

Ok, i'll take a look on to it. Thanks for informing about the bug. I'll let u know the result of my workaround.

[JohnSundarraj]

@andymccurdy I fixed the issue using gevent and singleton DB wrappers. Thanks for your suggestions.

[jeffwidman]

Looks like this issue should be closed.

[viveksinghtt]

@andymccurdy is this fixed now? I am also facing issues using redis with multiprocessing.

[JohnSundarraj]

@viveksinghtt can you please provide the code snippet, so that we can suggest something.

[viveksinghtt]

@JohnSundarraj i have posted the details here.Please have a look.
StackoverFlow

[andymccurdy]

3.2.0 now behaves well with multiprocessing and forked connections.