Dramatiq worker stuck at Dramatiq '0.14.0' is booting up.

[rakanalh]

I am experiencing a strange behavior when running dramatiq worker using django:

python manage.py rundramatiq --use-gevent

The log prints out:

[2017-12-02 20:36:53,804] [PID 15625] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '0.14.0' is booting up.

But no "workers" announce their readiness (according to the code, they should).
Pressing ctrl-c causes the worker to become ready.

[2017-12-02 20:36:53,804] [PID 15625] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '0.14.0' is booting up.
^CException in thread Thread-3:
Traceback (most recent call last):
  File "/home/vagrant/.pyenv/versions/3.6.2/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/vagrant/.pyenv/versions/3.6.2/envs/project/lib/python3.6/site-packages/watchdog/observers/inotify_buffer.py", line 59, in run
    inotify_events = self._inotify.read_events()
  File "/home/vagrant/.pyenv/versions/3.6.2/envs/project/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 296, in read_events
    event_buffer = os.read(self._inotify_fd, event_buffer_size)
KeyboardInterrupt

[2017-12-02 20:36:54,680] [PID 15664] [MainThread] [dramatiq.WorkerProcess(0)] [INFO] Worker process is ready for action.

Note: i have 1088 tasks in my database some of them are being retried while others are queued for processing. Not sure if that helps debugging this issue.

[Bogdanp]

Are you running it in Docker via Docker-for-mac by any chance? I've run into the same issue because inotify doesn't seem to work correctly under DFM:

https://dramatiq.io/cookbook.html#watching-for-source-changes-in-docker-for-mac

If that's the case, I'll add support for using the poll file watcher to django_dramatiq and that should fix your issue. Let me know!

[rakanalh]

I noticed this issue on docker-for-mac a while ago... but because things were a bit slow, i switched to using vagrant.
--no-reload does seem to make things go fast again.

[Bogdanp]

What do you use to share folders between the host and the Vagrant machine? NFS and Samba would both have the same issue as DFM so I'm assuming it's one of those... Either way, I'll add that flag to django_dramatiq later today and let you know.

I'll have to see if there's a good way to detect whether or not inotify will work and downgrade to polling in cases it doesn't, but the flag should be good enough for now.

[rakanalh]

I did not configure the synced folders for vagrant to use anything special so i am using the default mode, which is virtualbox's shared folders. Not sure about the details there.

[Bogdanp]

I've added --reload-use-polling to django_dramatiq. I think that'll fix your issue, but you'll need to upgrade to Dramatiq 0.15 for the --watch-use-polling flag. Thanks for the report!

[rakanalh]

Thanks... things are back to normal now with the polling watcher. However, it's using considerably more CPU... but i think it's a compromise that one has to make with vagrant and docker-for-mac.

[Bogdanp]

@rakanalh I was wrong about the root cause here. The issue was actually the fact that watchdog's inotify implementation isn't compatible with gevent so it blocks the event loop. I wrote watchdog_gevent and integrated it into Dramatiq (0.15.1) today and that solves the issue.

[rakanalh]

Cool... thanks for the heads up. I'll upgrade and report issues if any.