Airflow webserver using gevent halts booting workers if statsd_on = True

[trentonstrong]

Dear Airflow Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Airflow version:
  Release 1.6.2
• Setup:
  Using LocalExecutor, running webserver with gevent workers.
• Code snippet

[webserver]
worker_class = gevent
...
[scheduler]
statsd_on =  True

• Stack trace if applicable:
• Operating System: Linux ip-172-31-73-4 3.19.0-25-generic Backfill start_date to override the tasks's start_date #26-Ubuntu SMP Fri Jul 24 21:17:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 2.7.9

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

• What did you expect to happen?
  Statsd integration to report metrics to statsd, webserver to boot.
• What happened instead?
  Webserver hangs on boot, with exception:

Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: Traceback (most recent call last):
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: worker.init_process()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 192, in init_process
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: super(GeventWorker, self).init_process()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/workers/base.py", line 118, in init_process
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: self.wsgi = self.app.wsgi()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: self.callable = self.load()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: return self.load_wsgiapp()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: return util.import_app(self.app_uri)
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gunicorn/util.py", line 355, in import_app
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: __import__(module)
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/airflow/www/app.py", line 13, in <module>
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: from airflow import jobs
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/airflow/jobs.py", line 37, in <module>
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: prefix=configuration.get('scheduler', 'statsd_prefix'))
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/statsd/client.py", line 139, in __init__
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: host, port, fam, socket.SOCK_DGRAM)[0]
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gevent/socket.py", line 623, in getaddrinfo
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gevent/threadpool.py", line 222, in apply_e
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gevent/event.py", line 233, in get
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: result = self.hub.switch()
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: File "/opt/pipeline/pipeline/lib/python2.7/site-packages/gevent/hub.py", line 338, in switch
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: return greenlet.switch(self)
Apr 15 19:29:45 ip-172-31-73-4 with_env.sh[19235]: AssertionError: Impossible to call blocking function in the event loop callback

• Here is how you can reproduce this issue on your machine:

  Reproduction Steps

1. Install airflow release.
2. Install gevent
3. Configure airflow.cfg to use gevent workers
4. Enable statsd_on
5. Attempt to boot airflow webserver.