Fix rare race condition with large datasets

[sadsfae]

We've noticed a very rare race condition that can cause incorrect notifications to get sent out if the 7,5,3,1 day expirations are checked while we're making heavy use of /opt/quads/bin/quads-cli

This also may affect other notifications.

We will try implementing a timeout/retry to both urllib and requests library calls to work around this.

[sadsfae]

The bug seems to show up with a large yaml dataset (240kb / 10k lines+).

Traceback (most recent call last):
  File "/tmp/quadsVoV7/quads/bin/quads-cli", line 455, in <module>
    main(sys.argv[1:])
  File "/tmp/quadsVoV7/quads/bin/quads-cli", line 445, in main
    js = r.json()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 866, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[sadsfae]

and here is quads-daemon service side trace:

127.0.0.1 - - [06/Sep/2017 17:47:56] "POST /api/v1/query HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 50454)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/SocketServer.py", line 596, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/tmp/quadsGKFi/quads/bin/quads-daemon", line 650, in do_POST
    cloud_summary = quads.query_cloud_summary(data['date'][0], True)
  File "/tmp/quadsGKFi/quads/lib/Quads.py", line 798, in query_cloud_summary
    summary = self.query_cloud_hosts(datearg)
  File "/tmp/quadsGKFi/quads/lib/Quads.py", line 759, in query_cloud_hosts
    default_cloud, current_cloud, current_override = self.find_current(h, datearg)
  File "/tmp/quadsGKFi/quads/lib/Quads.py", line 198, in find_current
    start_obj = datetime.datetime.strptime(hosts[host]["schedule"][override]["start"], '%Y-%m-%d %H:%M')
TypeError: 'NoneType' object has no attribute '__getitem__'
----------------------------------------

[sadsfae]

Update: it seems moving to urllib2.urlopen(url, timeout=1200) does help but we're still sometimes accessing data in memory that has not yet been written to disk. We need one more safeguard here in the fcntl library code to ensure we flush before things are queried.