Service account info on GCE occasionally is missing keys.

[dhermes]

From googleapis/google-cloud-python#2314

Calls to _retrieve_info seem to be failing in a strange way, i.e. info is a valid dictionary but does not contain email.

[theacodes]

@matheuspatury it would be super helpful if we could get more details such as the response headers when this happens. Let me know if you're willing to help us reproduce.

@dhermes my best guess is some response from the metadata server that we're not expecting. I'll try to reproduce myself sometime this week.

[dhermes]

@jonparrott To repro I may just call _retrieve_info 10000 times and hope it fails one of them.

[matheuspatury]

So, I tried to reproduce the same error with the pseudo-code, but another error raised:

Traceback (most recent call last):
  File "test_datastore_client.py", line 32, in get_entities
    r = ds_client.get_multi(data)
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/client.py", line 301, in get_multi
    transaction_id=transaction and transaction.id,
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/client.py", line 123, in _extended_lookup
    transaction_id=transaction_id,
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 209, in lookup
    _datastore_pb2.LookupResponse)
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 124, in _rpc
    data=request_pb.SerializeToString())
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 93, in _request
    method='POST', headers=headers, body=data)
  File "/opt/python3/lib/python3.4/site-packages/oauth2client/transport.py", line 169, in new_request
    redirections, connection_type)
  File "/opt/python3/lib/python3.4/site-packages/httplib2/__init__.py", line 1314, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/python3/lib/python3.4/site-packages/httplib2/__init__.py", line 1064, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/python3/lib/python3.4/site-packages/httplib2/__init__.py", line 988, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 924, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 890, in send
    self.sock.sendall(data)
OSError: [Errno 9] Bad file descriptor

It also raised this error:

Traceback (most recent call last):
  File "test_datastore_client.py", line 32, in get_entities
    r = ds_client.get_multi(data)
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/client.py", line 301, in get_multi
    transaction_id=transaction and transaction.id,
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/client.py", line 123, in _extended_lookup
    transaction_id=transaction_id,
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 209, in lookup
    _datastore_pb2.LookupResponse)
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 124, in _rpc
    data=request_pb.SerializeToString())
  File "/opt/python3/lib/python3.4/site-packages/gcloud/datastore/connection.py", line 93, in _request
    method='POST', headers=headers, body=data)
  File "/opt/python3/lib/python3.4/site-packages/oauth2client/transport.py", line 169, in new_request
    redirections, connection_type)
  File "/opt/python3/lib/python3.4/site-packages/httplib2/__init__.py", line 1314, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/python3/lib/python3.4/site-packages/httplib2/__init__.py", line 1064, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/python3/lib/python3.4/site-packages/httplib2/__init__.py", line 1017, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python3.4/http/client.py", line 1172, in getresponse
    response.begin()
  File "/usr/lib/python3.4/http/client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.4/http/client.py", line 321, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: ''

(Pseudo-code)

from gcloud import datastore,bigquery
from datetime import datetime,timedelta
import threading
import logging

logging_path = '/var/log/testing.log'
logging.basicConfig(
    filename=logging_path,
    filemode='w',
    level=logging.DEBUG,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")

locker = threading.Lock()
dataQueue = queue.Queue()
ds_client = datastore.Client()
bq_client = bigquery.Client()
entities = []

def get_entities(num_thread,entities):
    logging.info('[thread-%d]: starting',num_thread)
    while True:
        try:
            data = dataQueue.get(block=False)
        except queue.Empty:
            break
        else:
            try:
                r = ds_client.get_multi(data)
            except Exception:
                logging.exception('[thread-%d]: error while fetching %s',
                        num_thread,
                        repr(data))
                break
            else:
                with locker:
                    entities.append(r)
                logging.info('[thread-%d]: added %d entities',
                    num_thread,
                    len(r))


def main():
    key = ds_client.key('Account',73170001)
    keys = [key for x in range(3)]
    dataQueue = queue.Queue()
    [dataQueue.put(keys) for x in range(100)]
    threads = []
    for i in range(10):
        thread = threading.Thread(
            target=get_entities,
            args=(i,entities))
        thread.start()
        threads.append(thread)
    for thread in threads: thread.join()

if __name__ == '__main__':
    main()

@jonparrott I changed my code now. Every time a new thread is created, I'm creating a new datastore client, and there is no errors so far; however, I do not believe that the error I described first should happen either way.

[dhermes]

@matheuspatury That error is because httplib2 is not threadsafe. @jonparrott and I are working on a divorce from httplib2 (#128 and googleapis/google-cloud-python#1214)

[theacodes]

Closing this for now, resolution of #128 will close this for good.

[dhermes]

@jonparrott that was an aside, the real issue is with GCE.

[theacodes]

Gotcha

[theacodes]

Thank you for creating this issue, however, this project is deprecatedand we will only be addressing critical security issues. You can read moreabout this deprecation here.

If you need support or help using this library, we recommend that you ask yourquestion on StackOverflow.

If you still think this issue is relevant and should be addressed, pleasecomment and let us know!