NoCredentialsError when reading creds from instance metadata

[shuoh]

We have a script running on EC2 that keeps making DynamoDB API calls. Since recently the script always fails with the error below after successfully running for 1-2 hours :

  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/client.py", line 324, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/client.py", line 609, in _make_api_call
    operation_model, request_dict)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/endpoint.py", line 143, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/endpoint.py", line 168, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/endpoint.py", line 152, in create_request
    operation_name=operation_model.name)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/signers.py", line 156, in sign
    auth.add_auth(request)
  File "/home/ec2-user/buddyai_chatbot_deployed_app/app/venv/local/lib/python2.7/site-packages/botocore/auth.py", line 352, in add_auth
    raise NoCredentialsError
NoCredentialsError: Unable to locate credentials

I enabled boto debug log and find this particular line prior to the exception:

Mar 13 21:51:44 DEBUG Metadata service returned non 200 status code of 200 for url: http://169.254.169.254/latest/meta-data/iam/security-credentials/, content body:  [.../site-packages/botocore/utils.py@207]

So it looks like the root cause is that http://169.254.169.254/latest/meta-data/iam/security-credentials/ for some reasons returned 200 response with an empty body. Is this a known issue? I tested manually on the same ec2 instance by curl, and I can see a response body containing the role name that is currently configured, so I suppose this issue only occurs after some duration when the instance metadata is refreshed.

I also notice that in the official EC2 documentation, iam/security-credentials/ is no longer listed as a supported url. Is the boto team aware of this?

Another interesting finding is that in the latest code, InstanceMetadataFetcher will NOT honor the num_attempts parameter if the content is empty; it will directly bypass retry and return empty data. Because of this, I'll have to workaround this by adding my own retry on top of boto API calls.

[jamesls]

Yep I can confirm that a 200 response with an empty response is not being retried as it should be. There's also a related issue about non-json responses coming back from instance metadata: #1049

Marking as a bug, I think there's a few fixes we need to make to our InstanceMetadataFetcher. Thanks for reporting.

[ciarancourtney]

Related boto/boto3#1408