Handle errors/garbage coming back for metadata server creds

[kapilt]

We've noticed that the ec2 metadata server isn't particularly stable with any concurrency, which in turn can cause exceptions in botocore, and in our app lockups.

metadata server concurrency issues

• https://ahmetalpbalkan.com/blog/comparison-of-instance-metadata-services/
• provider-aws: Intermittent Race Condition with Ec2 Metadata service/credentials hashicorp/terraform#6222

botocore traceback on retrieval

  File "/usr/local/custodian/local/lib/python2.7/site-packages/boto3/session.py", line 256, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/custodian/local/lib/python2.7/site-packages/botocore/session.py", line 815, in create_client
    credentials = self.get_credentials()
  File "/usr/local/custodian/local/lib/python2.7/site-packages/botocore/session.py", line 443, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/custodian/local/lib/python2.7/site-packages/botocore/credentials.py", line 1072, in load_credentials
    creds = provider.load()
  File "/usr/local/custodian/local/lib/python2.7/site-packages/botocore/credentials.py", line 477, in load
    metadata = fetcher.retrieve_iam_role_credentials()
  File "/usr/local/custodian/local/lib/python2.7/site-packages/botocore/utils.py", line 187, in retrieve_iam_role_credentials
    val = json.loads(val)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 369, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 9 column 2 - line 9 column 14 (char 846 - 858)

Ideally botocore should catch this on relevant error codes and retry with delay.

Source issue: cloud-custodian/cloud-custodian#506

[JordonPhillips]

Is the metadata service sending back an error response or just garbled data?

[kapilt]

its unclear, from the other links it looks like its sending back a non 200 error code and non json.

[joshuadfranklin]

Specifically a HTTP/1.1 429 Too Many Requests
"@phinze I was able to confirm that the metadata service does start sending 429s when it's being hit heavily. In my concurrency test it seems to break down after 25 concurrent connections are made."

[jamesls]

Yep, we should have more robust error handling in this case. Marking as bug.

[jhnlsn]

Just wanted to add a little more context to this issue. Looking at the code execution path, it looks like the _get_request call is getting back a 200 with a partial JSON response body. Allowing if val[0] == '{': to see the starting bracket. However, the full string passed to json.loads is not a valid json object, causing the failure noted in the traceback above.