Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError: 'RequestID' getting ec2 instances #339

Closed
jsternberg opened this issue Nov 2, 2015 · 9 comments
Closed

KeyError: 'RequestID' getting ec2 instances #339

jsternberg opened this issue Nov 2, 2015 · 9 comments
Assignees
@jamesls
Labels
bug This issue is a confirmed bug. ec2

Comments

@jsternberg
Copy link

When retrieving ec2 instances, we will occasionally get the following error from boto3:

  File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 82, in __iter__
    for page in self.pages():
  File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 165, in pages
    for page in pages:
  File "/usr/local/lib/python2.7/site-packages/botocore/paginate.py", line 83, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/paginate.py", line 155, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 301, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 375, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python2.7/site-packages/botocore/endpoint.py", line 111, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python2.7/site-packages/botocore/endpoint.py", line 138, in _send_request
    request, operation_model, attempts)
  File "/usr/local/lib/python2.7/site-packages/botocore/endpoint.py", line 193, in _get_response
    operation_model.output_shape)),
  File "/usr/local/lib/python2.7/site-packages/botocore/parsers.py", line 206, in parse
    parsed = self._do_error_parse(response, shape)
  File "/usr/local/lib/python2.7/site-packages/botocore/parsers.py", line 459, in _do_error_parse
    'RequestId': original.pop('RequestID')
KeyError: 'RequestID'

It appears some error happened on Amazon's side and there's no RequestID in the returned request. I have no reproducer since this happens intermittently. It might be worth having a default value of None if the RequestID doesn't exist in the response though.
@kyleknap
Copy link
Contributor

kyleknap commented Nov 3, 2015

Interesting. Is it possible we could get a debug log of this? It would be nice to see if the 'RequestID' is completely missing or just misplaced on the error response. To get debug logs, all you would have to do is add boto3.set_stream_logger('botocore') to right before you make the ec2 call.

@kyleknap kyleknap added response-requested Waiting on additional information or feedback. investigating This issue is being investigated and/or work is in progress to resolve the issue. ec2 labels Nov 3, 2015
@jsternberg
Copy link
Author

Unfortunately it happened in an alert and it's not easily reproducible since it likely requires AWS to fail. We have also modified the alert since we didn't need to use the AWS API.

@kyleknap
Copy link
Contributor

kyleknap commented Nov 3, 2015

Alright we will need to investigate it then. Are you just using a DescribeInstances operation or the instances collection on the service resource?

@jsternberg
Copy link
Author

It was using the instances collection on the service resource.

@kyleknap
Copy link
Contributor

kyleknap commented Nov 3, 2015

Thanks. Out of curiosity do you have a lot of instances to iterate over in the collection? I wonder if that is why the RequestId is missing.

@jsternberg
Copy link
Author

I think there are about 3 instances to iterate over, so we don't have a lot of instances to iterate over.

@kyleknap
Copy link
Contributor

kyleknap commented Nov 3, 2015

Thanks! We will look into this then.

@kyleknap kyleknap removed the response-requested Waiting on additional information or feedback. label Nov 3, 2015
@jonapich
Copy link

I have also stumbled upon this recently and got a bit of extra info (I caught it through sentry):

File "task/aws_utils.py", line 397, in describe_instances
    Filters=filters, NextToken=next_token)
  File "botocore/client.py", line 228, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "botocore/client.py", line 475, in _make_api_call
    operation_model, request_dict)
  File "botocore/endpoint.py", line 117, in make_request
    return self._send_request(request_dict, operation_model)
  File "botocore/endpoint.py", line 144, in _send_request
    request, operation_model, attempts)
  File "botocore/endpoint.py", line 203, in _get_response
    parser.parse(response_dict, operation_model.output_shape)),
  File "botocore/parsers.py", line 206, in parse
    parsed = self._do_error_parse(response, shape)
  File "botocore/parsers.py", line 468, in _do_error_parse
    'RequestId': original.pop('RequestID')

(sorry about the parsing for the following...)

original 'body' 'b' 'Http/1.1 Service Unavailable'
response    'body' '<html><body><b>Http/1.1 Service Unavailable</b></body> </html>'
'headers' {'content-length': '62', 'connection': 'close', 'pragma': 'no-cache', 'cache-control': 'no-cache,no-store'}
'status_code' 503

@jamesls
Copy link
Member

jamesls commented Mar 21, 2016

Thanks! This should be enough for us to implement a fix.

@jamesls jamesls added bug This issue is a confirmed bug. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Mar 21, 2016
@jamesls jamesls self-assigned this Mar 23, 2016
@jamesls jamesls mentioned this issue Mar 23, 2016
Merged
jamesls added a commit to jamesls/botocore that referenced this issue Mar 23, 2016
@jamesls
Handle the case where we receive a generic html error response
afe0ace 
Fixes boto/boto3#339.

I also had to fix an unrelated test because the response
body is always read as bytes(), and two of the functional
tests were using str().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jamesls jamesls

Labels
bug This issue is a confirmed bug. ec2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jamesls @jsternberg @kyleknap @jonapich