Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading S3 object encrypted using KMS keys fails with error: "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4" #1006

Closed
maitreya1975 opened this issue Nov 16, 2014 · 1 comment · Fixed by #1012
Labels
bug This issue is a bug.

Comments

@maitreya1975
Copy link

I uploaded an object to S3 encrypted with a KMS managed key using the S3 Console.
I am using:

$ aws --version
aws-cli/1.6.2 Python/2.7.6 Darwin/13.4.0

When I tried to download the object using aws-cli, I got the following error:

aws s3 cp s3://xxxx/yyyy.jpg ./
download failed: s3://xxxx/yyyy..jpg to ./ngc6960_FinalPugh900.jpg A client error (InvalidArgument) occurred when calling the GetObject operation: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.

I get the same error when using the low level s3api:

685b3588202f:lego_iam maitreyr$ aws s3api get-object --bucket maitreyr-kms-test --key ngc6960_FinalPugh900.jpg ./ngc.jpg --debug
2014-11-16 10:58:31,128 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.6.2 Python/2.7.6 Darwin/13.4.0, botocore version: 0.73.0
2014-11-16 10:58:31,128 - MainThread - botocore.service - DEBUG - Creating service object for: s3
2014-11-16 10:58:31,156 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function signature_overrides at 0x108628398>
2014-11-16 10:58:31,156 - MainThread - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x108628230>
2014-11-16 10:58:31,159 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-11-16 10:58:31,160 - MainThread - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2014-11-16 10:58:31,168 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function inject_assume_role_provider at 0x1089759b0>
2014-11-16 10:58:31,168 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function add_waiters at 0x108980758>
2014-11-16 10:58:31,173 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'bucket', <awscli.arguments.CLIArgument object at 0x108ab3550>), (u'if-match', <awscli.arguments.CLIArgument object at 0x108ab3590>), (u'if-modified-since', <awscli.arguments.CLIArgument object at 0x108ab35d0>), (u'if-none-match', <awscli.arguments.CLIArgument object at 0x108ab3610>), (u'if-unmodified-since', <awscli.arguments.CLIArgument object at 0x108ab3650>), (u'key', <awscli.arguments.CLIArgument object at 0x108ab3690>), (u'range', <awscli.arguments.CLIArgument object at 0x108ab36d0>), (u'response-cache-control', <awscli.arguments.CLIArgument object at 0x108ab3710>), (u'response-content-disposition', <awscli.arguments.CLIArgument object at 0x108ab3750>), (u'response-content-encoding', <awscli.arguments.CLIArgument object at 0x108ab37d0>), (u'response-content-language', <awscli.arguments.CLIArgument object at 0x108ab3810>), (u'response-content-type', <awscli.arguments.CLIArgument object at 0x108ab3850>), (u'response-expires', <awscli.arguments.CLIArgument object at 0x108ab3890>), (u'version-id', <awscli.arguments.CLIArgument object at 0x108ab38d0>), (u'sse-customer-algorithm', <awscli.arguments.CLIArgument object at 0x108ab3910>), (u'sse-customer-key', <awscli.arguments.CLIArgument object at 0x108ab3950>), (u'sse-customer-key-md5', <awscli.arguments.CLIArgument object at 0x108ab3990>), (u'ssekms-key-id', <awscli.arguments.CLIArgument object at 0x108ab39d0>)])
2014-11-16 10:58:31,173 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_streaming_output_arg at 0x108894320>
2014-11-16 10:58:31,175 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_cli_input_json at 0x10896db90>
2014-11-16 10:58:31,175 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function unify_paging_params at 0x1088971b8>
2014-11-16 10:58:31,175 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_generate_skeleton at 0x108975140>
2014-11-16 10:58:31,177 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.bucket: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,177 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.get-object: calling handler <awscli.argprocess.ParamShorthand object at 0x1089b1450>
2014-11-16 10:58:31,177 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2014-11-16 10:58:31,178 - MainThread - awscli.arguments - DEBUG - Unpacked value of "maitreyr-kms-test" for parameter "bucket": maitreyr-kms-test
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-match: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-modified-since: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-none-match: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-unmodified-since: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.key: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.get-object: calling handler <awscli.argprocess.ParamShorthand object at 0x1089b1450>
2014-11-16 10:58:31,178 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2014-11-16 10:58:31,178 - MainThread - awscli.arguments - DEBUG - Unpacked value of "ngc6960_FinalPugh900.jpg" for parameter "key": ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.range: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-cache-control: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-disposition: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-encoding: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-language: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-type: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-expires: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.version-id: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.sse-customer-algorithm: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.sse-customer-key: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.sse-customer-key-md5: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.ssekms-key-id: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.outfile: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2014-11-16 10:58:31,180 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2014-11-16 10:58:31,180 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2014-11-16 10:58:31,181 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2014-11-16 10:58:31,219 - MainThread - botocore.operation - DEBUG - Operation:GetObject called with kwargs: {u'Bucket': u'maitreyr-kms-test', u'Key': u'ngc6960_FinalPugh900.jpg'}
2014-11-16 10:58:31,221 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function sse_md5 at 0x108628050>
2014-11-16 10:58:31,222 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x108628410>
2014-11-16 10:58:31,222 - MainThread - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x108aac490> (verify_ssl=True) with params: {'query_string': {}, 'headers': {}, 'url_path': u'/maitreyr-kms-test/ngc6960_FinalPugh900.jpg', 'body': '', 'method': u'GET'}
2014-11-16 10:58:31,222 - MainThread - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x108628140>
2014-11-16 10:58:31,222 - MainThread - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-2.amazonaws.com/maitreyr-kms-test/ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,222 - MainThread - botocore.handlers - DEBUG - URI updated to: https://maitreyr-kms-test.s3.amazonaws.com/ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,222 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-11-16 10:58:31,223 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2014-11-16 10:58:31,223 - MainThread - botocore.auth - DEBUG - StringToSign:
GET


Sun, 16 Nov 2014 16:58:31 GMT
/maitreyr-kms-test/ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,230 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2014-11-16 10:58:31,231 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): maitreyr-kms-test.s3.amazonaws.com
2014-11-16 10:58:31,481 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /ngc6960_FinalPugh900.jpg HTTP/1.1" 307 None
2014-11-16 10:58:31,483 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): maitreyr-kms-test.s3-us-west-2.amazonaws.com
2014-11-16 10:58:32,419 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /ngc6960_FinalPugh900.jpg HTTP/1.1" 400 None
2014-11-16 10:58:32,423 - MainThread - botocore.parsers - DEBUG - Response headers:
{'connection': 'close',
 'content-type': 'application/xml',
 'date': 'Sun, 16 Nov 2014 16:58:17 GMT',
 'server': 'AmazonS3',
 'transfer-encoding': 'chunked',
 'x-amz-id-2': 'VCLi4Tph0mzbatYuzb3vNY16/eY0hJn9KcN/oMlMbLYS/gMBQI1+YxvpeatReZ1PLzp8ahhN1HQ=',
 'x-amz-request-id': 'AD42152AA3A28C6A'}
2014-11-16 10:58:32,423 - MainThread - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>null</ArgumentValue><RequestId>AD42152AA3A28C6A</RequestId><HostId>VCLi4Tph0mzbatYuzb3vNY16/eY0hJn9KcN/oMlMbLYS/gMBQI1+YxvpeatReZ1PLzp8ahhN1HQ=</HostId></Error>
2014-11-16 10:58:32,424 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x108a1f850>
2014-11-16 10:58:32,424 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2014-11-16 10:58:32,424 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <bound method StreamingOutputArgument.save_file of <awscli.customizations.streamingoutputarg.StreamingOutputArgument object at 0x108ab3a10>>
2014-11-16 10:58:32,424 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 359, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 494, in __call__
    self._operation_object, call_parameters, parsed_globals)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 599, in invoke
    **parameters)
  File "/usr/local/lib/python2.7/site-packages/botocore/operation.py", line 98, in call
    parsed=response[1])
  File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 729, in emit
    return self._events.emit(event_name, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/hooks.py", line 185, in emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/awscli/customizations/streamingoutputarg.py", line 92, in save_file
    body = parsed[self._response_key]
KeyError: u'Body'
2014-11-16 10:58:32,425 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

u'Body'
@jamesls jamesls added confirmed bug This issue is a bug. labels Nov 18, 2014
@jamesls
Copy link
Member

jamesls commented Nov 19, 2014

Looks like there's two issues here. The first issue is fixed via boto/botocore#382, which is that there needs to be a way to enable signature version 4 for S3. You can do this by running:

aws configure set default.s3.signature_version s3v4

# Or for a specific profile
aws configure set profile.<profilename>.s3.signature_version s3v4

The second issue is that we aren't handling the specific case where we're streaming to a file and receive an error response from S3, which I'm fixing now.

jamesls added a commit to jamesls/aws-cli that referenced this issue Nov 19, 2014
Fixes the second half of aws#1006.  Now you'll get a proper error
message:

```
$ aws s3api get-object --bucket b --key k /tmp/outfile

A client error (InvalidRequest) occurred when calling the GetObject
operation: The object was stored using a form of Server Side Encryption.
The correct parameters must be provided to retrieve the object.

```

Closes aws#1006.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants