Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws s3 (as opposed to s3api) not honoring --no-verify-ssl #696

Closed
asieira opened this issue Mar 7, 2014 · 4 comments · Fixed by #698
Closed

aws s3 (as opposed to s3api) not honoring --no-verify-ssl #696

asieira opened this issue Mar 7, 2014 · 4 comments · Fixed by #698

Comments

@asieira
Copy link
Contributor

asieira commented Mar 7, 2014

Testing aws-cli/1.3.1 Python/2.7.5 Windows/2008ServerR2 in an environment where a corporate security solution acts as a 'man in the middle' for all HTTPS communications. So I need to be either able to add a new trusted CA certificate or disable certificate CA validation for the AWS endpoints entirely.

Running this on Windows, even when passing the --no-verify-ssl option the command still fails because of the 'invalid' remote certificate.

Here is the full debug output:

C:\Windows>aws --debug --no-verify-ssl s3 ls s3://redactedbucketname/upload
2014-03-07 11:56:48,132 - awscli.clidriver - DEBUG - CLI version: aws-cli/1.3.1
Python/2.7.5 Windows/2008ServerR2, botocore version: 0.35.0
2014-03-07 11:56:48,132 - botocore.service - DEBUG - Creating service object for
: s3
2014-03-07 11:56:48,132 - botocore.base - DEBUG - Attempting to load: aws/s3
2014-03-07 11:56:48,164 - botocore.base - DEBUG - Found data file: C:\Program Fi
les\Amazon\AWSCLI\botocore\data\aws/s3.json
2014-03-07 11:56:48,164 - botocore.hooks - DEBUG - Event service-data-loaded.s3:
 calling handler <function signature_overrides at 0x00000000029B8978>
2014-03-07 11:56:48,164 - botocore.hooks - DEBUG - Event service-created: callin
g handler <function register_retries_for_service at 0x00000000029B87B8>
2014-03-07 11:56:48,164 - botocore.handlers - DEBUG - Registering retry handlers
 for service: Service(s3)
2014-03-07 11:56:48,164 - botocore.hooks - DEBUG - Event creating-endpoint.s3: c
alling handler <function maybe_switch_to_s3sigv4 at 0x00000000029B8898>
2014-03-07 11:56:48,164 - botocore.credentials - INFO - Found credentials in con
fig file.
2014-03-07 11:56:48,164 - botocore.service - DEBUG - Creating operation objects
for: Service(s3)
2014-03-07 11:56:48,257 - botocore.operation - DEBUG - Operation:ListObjects cal
led with kwargs: {'prefix': u'upload', 'bucket': u'redactedbucketname', 'delimiter':
 '/'}
2014-03-07 11:56:48,257 - botocore.operation - DEBUG - Creating parameter object
s for: Operation:ListObjects
2014-03-07 11:56:48,257 - botocore.endpoint - DEBUG - Making request for Operati
on:ListObjects (verify_ssl=True) with params: {'headers': {}, 'uri_params': {u'D
elimiter': '/', u'Bucket': u'redactedbucketname', u'Prefix': u'upload'}, 'payload':
<botocore.payload.XMLPayload object at 0x0000000003401A90>}
2014-03-07 11:56:48,257 - botocore.endpoint - DEBUG - Building URI for rest endp
oint.
2014-03-07 11:56:48,257 - botocore.endpoint - DEBUG - Templated URI path: /{Buck
et}
2014-03-07 11:56:48,257 - botocore.endpoint - DEBUG - Templated URI query_params
: delimiter={Delimiter}&marker={Marker}&max-keys={MaxKeys}&prefix={Prefix}&encod
ing-type={EncodingType}
2014-03-07 11:56:48,257 - botocore.endpoint - DEBUG - Rendered path: /redactedbu
cketname
2014-03-07 11:56:48,257 - botocore.endpoint - DEBUG - Rendered query_params: del
imiter=/&prefix=upload
2014-03-07 11:56:48,257 - botocore.hooks - DEBUG - Event before-auth.s3: calling
 handler <function fix_s3_host at 0x00000000029B86D8>
2014-03-07 11:56:48,257 - botocore.handlers - DEBUG - Checking for DNS compatibl
e bucket for: https://s3.amazonaws.com/redactedbucketname?delimiter=/&prefix=upl
oad
2014-03-07 11:56:48,257 - botocore.handlers - DEBUG - URI updated to: https://ne
vermind-logs.s3.amazonaws.com?delimiter=/&prefix=upload
2014-03-07 11:56:48,257 - botocore.auth - DEBUG - Calculating signature using hm
acv1 auth.
2014-03-07 11:56:48,257 - botocore.auth - DEBUG - HTTP request method: GET
2014-03-07 11:56:48,257 - botocore.auth - DEBUG - StringToSign:
GET


Fri, 07 Mar 2014 19:56:48 GMT
/redactedbucketname/
2014-03-07 11:56:48,273 - botocore.endpoint - DEBUG - Sending http request: <Pre
paredRequest [GET]>
2014-03-07 11:56:48,273 - botocore.vendored.requests.packages.urllib3.connection
pool - INFO - Starting new HTTPS connection (1): redactedbucketname.s3.amazonaws
.com

2014-03-07 11:56:49,446 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObje
cts: calling handler <botocore.retryhandler.RetryHandler object at 0x0000000002D
4DA90>
2014-03-07 11:56:49,446 - botocore.retryhandler - DEBUG - retry needed, retryabl
e exception caught: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_S
ERVER_CERTIFICATE:certificate verify failed
Traceback (most recent call last):
  File "botocore\retryhandler.pyc", line 262, in _should_retry
  File "botocore\retryhandler.pyc", line 310, in __call__
  File "botocore\retryhandler.pyc", line 219, in __call__
  File "botocore\retryhandler.pyc", line 352, in _check_caught_exception
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERT
IFICATE:certificate verify failed
2014-03-07 11:56:49,446 - botocore.retryhandler - DEBUG - Retry needed, action o
f: 0.626861716161
2014-03-07 11:56:49,446 - botocore.endpoint - DEBUG - Response received to retry
, sleeping for 0.626861716161 seconds
2014-03-07 11:56:50,085 - botocore.endpoint - DEBUG - Sending http request: <Pre
paredRequest [GET]>
2014-03-07 11:56:50,085 - botocore.vendored.requests.packages.urllib3.connection
pool - INFO - Starting new HTTPS connection (2): redactedbucketname.s3.amazonaws
.com

2014-03-07 11:56:50,335 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObje
cts: calling handler <botocore.retryhandler.RetryHandler object at 0x0000000002D
4DA90>
2014-03-07 11:56:50,335 - botocore.retryhandler - DEBUG - retry needed, retryabl
e exception caught: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_S
ERVER_CERTIFICATE:certificate verify failed
Traceback (most recent call last):
  File "botocore\retryhandler.pyc", line 262, in _should_retry
  File "botocore\retryhandler.pyc", line 310, in __call__
  File "botocore\retryhandler.pyc", line 219, in __call__
  File "botocore\retryhandler.pyc", line 352, in _check_caught_exception
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERT
IFICATE:certificate verify failed
2014-03-07 11:56:50,335 - botocore.retryhandler - DEBUG - Retry needed, action o
f: 0.0473785082195
2014-03-07 11:56:50,335 - botocore.endpoint - DEBUG - Response received to retry
, sleeping for 0.0473785082195 seconds
2014-03-07 11:56:50,398 - botocore.endpoint - DEBUG - Sending http request: <Pre
paredRequest [GET]>
2014-03-07 11:56:50,398 - botocore.vendored.requests.packages.urllib3.connection
pool - INFO - Starting new HTTPS connection (3): redactedbucketname.s3.amazonaws
.com

2014-03-07 11:56:50,696 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObje
cts: calling handler <botocore.retryhandler.RetryHandler object at 0x0000000002D
4DA90>
2014-03-07 11:56:50,696 - botocore.retryhandler - DEBUG - retry needed, retryabl
e exception caught: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_S
ERVER_CERTIFICATE:certificate verify failed
Traceback (most recent call last):
  File "botocore\retryhandler.pyc", line 262, in _should_retry
  File "botocore\retryhandler.pyc", line 310, in __call__
  File "botocore\retryhandler.pyc", line 219, in __call__
  File "botocore\retryhandler.pyc", line 352, in _check_caught_exception
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERT
IFICATE:certificate verify failed
2014-03-07 11:56:50,696 - botocore.retryhandler - DEBUG - Retry needed, action o
f: 1.79773347922
2014-03-07 11:56:50,696 - botocore.endpoint - DEBUG - Response received to retry
, sleeping for 1.79773347922 seconds
2014-03-07 11:56:52,507 - botocore.endpoint - DEBUG - Sending http request: <Pre
paredRequest [GET]>
2014-03-07 11:56:52,507 - botocore.vendored.requests.packages.urllib3.connection
pool - INFO - Starting new HTTPS connection (4): redactedbucketname.s3.amazonaws
.com

2014-03-07 11:56:52,773 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObje
cts: calling handler <botocore.retryhandler.RetryHandler object at 0x0000000002D
4DA90>
2014-03-07 11:56:52,773 - botocore.retryhandler - DEBUG - retry needed, retryabl
e exception caught: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_S
ERVER_CERTIFICATE:certificate verify failed
Traceback (most recent call last):
  File "botocore\retryhandler.pyc", line 262, in _should_retry
  File "botocore\retryhandler.pyc", line 310, in __call__
  File "botocore\retryhandler.pyc", line 219, in __call__
  File "botocore\retryhandler.pyc", line 352, in _check_caught_exception
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERT
IFICATE:certificate verify failed
2014-03-07 11:56:52,773 - botocore.retryhandler - DEBUG - Retry needed, action o
f: 7.28444392423
2014-03-07 11:56:52,773 - botocore.endpoint - DEBUG - Response received to retry
, sleeping for 7.28444392423 seconds
2014-03-07 11:57:00,071 - botocore.endpoint - DEBUG - Sending http request: <Pre
paredRequest [GET]>
2014-03-07 11:57:00,071 - botocore.vendored.requests.packages.urllib3.connection
pool - INFO - Starting new HTTPS connection (5): redactedbucketname.s3.amazonaws
.com

2014-03-07 11:57:00,321 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObje
cts: calling handler <botocore.retryhandler.RetryHandler object at 0x0000000002D
4DA90>
2014-03-07 11:57:00,321 - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli\clidriver.pyc", line 188, in main
  File "awscli\customizations\s3\s3.pyc", line 151, in __call__
  File "awscli\customizations\s3\s3.pyc", line 262, in __call__
  File "awscli\customizations\s3\s3.pyc", line 357, in _do_command
  File "awscli\customizations\s3\s3.pyc", line 364, in _list_all_objects
  File "botocore\paginate.pyc", line 141, in __iter__
  File "botocore\operation.pyc", line 68, in call
  File "botocore\endpoint.pyc", line 76, in make_request
  File "botocore\endpoint.pyc", line 98, in _send_request
  File "botocore\endpoint.pyc", line 129, in _needs_retry
  File "botocore\session.pyc", line 621, in emit_first_non_none_response
  File "botocore\hooks.pyc", line 150, in emit
  File "botocore\retryhandler.pyc", line 180, in __call__
  File "botocore\retryhandler.pyc", line 247, in __call__
  File "botocore\retryhandler.pyc", line 270, in _should_retry
  File "botocore\retryhandler.pyc", line 310, in __call__
  File "botocore\retryhandler.pyc", line 219, in __call__
  File "botocore\retryhandler.pyc", line 352, in _check_caught_exception
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERT
IFICATE:certificate verify failed
2014-03-07 11:57:00,321 - awscli.clidriver - DEBUG - Exiting with rc 255

[Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:ce
rtificate verify failed
@jamesls
Copy link
Member

jamesls commented Mar 7, 2014

I think the issue is that because s3 is a high level custom command, we didn't port the verify logic over to this command. Can you verify that using --no-verify-ssl with s3api works as expected?

@asieira
Copy link
Contributor Author

asieira commented Mar 7, 2014

You're absolutely right. Running aws --no-ssl-verify s3api list-objects --bucket redactedbucketname on the same environment works perfectly.

It would be really valuable to me if aws-cli also honored this option on the s3 high level commands.

@asieira
Copy link
Contributor Author

asieira commented Mar 7, 2014

BTW, +1 for how quickly you responded to this issue. :)

jamesls added a commit to jamesls/aws-cli that referenced this issue Mar 10, 2014
@jamesls
Update s3 high level commands to honor --no-verify-ssl
2b755f0 
Fixes aws#696.
@jamesls jamesls mentioned this issue Mar 10, 2014
Merged
jamesls added a commit that referenced this issue Mar 10, 2014
@jamesls
Add #696 to changelog
d29b2f1
jamesls added a commit that referenced this issue Mar 10, 2014
@jamesls
Merge branch 's3-ssl-verify' into develop
fd79a55 
* s3-ssl-verify:
  Add #696 to changelog
  Update s3 high level commands to honor --no-verify-ssl
  Add customization to allow for "types" of top level args
@asieira
Copy link
Contributor Author

asieira commented Mar 11, 2014

+1 again for the responsiveness! Can't wait for the next release. :) Thanks guys!

danielgtaylor added a commit that referenced this issue Mar 20, 2014
@danielgtaylor
Merge branch 'release-1.3.2'
ebb8ddc 
* release-1.3.2:
  Bumping version to 1.3.2
  Add entry to changelog
  Account for list values in def->api translation
  Updated for the underlying multiple version support.
  Disable newline integ test on windows
  Add space in dryrundocs
  Update changelog
  Fix comment in test
  Add integ test for removing file with newlines
  URLEncode keys
  Split out object listing into separate object
  Don't compile jmespath expression in formatter
  Add #696 to changelog
  Update s3 high level commands to honor --no-verify-ssl
  Add customization to allow for "types" of top level args
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update s3 high level commands to honor "--no-verify-ssl" jamesls/aws-cli
2 participants
@jamesls @asieira