Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws kms has 'utf8' codec can't decode byte error #1001

Closed
mtougeron opened this issue Nov 13, 2014 · 5 comments · Fixed by #1015
Closed

aws kms has 'utf8' codec can't decode byte error #1001

mtougeron opened this issue Nov 13, 2014 · 5 comments · Fixed by #1015
Labels
bug This issue is a bug.

Comments

@mtougeron
Copy link

I upgraded the awscli this afternoon to try and take advantage of the new kms system. I'm getting the following error when I try to use the cli.

$> python -V
Python 2.7.6
$> aws kms generate-random --number-of-bytes 128                                
{
    "Plaintext": 
'utf8' codec can't decode byte 0xee in position 0: invalid continuation byte
$> python -V
Python 2.7.6
$> aws kms encrypt --key-id REDACTED --plaintext foo
{
    "KeyId": "arn:aws:kms:us-west-1:REDACTED:key/REDACTED", 
    "CiphertextBlob": 
'utf8' codec can't decode byte 0xf4 in position 3: invalid continuation byte

pip modules installed:

Babel==1.3
Fabric==1.10.0
Jinja2==2.7.3
MarkupSafe==0.23
Pillow==2.6.1
PyChef==0.2.3
PyYAML==3.11
Pygments==2.0.1
Sphinx==1.2.3
argparse==1.2.1
astroid==1.2.1
awscli==1.6.1
bcdoc==0.12.2
binaryornot==0.3.0
boto==2.34.0
botocore==0.72.0
box.py==1.2.8
cffi==0.8.6
cliff==1.8.0
cmd2==0.6.7
colorama==0.3.2
cookiecutter==0.8.0
coverage==3.7.1
cryptography==0.6.1
decorator==3.4.0
docutils==0.12
dogapi==1.8.5
ecdsa==0.11
futures==2.2.0
httplib2==0.9
httpretty==0.8.3
iso8601==0.1.10
jmespath==0.5.0
jsonpatch==1.9
jsonpointer==1.5
jsonschema==2.4.0
kazoo==2.0
keyring==4.0
logilab-common==0.63.0
lxml==3.4.0
mock==1.0.1
mockito==0.5.2
netaddr==0.7.12
nose==1.3.4
numpy==1.9.1
oath==1.2
oslo.config==1.4.0
oslo.i18n==1.0.0
oslo.serialization==1.0.0
oslo.utils==1.0.0
paramiko==1.15.1
pbr==0.10.0
prettytable==0.7.2
pyOpenSSL==0.14
pyasn1==0.1.7
pycparser==2.10
pycrypto==2.6.1
pylint==1.3.1
pyparsing==2.0.3
python-ceilometerclient==1.0.12
python-cinderclient==1.1.1
python-dateutil==2.2
python-glanceclient==0.14.2
python-heatclient==0.2.12
python-keystoneclient==0.11.2
python-neutronclient==2.3.9
python-novaclient==2.20.0
python-openstackclient==0.4.1
python-swiftclient==2.3.1
python-troveclient==1.0.7
pytz==2014.9
qrcode==5.1
requests==2.4.3
rsa==3.1.4
scipy==0.14.0
simplejson==3.6.5
six==1.8.0
stevedore==1.1.0
urllib3==1.9.1
virtualenv==1.11.6
warlock==1.1.0
wsgiref==0.1.2

I'm on OS X 10.9.5

@mtougeron
Copy link
Author

Also, in case the awscli module version isn't sufficient.

$> aws --version
aws-cli/1.6.1 Python/2.7.6 Darwin/13.4.0

@stripecodahale
Copy link

Pretty sure this is due to it decoding the Base64-encoded keys and random data and trying to re-format the result as JSON or text.

@jamesls jamesls added confirmed bug This issue is a bug. labels Nov 18, 2014
@jamesls
Copy link
Member

jamesls commented Nov 19, 2014

Working on a fix now. Just to give a little more background on what's going on, this appears to be a regression from earlier versions from of the AWS CLI.

Typically, an AWS service response will return binary data base64 encoded. What the various AWS SDKs then do is decode the base64 data to then get the actual binary contents.

What the AWS CLI has historically done is take the base64 encoded response from the server and not decoded it. Instead we print the base64 contents directly to stdout, and the user can then base64 decode the contents when they need the actual binary contents.

However, in recent versions of the AWS CLI we are actually decoding the base64 contents which will then either

  1. cause errors when we try to decode as utf-8 so we can serialize it as json
  2. cause issues with --output text where we'd write raw bytes to the terminal

What I'd propose, and plan on implementing, is putting back the old behavior (and fixing the regression) such that binary content is always printed as base64 encoded.

@jamesls
Copy link
Member

jamesls commented Nov 19, 2014

Related:

#970
#1005

jamesls added a commit to jamesls/aws-cli that referenced this issue Nov 20, 2014
This fixes aws#1001.  This also fixes a regression we had
when displaying binary content.

As mentioned on the github issue, we were previously always
showing binary content as base64 encoded.  This PR adds back
this behavior.

In addition, this also gives us a hook into changing the
timestamp parsing so we can later add a config option to
unify the way timestamps are displayed.

Also fixes aws#970, and closes aws#1005.

I've also added an integration test for aws#1001 to ensure
we are in fact displaying a response that can be printed to
stdout.
@jimmynyny
Copy link

Looks like this one might be back?

mooney:~ jimmymooney$ aws --version
aws-cli/1.6.2 Python/2.7.10 Darwin/15.3.0
mooney:~ jimmymooney$ aws kms generate-random --number-of-bytes 128 
{
    "Plaintext": 
'utf8' codec can't decode byte 0xc3 in position 5: invalid continuation byte
mooney:~ jimmymooney$ aws kms encrypt --key-id alias/REDACTED --plaintext "bar"
{
    "KeyId": "REDACTED", 
    "CiphertextBlob": 
'utf8' codec can't decode byte 0xc6 in position 2: invalid continuation byte
mooney:~ jimmymooney$ 

thoward-godaddy pushed a commit to thoward-godaddy/aws-cli that referenced this issue Feb 12, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants