Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed .loads error for non decoded json in Python 3 #32065

Merged
merged 3 commits into from
Nov 17, 2017
Merged

fixed .loads error for non decoded json in Python 3 #32065

merged 3 commits into from
Nov 17, 2017

Conversation

andy-pi
Copy link
Contributor

@andy-pi andy-pi commented Oct 24, 2017

SUMMARY

Using Python 3 this module throws an error when parsing the JSON repsonse, as it has not been decoded from its native utf-8.

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

Cloudflare DNS

ANSIBLE VERSION
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible']
  ansible python module location = /home/ubuntu/workspace/ansible-roles/env/lib/python3.5/site-packages/ansible
  executable location = /home/ubuntu/workspace/ansible-roles/env/bin/ansible
  python version = 3.5.1 (default, Dec 18 2015, 00:00:00) [GCC 4.8.4]
ADDITIONAL INFORMATION

Ansible command used:

 name: Set A record to point to this server in cloudflare
 cloudflare_dns:
     zone: "{{ domain }}"
     record: "@"
     type: A
     value: "{{ ansible_default_ipv4.address }}"
     account_email: "{{ cloudflare_email }}"
     account_api_token: "{{ cloudflare_api_key }}"

Before:

TASK [config-vm : Set A record to point to this server in cloudflare] 
****************************************************
--TRUNCATED--
line 372, in _cf_simple_api_call\r\n    
result = json.loads(content)\r\n  File \"/usr/lib/python3.5/json/__init__.py\", 
line 312, in loads\r\n    s.__class__.__name__))\r\n
TypeError: the JSON object must be str, not 'bytes'\r\n", "msg": "MODULE FAILURE", "rc": 0}

After:

TASK [config-vm : Set A record to point to this server in cloudflare] ****************************************************
changed:

@ansibot
Copy link
Contributor

ansibot commented Oct 24, 2017

@ansibot ansibot added affects_2.5 This issue/PR affects Ansible v2.5 bugfix_pull_request community_review In order to be merged, this PR must follow the community review workflow. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. net_tools Net-tools category new_contributor This PR is the first contribution by a new community member. python3 support:community This issue/PR relates to code supported by the Ansible community. labels Oct 24, 2017
@jborean93 jborean93 removed the needs_triage Needs a first human triage before being processed. label Oct 25, 2017
@ansibot
Copy link
Contributor

ansibot commented Oct 27, 2017

The test ansible-test compile --python 2.6 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:6718: SyntaxError: RETURN = '''

The test ansible-test compile --python 2.7 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:6719: SyntaxError: RETURN = '''

The test ansible-test compile --python 3.5 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:6719: SyntaxError: RETURN = '''

The test ansible-test compile --python 3.6 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:6719: SyntaxError: RETURN = '''

The test ansible-test sanity --test ansible-doc --python 2.6 [?] failed with the following error:

Command "ansible-doc cloudflare_dns fly" returned exit status 1.
>>> Standard Error
ERROR! module fly missing documentation (or could not parse documentation): Parsing produced an empty object.

The test ansible-test sanity --test ansible-doc --python 2.7 [?] failed with the following error:

Command "ansible-doc cloudflare_dns fly" returned exit status 1.
>>> Standard Error
ERROR! module fly missing documentation (or could not parse documentation): Parsing produced an empty object.

The test ansible-test sanity --test ansible-doc --python 3.5 [?] failed with the following error:

Command "ansible-doc cloudflare_dns fly" returned exit status 1.
>>> Standard Error
ERROR! module fly missing documentation (or could not parse documentation): Parsing produced an empty object.

The test ansible-test sanity --test ansible-doc --python 3.6 [?] failed with the following error:

Command "ansible-doc cloudflare_dns fly" returned exit status 1.
>>> Standard Error
ERROR! module fly missing documentation (or could not parse documentation): Parsing produced an empty object.

The test ansible-test sanity --test boilerplate [?] failed with the following error:

Command "test/sanity/code-smell/boilerplate.sh" returned exit status 2.
>>> Standard Output
== Missing __metaclass__ = type ==
./lib/ansible/modules/net_tools/fly.py

== Missing from __future__ import (absolute_import, division, print_function) ==
./lib/ansible/modules/net_tools/fly.py

The test ansible-test sanity --test import --python 2.6 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:233:6719: SyntaxError: EOF while scanning triple-quoted string literal

The test ansible-test sanity --test import --python 2.7 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:233:6719: SyntaxError: EOF while scanning triple-quoted string literal

The test ansible-test sanity --test import --python 3.5 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:6719: SyntaxError: EOF while scanning triple-quoted string literal

The test ansible-test sanity --test import --python 3.6 [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:6719: SyntaxError: EOF while scanning triple-quoted string literal

The test ansible-test sanity --test no-wildcard-import [?] failed with the following error:

Command "test/sanity/code-smell/no-wildcard-import.sh" returned exit status 3.
>>> Standard Output
== Wildcard imports detected ==
./lib/ansible/modules/net_tools/fly.py:from ansible.module_utils.basic import *
./lib/ansible/modules/net_tools/fly.py:from ansible.module_utils.urls import *

The test ansible-test sanity --test pep8 [?] failed with the following errors:

lib/ansible/modules/net_tools/fly.py:17:1: W293 blank line contains whitespace
lib/ansible/modules/net_tools/fly.py:51:24: W291 trailing whitespace
lib/ansible/modules/net_tools/fly.py:73:10: E901 TokenError: EOF in multi-line string
lib/ansible/modules/net_tools/fly.py:232:11: W292 no newline at end of file

The test ansible-test sanity --test pylint [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:232:0: syntax-error EOF while scanning triple-quoted string literal (<string>, line 232)

The test ansible-test sanity --test validate-modules [?] failed with the following error:

lib/ansible/modules/net_tools/fly.py:0:0: E401 Python SyntaxError while parsing module

click here for bot help

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. new_module This PR includes a new module. and removed community_review In order to be merged, this PR must follow the community review workflow. new_module This PR includes a new module. labels Oct 27, 2017
@andy-pi
Copy link
Contributor Author

andy-pi commented Oct 27, 2017

Tried to submit a new module, but rolled back to previous commit. Will wait until this pull request accepted before adding another.

@ansibot ansibot added community_review In order to be merged, this PR must follow the community review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Oct 27, 2017
@ansibot ansibot added stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. and removed new_contributor This PR is the first contribution by a new community member. labels Nov 4, 2017
@samdoran samdoran self-assigned this Nov 13, 2017
@andy-pi
Copy link
Contributor Author

andy-pi commented Nov 14, 2017

bot_status

@andy-pi
Copy link
Contributor Author

andy-pi commented Nov 14, 2017

@samdoran thanks for taking on the review for this. It's my first open source contribution, so please let me know if there are any mistakes - hoping to submit a new module after this.
This fix is important for python 3.5 (json loads cannot accept bytes as input) but is not required for 3.6+ see https://docs.python.org/3.6/library/json.html#json.loads

@ansibot
Copy link
Contributor

ansibot commented Nov 14, 2017

waiting_on: ansible
module: cloudflare_dns
supported_by: community
maintainers: mgruener
changes_requested_by: null
needs_info: False
needs_revision: False
needs_rebase: False
merge_commits: []
mergeable_state: clean
shippable_status: success
maintainer_shipits (module maintainers): 0
community_shipits (namespace maintainers): 0
ansible_shipits (core team members): 0
shipit_actors (maintainer or core team member): []
shipit_actors_other: []

click here for bot help

@@ -369,7 +369,8 @@ def _cf_simple_api_call(self,api_call,method='GET',payload=None):

if content:
try:
result = json.loads(content)
resp_json = content.decode('utf-8')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than using separate variables, you can accomplish this in one line:

result = json.loads(content.decode('utf-8'))

@samdoran
Copy link
Contributor

Thanks for the contribution! You did everything correctly as far as contributions go. Please see my comment on your code.

@ansibot ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Nov 16, 2017
@andy-pi
Copy link
Contributor Author

andy-pi commented Nov 16, 2017

@samdoran thankyou for your review - code refactored now as per your suggestion!

@@ -369,7 +369,7 @@ def _cf_simple_api_call(self,api_call,method='GET',payload=None):

if content:
try:
result = json.loads(content)
result = json.loads(content.decode('utf-8'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use to_text here instead of .decode. to_text is resistant to unicode errors in a way that .decode is not.

from ansible.module_utils._text import to_native, to_text
[...]
try:
    result = json.loads(to_text(content, errors='surrogate_then_strict'))
except (json.JSONDecodeError, UnicodeError) as e:
    error_msg += "; Failed to parse API response with error {0}: {1}".format(to_native(e), content)

@samdoran
Copy link
Contributor

@andy-pi Sorry to give you bad advice. I asked for a wiser opinion and got it. Toshio's recommendation is the most resilient way to fix this issue.

@andy-pi
Copy link
Contributor Author

andy-pi commented Nov 17, 2017

@samdolan no probs @abadger thanks for the advice. I don't understand the detail of how to_text works exactly but I do prefer giving a user friendly error - which I did not get when fixing this bug.
Anyway I will update again with your suggestion shortly.
(EDITED - 1st draft not relevant)

@samdoran samdoran merged commit 67d5e1d into ansible:devel Nov 17, 2017
@samdoran
Copy link
Contributor

@andy-pi to_text and to_native encode/decode text and byte strings properly depending on the Python version. That's the high level overview.

samdoran pushed a commit that referenced this pull request Nov 17, 2017
* fixed .loads error for non decoded json in Python 3

* fixed .loads error Python 3.5 - refactor code to one line

* fixed .loads error python 3.5 - mod to use to_text instead of .decode as per reviewer comment

(cherry picked from commit 67d5e1d)
@samdoran
Copy link
Contributor

Cherry-picked to stable-2.4 for inclusion in Ansible 2.4.2

@andy-pi
Copy link
Contributor Author

andy-pi commented Nov 18, 2017

@samdoran thanks!!!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.5 This issue/PR affects Ansible v2.5 bug This issue/PR relates to a bug. community_review In order to be merged, this PR must follow the community review workflow. module This issue/PR relates to a module. net_tools Net-tools category python3 support:community This issue/PR relates to code supported by the Ansible community.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants