Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Receiving 'NoneType' error when using meraki_management_interface on some MR devices #250

Open
sstone173 opened this issue Jul 15, 2021 · 20 comments · May be fixed by #278
Open

Receiving 'NoneType' error when using meraki_management_interface on some MR devices #250

sstone173 opened this issue Jul 15, 2021 · 20 comments · May be fixed by #278
Labels
bug Something isn't working ready to test Ready for testing waiting_for_response Waiting for response from someone

Comments

@sstone173
Copy link

While updating DNS settings on a large group of MR devices some fail with the error "AttributeError: 'NoneType' object has no attribute 'read'\n". Haven't been able to discover any correlation when updating like devices and configurations, some devices just error out while others succeed.

Play

- name: Meraki Playbook - Update AP DNS
  collections:
    - cisco.meraki
  connection: local
  hosts: localhost

  vars:
    auth_key: XXXXXXXXXXXXXXXXXXXXXXXXXXX
    org_id: "XXXXXX"
    net_name: "XXXXXXXXXXXX"

  tasks:

    - name: Query Network Devices
      meraki_device:
        auth_key: "{{ auth_key }}"
        org_id: "{{ org_id }}"
        net_name: "{{ net_name }}"
        state: query
      delegate_to: localhost
      register: response

    - name: Update AP Network Settings
      include_tasks: loop_meraki_update_mr_ip_settings.yaml
      loop: "{{ response.data }}"

Loop Tasks (loop_meraki_update_mr_ip_settings.yaml)

- name: Gather Existing AP Network Configuration
  meraki_management_interface:
    auth_key: "{{ auth_key }}"
    state: query
    org_id: "{{ org_id }}"
    net_id: "{{ item.network_id }}"
    serial: "{{ item.serial }}"
  register: output

- name: Update Network Configuration on AP
  meraki_management_interface:
    auth_key: "{{ auth_key }}"
    state: present
    org_id: "{{ org_id }}"
    net_id: "{{ item.network_id }}"
    serial: "{{ item.serial }}"
    wan1:
      wan_enabled: enabled
      using_static_ip: yes
      static_ip: "{{ output.data.wan1.static_ip }}"
      static_gateway_ip: "{{ output.data.wan1.static_gateway_ip }}"
      static_subnet_mask: "{{ output.data.wan1.static_subnet_mask }}"
      static_dns:
        - 10.1.2.3
        - 10.4.5.6
      vlan: "{{ output.data.wan1.vlan }}"

Output

TASK [Gather Existing AP Network Configuration] ***************************************************************************************************************************************************
task path: /mnt/c/Users/user1/Automation/projects/ansible/production/loop_meraki_update_mr_ip_settings.yaml:7
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: user1
<127.0.0.1> EXEC /bin/sh -c 'echo ~user1 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user1/.ansible/tmp `"&& mkdir "` echo /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398 `" && echo ansible-tmp-1626361069.196938-18378-119557732644398="` echo /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398 `" ) && sleep 0'
Using module file /home/user1/.ansible/collections/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py
<127.0.0.1> PUT /home/user1/.ansible/tmp/ansible-local-18161xfzhqhe0/tmp9xgz21nq TO /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/AnsiballZ_meraki_management_interface.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/ /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "data": {
        "wan1": {
            "static_dns": [
                "8.8.4.4",
                "8.8.8.8"
            ],
            "static_gateway_ip": "192.168.1.1",
            "static_ip": "192.138.1.24",
            "static_subnet_mask": "255.255.255.0",
            "using_static_ip": true,
            "vlan": 100
        }
    },
    "invocation": {
        "module_args": {
            "auth_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "follow_redirects": "all",
            "host": "api.meraki.com",
            "internal_error_retry_time": 60,
            "net_id": "N_XXXXXXXXXXXXX",
            "net_name": null,
            "org_id": "XXXXXX",
            "org_name": null,
            "output_format": "snakecase",
            "output_level": "normal",
            "protocol": "https",
            "rate_limit_retry_time": 165,
            "serial": "XXXX-XXXX-XXXX",
            "state": "query",
            "timeout": 30,
            "use_https": true,
            "use_proxy": false,
            "validate_certs": true,
            "wan1": null,
            "wan2": null
        }
    },
    "response": "OK (unknown bytes)",
    "status": 200
}

TASK [Update Network Configuration on AP] *********************************************************************************************************************************************************
task path: /mnt/c/Users/user1/Automation/projects/ansible/production/loop_meraki_update_mr_ip_settings.yaml:17
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: user1
<127.0.0.1> EXEC /bin/sh -c 'echo ~user1 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user1/.ansible/tmp `"&& mkdir "` echo /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844 `" && echo ansible-tmp-1626361070.1113186-18405-131799258753844="` echo /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844 `" ) && sleep 0'
Using module file /home/user1/.ansible/collections/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py
<127.0.0.1> PUT /home/user1/.ansible/tmp/ansible-local-18161xfzhqhe0/tmp8mqcq264 TO /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/ /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py", line 100, in <module>
    _ansiballz_main()
  File "/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py", line 92, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', init_globals=dict(_module_fqn='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', _modlib_path=modlib_path),
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py", line 384, in <module>
  File "/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py", line 347, in main
  File "/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/module_utils/network/meraki/meraki.py", line 477, in request
AttributeError: 'NoneType' object has no attribute 'read'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py\", line 100, in <module>\n    _ansiballz_main()\n  File \"/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py\", line 92, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', init_globals=dict(_module_fqn='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', _modlib_path=modlib_path),\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py\", line 384, in <module>\n  File \"/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py\", line 347, in main\n  File \"/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/module_utils/network/meraki/meraki.py\", line 477, in request\nAttributeError: 'NoneType' object has no attribute 'read'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}
@kbreit
Copy link
Collaborator

kbreit commented Jul 16, 2021

Thank you for the report. I'll try to take a look at it this weekend.

Do you have any custom settings that may cause this? Have you tried this same function using a Python script natively? I'm trying to determine if it's a bug with my code (probably) or if it's in the API (much less likely) that's triggering a bug.

@sstone173
Copy link
Author

I don't think I have anything custom that would cause this, my environment is pretty simple, but who knows. I haven't tried the same function using Python natively, I was happy to just get this working. The issue didn't occur very often, maybe only 5% of the time and sometimes repeating the play 3-4 times would finally get it to work, so it seems random to me.

@kbreit
Copy link
Collaborator

kbreit commented Jul 19, 2021

@sstone173 I tested this on my end and it seems to work fine. Admittedly, it's intermittent for you so I didn't expect it would reproduce easily here. A few thoughts:

  • If I send you an updated file for improved debugging, are you able to run it and reproduce the bug with that?
  • Do you find it's the same AP this happens on? Different APs? Same model? Anything like that we can use to hone in on the issue?
  • I can work with Meraki to see if they have an idea where this could be a problem.

Either way, I should be handling this exception. That fixes the symptom but not the bug though.

@sstone173
Copy link
Author

I can perform some additional testing if you send me the updated file, shouldn't be an issue. The issue didn't seem to be specific to model or any particular AP configuration, that's what made it a bit confusing to troubleshoot on my end as well.

@kbreit
Copy link
Collaborator

kbreit commented Jul 21, 2021

I'll need a few days to develop and test the code. My current thought is if it catches an exception in that area, it will output the payload from the request so we can see what is being submitted. I'll also have it output the HTTP code to see if we're getting back a bad code. It's probably not a bad thing to have the capability for anyways.

@kbreit
Copy link
Collaborator

kbreit commented Jul 21, 2021

I haven't tested this since I can't reproduce the bug. But please find the location where the Meraki collection is installed. Go into plugins/module_utils/network/meraki and you should see a meraki.py file. Replace that file with https://gist.github.com/kbreit/2f1d78e935f2dee7cac83b912d6a0275 . Reproduce the bug and it should give some more information if it fails with NoneType. If it doesn't, let me know and we can work on improvements.

Update: I had an error in the syntax and fixed it. Please redownload if you have it already.

@sstone173
Copy link
Author

Updated the meraki.py file, but haven't been able to reproduce the issue so far, so I'm thinking it may have just been a temporary issue on my end, possibly network related.

@kbreit
Copy link
Collaborator

kbreit commented Jul 29, 2021

You bring up a good point. Before the custom code, it would fail as an unhandled exception. Now, catching the exception so it should still continue to execute. I'll modify the code to catch the exception and fail with better output.

I'm on PTO the next couple of days so I should be able to make the changes by the weekend.

@kbreit
Copy link
Collaborator

kbreit commented Aug 3, 2021

@sstone173 Sorry about the delay. I updated the gist so it should fail if NoneType happens. Give it a few more tries to see if you can reproduce the issue.

@sstone173
Copy link
Author

@kbreit Updated the gist and ran the playbook against a bunch of our sites and was not able to replicate the issue. I'll be updating a large number of sites again in a month or so, so feel free to close this issue and I can reopen or create a new one if I run into it again during that project.

@kbreit
Copy link
Collaborator

kbreit commented Aug 23, 2021

Go ahead and keep this open for a while. No need to close it. I'll check back in later this year.

@kbreit kbreit assigned kbreit and unassigned kbreit Aug 24, 2021
@kbreit kbreit added the bug Something isn't working label Aug 24, 2021
@kbreit
Copy link
Collaborator

kbreit commented Aug 27, 2021

@sstone173 Funny story. I sit down tonight to start some development on a new module. I'm putting together the framework for the integration tests and run one. I'll give you one guess on what error I received. I ran it again, worked fine.

@sstone173
Copy link
Author

Yes, that does sound familiar :)

@mtb-xt
Copy link

mtb-xt commented Oct 19, 2021

still happens in other modules too, from time to time. usually, running again makes the error go away

 - An exception occurred during task execution. To see the full traceback, use -vvv. 
The error was: AttributeError: 'NoneType' object has no attribute 'read'

failed: [127.0.0.1] (item=MS350-1- port 5) => {"ansible_loop_var": "item", "changed": false, "item": [{"lat": 37.4180951010362, "lng": -122.098531723022, "model": "MS350-24", "name": "MS350-1-VA", "ports": [{"allowed_vlans": "all", "enabled": true, "isolation_enabled": false, "link_negotiation": "Auto negotiate", "poe_enabled": false, "port_id": "1", "rstp_enabled": true, 

.../ansible_meraki_switchport_payload_gqu5by09/ansible_meraki_switchport_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_switchport.py\", line 549, in main
File \"/tmp/ansible_meraki_switchport_payload_gqu5by09/ansible_meraki_switchport_payload.zip/ansible_collections/cisco/meraki/plugins/module_utils/network/meraki/meraki.py\", line 477, in request
AttributeError: 'NoneType' object has no attribute 'read'
", "module_stdout": "", "msg": "MODULE FAILURE
See stdout/stderr for the exact error", "rc": 1}

@kbreit
Copy link
Collaborator

kbreit commented Oct 20, 2021

That's what I've seen. My theory is it's a server side error but can't reproduce it enough to be helpful. If you'd like, I can see if I can get the Meraki team in contact with you to troubleshoot.

@mtb-xt
Copy link

mtb-xt commented Oct 26, 2021

That's what I've seen. My theory is it's a server side error but can't reproduce it enough to be helpful. If you'd like, I can see if I can get the Meraki team in contact with you to troubleshoot.

I'm communicating with them about some other issues, and this happens... sporadically.
But, would it be possible to not error out on this - the modules should probably catch that error and not crash, and possible allow retrying the request?

@kbreit
Copy link
Collaborator

kbreit commented Oct 27, 2021

Makes sense. I'm going to at a minimum catch the exception and handle it so it doesn't crash. If I can get it to redo the request, I'll add that in too.

@kbreit
Copy link
Collaborator

kbreit commented Oct 28, 2021

@mtb-xt I'm doing a little troubleshooting and added debugging code. I've had this issue three times tonight and got this output.

fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "body": null, "changed": false, "msg": "Failure occurred", "response": "Request failed: <urlopen error [Errno 99] Cannot assign requested address>", "status": -1}

A quick search with that error and urllib, which I think Ansible uses, returns https://stackoverflow.com/questions/11981933/python-urllib2-cannot-assign-requested-address. I'm looking into this further but made progress.

To your point, yes, this should be caught somehow. But if I can solve the problem, that's even better!

@kbreit
Copy link
Collaborator

kbreit commented Oct 29, 2021

@mtb-xt When you receive these errors, are they local on your system or a dedicated server? I ask because I had a lot of errors while addressing #257 but after quitting Slack and Discord, the errors seem to have largely disappeared.

Update: This may not be the problem. Having lots of problems still.

@kbreit kbreit linked a pull request Nov 12, 2021 that will close this issue
@kbreit
Copy link
Collaborator

kbreit commented Nov 12, 2021

@sstone173 I created a pull request to handle and retry the request 5 times before failing. I think this is due to something with Docker because as soon as I started running my tests locally and not in Docker, the errors disappeared. Because this problem is intermittent I can't test it. But please try this patch in a test environment and let me know if this fixes the issues. Thank you.

@kbreit kbreit added the ready to test Ready for testing label Nov 12, 2021
@kbreit kbreit added the waiting_for_response Waiting for response from someone label Jun 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working ready to test Ready for testing waiting_for_response Waiting for response from someone
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants