azure_rm inventory batch_fetch not working in v1.19.0

[chipselden]

SUMMARY

After upgrading my azure collection from 1.18.1 to 1.19.0, a dynamic inventory I've been using stopped working. The error seems to have to do with the batch_fetch functionality when running against an Azure subscription with many VMs. I'm seeing this behavior with a subscription that has 71 VMs, but not seeing it with one that has 8 VMs. The issue doesn't happen with v1.18.1 of the azure collection.

ISSUE TYPE

• Bug Report

COMPONENT NAME

azure.azcollection.azure_rm inventory – Azure Resource Manager inventory plugin

ANSIBLE VERSION

ansible [core 2.15.6]
  config file = /Users/Chip.Selden/Documents/_GIT/ansible-azure-test/ansible.cfg
  configured module search path = ['/Users/Chip.Selden/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/Chip.Selden/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/bin/ansible
  python version = 3.9.17 (main, Jun 20 2023, 17:20:08) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/lib/python3.9/site-packages/ansible_collections
Collection         Version
------------------ -------
azure.azcollection 1.19.0 

CONFIGURATION

CONFIG_FILE() = /Users/Chip.Selden/Documents/_GIT/ansible-azure-test/ansible.cfg
INVENTORY_ANY_UNPARSED_IS_FAILED(/Users/Chip.Selden/Documents/_GIT/ansible-azure-test/ansible.cfg) = True
INVENTORY_ENABLED(/Users/Chip.Selden/Documents/_GIT/ansible-azure-test/ansible.cfg) = ['azure_rm']

OS / ENVIRONMENT

The machine I'm running ansible from is macOS Ventura. I have also seen this issue when running through AWX in a Centos Execution Environment.

STEPS TO REPRODUCE

1. Create and activate a python virtual environment with:
   • python 3.9+
   • ansible 8.4.0+
   • the packages from requirements-azure.txt
2. Create an empty playbook directory and move to it
3. Create the ansible.cfg file below
4. Create the azure_rm.yml inventory file below
   • This uses batch_fetch: true which is the default
5. Log in to azure using az login
   • The azure account I'm experiencing this with has 71 VMs
   • This issue does not occur with a different account which only has 8 VMs
6. Run the generate inventory command below --> FAILS
7. Change batch_fetch to false in azure_rm.yml
8. Run the generate inventory command below --> SUCCEEDS

# ansible.cfg 
[inventory]
any_unparsed_is_failed = true
enable_plugins = azure_rm

# azure_rm.yml
---
plugin: azure.azcollection.azure_rm
subscription_id: "some-subscription-id"
batch_fetch: true

# generate inventory
ansible-inventory -i azure_rm.yml --list -vvvv --output inventory.log

EXPECTED RESULTS

The inventory file to parse without error.

ACTUAL RESULTS

The inventory file fails to parse due to a json decoding issue during the second execution of the _send_batch method from _process_queue_batch. The response body from the second call is an empty byte string.

ansible-inventory [core 2.15.6]
  config file = /Users/Chip.Selden/Documents/_GIT/ansible-azure-test/ansible.cfg
  configured module search path = ['/Users/Chip.Selden/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/Chip.Selden/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/bin/ansible-inventory
  python version = 3.9.17 (main, Jun 20 2023, 17:20:08) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/bin/python)
  jinja version = 3.1.2
  libyaml = True
Using /Users/Chip.Selden/Documents/_GIT/ansible-azure-test/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from 
redirecting (type: inventory) ansible.builtin.azure_rm to azure.azcollection.azure_rm
Loading collection azure.azcollection from /Users/Chip.Selden/.ansible/collections/ansible_collections/azure/azcollection
[WARNING]:  * Failed to parse /Users/Chip.Selden/Documents/_GIT/ansible-azure-test/azure_rm.yml with
ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin: Expecting value: line 1 column 1 (char 0)
  File "/Users/Chip.Selden/.local/share/virtualenvs/ansible-azure-test-g8mmdU37/lib/python3.9/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/Users/Chip.Selden/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 220, in parse
    self._get_hosts()
  File "/Users/Chip.Selden/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 283, in _get_hosts
    self._process_queue_batch()
  File "/Users/Chip.Selden/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 414, in _process_queue_batch
    batch_resp = self._send_batch(batch_requests)
  File "/Users/Chip.Selden/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 445, in _send_batch
    return json.loads(response.body())
  File "/usr/local/Cellar/[email protected]/3.9.17_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/[email protected]/3.9.17_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/[email protected]/3.9.17_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
ERROR! Completely failed to parse inventory source /Users/Chip.Selden/Documents/_GIT/ansible-azure-test/azure_rm.yml

[Fred-sun]

@chipselden I have repeatedly tested on my environment and have not encountered the problem you submitted? No matter whether batch_fetch is true or false, it can be returned normally. Could you please help confirm the situation and problems on your side? Thank you!

[chipselden]

@Fred-sun - Thanks for looking into this. I think the error also depends on the azure subscription having enough VMs in it so that the resulting calls are large enough for pagination to be needed.

The account where I've observed the bug has 70+ VMs. In a smaller account I have access to (8 VMs), the inventory plugin works fine no matter if batch_fetch is enabled or disabled.

[Fred-sun]

@chipselden Thank you for your feedback! Will I look into the matter again?

[rick-ha]

Having the same issue here as well. We're querying ~12 VMs and using a MI with pip3 Az CLI.

Setting batch_fetch to false allowed dynamic inventory to return the proper VMs.

When batch_fetch is true:

When batch_fetch is false:

[mystery-rabbit]

Hi; I am also observing this issue presented by @chipselden - exactly the same details.

I have my original venv with azure.azcollection 1.16.0 working fine, but a second venv with azure.azcollection 1.19.0 which fails against the same subscription with 11 (or more) VMs in the resource group (batch_fetch is not explicitly defined, and is presumably defaulting to true).

Doing either of the following results in the 1.19.0 code working: neither are long term solutions.

• reducing the number of VMs in the RG to 10
• setting batch_fetch: false

@Fred-sun - in answer to your question - yes please; Add more VMs to trigger the behaviour.

[Fred-sun]

@rick-ha @mystery-rabbit Thanks for your feedback! I will retry it!

[vijayreddiar]

I am also facing the same issue.

We have 500+ VMs and as of now, the workaround is found to be setting batch_fetch: false

[Fred-sun]

@chipselden @vijayreddiar I've found out why, and I'm trying to fix it. Thank you!

[hkrutzer]

I also got this error. I added

        print(response)
        print(response.body())

to _send_batch, which printed:

<RequestsTransportResponse: 202 Accepted>
b''

So it seems in some cases the API doesn't return data which leads to a JSON decode error. In my case it seems to be related to having different subscriptions for different VMs.

[mengelld]

I've also got this issue from 22 out of 55 azure tenant imports.

Setting the batch_fetch: false as a source variable (in AWX) has allowed the imports to work.

[Fred-sun]

Fixes by #1344

[harCamConsulting]

Can confirm that I was having the same issue, and manually applying the code in the PR fixes it for me (even with batch_mode: true).

[Fred-sun]

@harCamConsulting Thanks for your feedback! I will push for merge as soon as possible!

[dawid-gupta]

@Fred-sun @xuzhang3 reported issue was in version 1.19 while the fix was merged to version 2.1.0. There are a lot of breaking changes between 1.x and 2.x version. Before people migrate to 2.x, is it possible to have that fix merged to version 1.19.0 (or like 1.19.1)?