az cli 2.24.0 breaks iotedgedev "Deploy to IoT Edge devices" devops task

[jackt-moran]

• iotedgedev Version: 2.4.7
• Python Version: 3.6.10
• Pip Version: whatever is installed in iotedgedev container
• Development machine OS Version: Ubuntu 18.04
• IoT Edge device OS Version: N/A

Steps to Reproduce:

1. Try to use the "Deploy to IoT Edge devices" task in devops release pipeline.
2. Observe error (similar to below logs).

Possible short-term fix:

• You can see in the two log outputs below, the only difference is the az-cli version (2.23 yesterday, 2.24 today). Possible fix is to require az-cli version 2.23.* for the iotedgedev tool.

Here are sample logs from yesterday when this task worked:

2021-06-02T16:10:15.5088662Z ##[section]Starting: Azure IoT Edge - Deploy to IoT Edge QA devices
2021-06-02T16:10:15.5096275Z ==============================================================================
2021-06-02T16:10:15.5096643Z Task         : Azure IoT Edge
2021-06-02T16:10:15.5096932Z Description  : Build and deploy an Azure IoT Edge image
2021-06-02T16:10:15.5097231Z Version      : 2.4.7
2021-06-02T16:10:15.5097495Z Author       : Microsoft Corporation
2021-06-02T16:10:15.5097869Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2021-06-02T16:10:15.5098303Z ==============================================================================
2021-06-02T16:10:15.8261015Z Start deploying...
2021-06-02T16:10:15.8271870Z Deployment task is running in build pipeline? false
2021-06-02T16:10:15.8275751Z /home/vsts/work/r1/a/config/deployment.json
2021-06-02T16:10:15.8276626Z Checking if the following file is a valid json: /home/vsts/work/r1/a/config/deployment.json
2021-06-02T16:10:15.8279754Z Valid
2021-06-02T16:10:35.3228175Z Normalized deployment id is: moraniot-devops-deployment-qa
2021-06-02T16:10:39.9148735Z [command]/usr/bin/az --version
2021-06-02T16:10:39.9149303Z azure-cli                         2.23.0 *
2021-06-02T16:10:39.9149487Z 
2021-06-02T16:10:39.9149775Z core                              2.23.0 *
2021-06-02T16:10:39.9150461Z telemetry                          1.0.6
2021-06-02T16:10:39.9150785Z 
2021-06-02T16:10:39.9151009Z Extensions:
2021-06-02T16:10:39.9151408Z azure-devops                      0.18.0
2021-06-02T16:10:39.9151575Z 
2021-06-02T16:10:39.9151955Z Python location '/opt/az/bin/python3'
2021-06-02T16:10:39.9152416Z Extensions directory '/opt/az/azcliextensions'
2021-06-02T16:10:39.9152579Z 
2021-06-02T16:10:39.9152879Z Python (Linux) 3.6.10 (default, Apr 29 2021, 12:10:03) 
2021-06-02T16:10:39.9153229Z [GCC 7.5.0]
2021-06-02T16:10:39.9153333Z 
2021-06-02T16:10:39.9153609Z Legal docs and information: aka.ms/AzureCliLegal
2021-06-02T16:10:39.9153776Z 
2021-06-02T16:10:39.9153891Z 
2021-06-02T16:10:39.9154406Z WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'
2021-06-02T16:10:39.9154639Z 
2021-06-02T16:10:39.9154951Z Please let us know how we are doing: https://aka.ms/azureclihats
2021-06-02T16:10:39.9155597Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2021-06-02T16:10:39.9155855Z 
2021-06-02T16:10:54.8759319Z [command]/usr/bin/az iot edge deployment create --deployment-id qa --hub-name <omitted> --content /tmp/deployment_1622650235320.json --target-condition tags.environment='qa' --priority 0 --output none
2021-06-02T16:10:57.6374748Z Finished Deploying
2021-06-02T16:11:58.8853697Z ##[section]Finishing: Azure IoT Edge - Deploy to IoT Edge QA devices

And here are sample logs from a couple hours later yesterday when this task stopped working:

2021-06-02T18:42:48.2441632Z ##[section]Starting: Azure IoT Edge - Deploy to IoT Edge QA devices
2021-06-02T18:42:48.2449531Z ==============================================================================
2021-06-02T18:42:48.2449938Z Task         : Azure IoT Edge
2021-06-02T18:42:48.2450277Z Description  : Build and deploy an Azure IoT Edge image
2021-06-02T18:42:48.2450573Z Version      : 2.4.7
2021-06-02T18:42:48.2450862Z Author       : Microsoft Corporation
2021-06-02T18:42:48.2451270Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2021-06-02T18:42:48.2451753Z ==============================================================================
2021-06-02T18:42:48.5458570Z Start deploying...
2021-06-02T18:42:48.5468753Z Deployment task is running in build pipeline? false
2021-06-02T18:42:48.5472090Z /home/vsts/work/r1/a/config/deployment.json
2021-06-02T18:42:48.5472878Z Checking if the following file is a valid json: /home/vsts/work/r1/a/config/deployment.json
2021-06-02T18:42:48.5475706Z Valid
2021-06-02T18:43:02.1903889Z Normalized deployment id is: moraniot-devops-deployment-qa
2021-06-02T18:43:05.8974962Z [command]/usr/bin/az --version
2021-06-02T18:43:06.4154048Z azure-cli                         2.24.0 *
2021-06-02T18:43:06.4154864Z 
2021-06-02T18:43:06.4155455Z core                              2.24.0 *
2021-06-02T18:43:06.4156066Z telemetry                          1.0.6
2021-06-02T18:43:06.4156430Z 
2021-06-02T18:43:06.4156911Z Extensions:
2021-06-02T18:43:06.4157676Z azure-devops                      0.18.0
2021-06-02T18:43:06.4158064Z 
2021-06-02T18:43:06.4158770Z Python location '/opt/az/bin/python3'
2021-06-02T18:43:06.4159582Z Extensions directory '/opt/az/azcliextensions'
2021-06-02T18:43:06.4159990Z 
2021-06-02T18:43:06.4160558Z Python (Linux) 3.6.10 (default, May 21 2021, 07:22:10) 
2021-06-02T18:43:06.4161183Z [GCC 7.5.0]
2021-06-02T18:43:06.4161506Z 
2021-06-02T18:43:06.4162051Z Legal docs and information: aka.ms/AzureCliLegal
2021-06-02T18:43:06.4162439Z 
2021-06-02T18:43:06.4162749Z 
2021-06-02T18:43:06.4163701Z WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'
2021-06-02T18:43:06.4164189Z 
2021-06-02T18:43:06.4164777Z Please let us know how we are doing: https://aka.ms/azureclihats
2021-06-02T18:43:06.4165836Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2021-06-02T18:43:06.4166365Z 
2021-06-02T18:43:16.8499501Z [command]/usr/bin/az iot edge deployment create --deployment-id qa --hub-name <omitted> --content /tmp/deployment_1622659382188.json --target-condition tags.environment='qa' --priority 0 --output none
2021-06-02T18:43:17.6753176Z ERROR: The command failed with an unexpected error. Here is the traceback:
2021-06-02T18:43:17.6763566Z ERROR: 'IotHubResourceOperations' object has no attribute 'config'
2021-06-02T18:43:17.6764932Z Traceback (most recent call last):
2021-06-02T18:43:17.6766471Z   File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
2021-06-02T18:43:17.6767592Z     cmd_result = self.invocation.execute(args)
2021-06-02T18:43:17.6769194Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
2021-06-02T18:43:17.6770348Z     raise ex
2021-06-02T18:43:17.6771881Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
2021-06-02T18:43:17.6773141Z     results.append(self._run_job(expanded_arg, cmd_copy))
2021-06-02T18:43:17.6774753Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
2021-06-02T18:43:17.6776255Z     result = cmd_copy(params)
2021-06-02T18:43:17.6777830Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
2021-06-02T18:43:17.6779461Z     return self.handler(*args, **kwargs)
2021-06-02T18:43:17.6781075Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
2021-06-02T18:43:17.6782450Z     return op(**command_args)
2021-06-02T18:43:17.6784079Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/operations/hub.py", line 1015, in iot_edge_deployment_create
2021-06-02T18:43:17.6785197Z     config_type=config_type,
2021-06-02T18:43:17.6787577Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/operations/hub.py", line 1067, in _iot_hub_configuration_create
2021-06-02T18:43:17.6788731Z     hub_name=hub_name, resource_group_name=resource_group_name, login=login
2021-06-02T18:43:17.6789540Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 136, in get_target
2021-06-02T18:43:17.6790111Z     target_iothub = self.find_iothub(hub_name=hub_name, rg=rg)
2021-06-02T18:43:17.6790892Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 72, in find_iothub
2021-06-02T18:43:17.6791410Z     self._initialize_client()
2021-06-02T18:43:17.6792159Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 35, in _initialize_client
2021-06-02T18:43:17.6792728Z     self.sub_id = self.client.config.subscription_id
2021-06-02T18:43:17.6793428Z AttributeError: 'IotHubResourceOperations' object has no attribute 'config'
2021-06-02T18:43:17.6794095Z To open an issue, please run: 'az feedback'
2021-06-02T18:43:17.6835931Z ##[error]Error: Error: The process '/usr/bin/az' failed with exit code 1
2021-06-02T18:44:18.1308168Z ##[section]Finishing: Azure IoT Edge - Deploy to IoT Edge QA devices

I changed nothing at all on my end, so it is likely the result of the latest version of the iotedgedev container. Since I am just using the built-in devops task there is no way to upgrade az cli as the logs recommend.

Perhaps this issue will go away with a nightly build (for example) incorporating the az cli 2.24 patches, but I would argue there is a more general issue here which is the iotedgedev tool breaks periodically without any changes/updates on the user's end. Maybe a stricter versioning scheme would be useful? Luckily I am only stuck unable to deploy dev tests to our QA machine, but in the event of a hotfix it is possible this disruption would cause a major outage because of our inability to make deployments.

[jackt-moran]

FYI, I am using Microsoft hosted agents in DevOps.

[cindydeng1998]

Hi @jackt-moran, thank you for bringing this to our attention. We are aware of this issue and it is caused by an update to the az cli that happened last week. We have deployed a hotfix to the DevOps task that will be effective next week. Meanwhile, you can actually upgrade azure-iot extension with this hotfix here: microsoft/azure-pipelines-tasks#14929.

Let us know if you need any more help!

[jackt-moran]

@cindydeng1998 that linked workaround works, thank you!