Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AzureIoTEdge Task fails with a cryptic error #14929

Closed
YarekTyshchenko opened this issue Jun 3, 2021 · 5 comments
Closed

AzureIoTEdge Task fails with a cryptic error #14929

YarekTyshchenko opened this issue Jun 3, 2021 · 5 comments
Assignees
@marianan
Labels
Area: Release bug stale

Comments

@YarekTyshchenko
Copy link

Type: Bug

Enter Task Name: AzureIoTEdge

Environment

  • Server - Azure Pipelines

    • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:

  • Agent - Hosted or Private: Hosted

    • If using Hosted agent, provide agent queue name:

Issue Description

Deploy command fails:

2021-06-03T09:35:50.4873350Z ERROR: The command failed with an unexpected error. Here is the traceback:
2021-06-03T09:35:50.4875763Z ERROR: 'IotHubResourceOperations' object has no attribute 'config'
  • Pipeline vmImage: ubuntu-latest
  • Agent: Hosted Agent
  • Azure IoT Edge Task version: 2.4.7

A bash task was ran before the AzureIoTEdge Task that installed iotedgedev, because of a previous issue #14167. The script is just pip install -U iotedgedev

Task logs

[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

2021-06-03T09:35:28.1113371Z ##[section]Starting: Azure IoT Edge - Deploy to HarkConnect Devices
2021-06-03T09:35:28.1123109Z ==============================================================================
2021-06-03T09:35:28.1123478Z Task         : Azure IoT Edge
2021-06-03T09:35:28.1123758Z Description  : Build and deploy an Azure IoT Edge image
2021-06-03T09:35:28.1124034Z Version      : 2.4.7
2021-06-03T09:35:28.1124409Z Author       : Microsoft Corporation
2021-06-03T09:35:28.1124803Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2021-06-03T09:35:28.1125241Z ==============================================================================
2021-06-03T09:35:28.4053904Z Start deploying...
2021-06-03T09:35:28.4063821Z Deployment task is running in build pipeline? true
2021-06-03T09:35:28.4068435Z /home/vsts/work/1/s/deployment_int.json
2021-06-03T09:35:28.4069144Z Checking if the following file is a valid json: /home/vsts/work/1/s/deployment_int.json
2021-06-03T09:35:28.4070880Z Valid
2021-06-03T09:35:30.7060694Z Normalized deployment id is: platform-devops-deployment
2021-06-03T09:35:34.4486684Z [command]/usr/bin/az --version
2021-06-03T09:35:35.0240960Z azure-cli                         2.24.0 *
2021-06-03T09:35:35.0242289Z 
2021-06-03T09:35:35.0242654Z core                              2.24.0 *
2021-06-03T09:35:35.0243038Z telemetry                          1.0.6
2021-06-03T09:35:35.0243238Z 
2021-06-03T09:35:35.0243495Z Extensions:
2021-06-03T09:35:35.0244233Z azure-devops                      0.18.0
2021-06-03T09:35:35.0244410Z 
2021-06-03T09:35:35.0244883Z Python location '/opt/az/bin/python3'
2021-06-03T09:35:35.0245616Z Extensions directory '/opt/az/azcliextensions'
2021-06-03T09:35:35.0245787Z 
2021-06-03T09:35:35.0246100Z Python (Linux) 3.6.10 (default, May 21 2021, 07:23:15) 
2021-06-03T09:35:35.0246414Z [GCC 9.3.0]
2021-06-03T09:35:35.0246534Z 
2021-06-03T09:35:35.0246804Z Legal docs and information: aka.ms/AzureCliLegal
2021-06-03T09:35:35.0246990Z 
2021-06-03T09:35:35.0247080Z 
2021-06-03T09:35:35.0247650Z WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'
2021-06-03T09:35:35.0247890Z 
2021-06-03T09:35:35.0248216Z Please let us know how we are doing: https://aka.ms/azureclihats
2021-06-03T09:35:35.0249032Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2021-06-03T09:35:35.0249458Z 
2021-06-03T09:35:49.5446075Z [command]/usr/bin/az iot edge deployment create --deployment-id platform-devops-deployment --hub-name meshnetint --content /tmp/deployment_1622712930704.json --target-condition deviceId='HarkConnectVM' --priority 0 --output none
2021-06-03T09:35:50.4873350Z ERROR: The command failed with an unexpected error. Here is the traceback:
2021-06-03T09:35:50.4875763Z ERROR: 'IotHubResourceOperations' object has no attribute 'config'
2021-06-03T09:35:50.4876475Z Traceback (most recent call last):
2021-06-03T09:35:50.4877380Z   File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
2021-06-03T09:35:50.4878199Z     cmd_result = self.invocation.execute(args)
2021-06-03T09:35:50.4879179Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
2021-06-03T09:35:50.4879898Z     raise ex
2021-06-03T09:35:50.4881092Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
2021-06-03T09:35:50.4882509Z     results.append(self._run_job(expanded_arg, cmd_copy))
2021-06-03T09:35:50.4883332Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
2021-06-03T09:35:50.4883819Z     result = cmd_copy(params)
2021-06-03T09:35:50.4884519Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
2021-06-03T09:35:50.4885232Z     return self.handler(*args, **kwargs)
2021-06-03T09:35:50.4886038Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
2021-06-03T09:35:50.4887183Z     return op(**command_args)
2021-06-03T09:35:50.4888113Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/operations/hub.py", line 1015, in iot_edge_deployment_create
2021-06-03T09:35:50.4888605Z     config_type=config_type,
2021-06-03T09:35:50.4890045Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/operations/hub.py", line 1067, in _iot_hub_configuration_create
2021-06-03T09:35:50.4890660Z     hub_name=hub_name, resource_group_name=resource_group_name, login=login
2021-06-03T09:35:50.4891429Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 136, in get_target
2021-06-03T09:35:50.4891960Z     target_iothub = self.find_iothub(hub_name=hub_name, rg=rg)
2021-06-03T09:35:50.4892716Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 72, in find_iothub
2021-06-03T09:35:50.4893190Z     self._initialize_client()
2021-06-03T09:35:50.4893889Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 35, in _initialize_client
2021-06-03T09:35:50.4894538Z     self.sub_id = self.client.config.subscription_id
2021-06-03T09:35:50.4895326Z AttributeError: 'IotHubResourceOperations' object has no attribute 'config'
2021-06-03T09:35:50.4895976Z To open an issue, please run: 'az feedback'
2021-06-03T09:35:50.4933689Z ##[error]Error: Error: The process '/usr/bin/az' failed with exit code 1
2021-06-03T09:36:50.8188509Z ##[section]Finishing: Azure IoT Edge - Deploy to HarkConnect Devices
@github-actions github-actions bot added Area: Release bug environment:need-to-triage Issues need to be triage by environment-deployment team labels Jun 3, 2021
@rsmit3
Copy link

rsmit3 commented Jun 3, 2021

I was getting the same error on Ubuntu-20.04 using [email protected]

I found this bulletpoint in the News section of the Readme for the azure-iot-cli-extension,

  • Azure CLI 2.24.0 requires an azure-iot extension update to 0.10.11 or later for IoT Hub commands to work properly. This can be done with az extension update --name azure-iot. A common error that arises when using an older azure-iot with Azure CLI 2.24.0 looks like AttributeError: 'IotHubResourceOperations' object has no attribute 'config'.

Adding the azure-iot extension update before using the AzureIoTEdge task appears to fix the problem for me:

- script: |
    az extension add --name azure-iot
  displayName: 'Update Azure IoT extension'

@cindydeng1998
Copy link
Contributor

Hello @YarekTyshchenko and @rsmit3, thank you for letting us know about this issue. We are aware of this error and have deployed a hotfix to the DevOps Task that will be effective next week. Meanwhile, the solution provided by @rsmit3 is a good work around, and you can reference this Github issue here to track our bugfix progress.

Thank you so much!

@cindydeng1998 cindydeng1998 mentioned this issue Jun 3, 2021
Closed
@AmrutaKawade AmrutaKawade removed the environment:need-to-triage Issues need to be triage by environment-deployment team label Jun 4, 2021
@cindydeng1998
Copy link
Contributor

@YarekTyshchenko @rsmit3 , the hotfix is now live. Please let us know if you are still running into this issue. If not, please feel free to close this Github issue. Let us know if you need anything else, thank you!

@toolboc
Copy link
Contributor

toolboc commented Jun 23, 2021

@cindydeng1998, you mentioned that a hotfix is now live, but it does not seem that all Azure DevOps instances are receiving the hotfix.

The latest version of the AzureIoTEdgeV2 task at this time is version 2.4.8. When, I attempt to run a"Deploy to IoT Edge devices" task against this version, everything works fine as expected.

2021-06-22T22:16:35.4445588Z ##[section]Starting: Azure IoT Edge - Deploy to IoT Edge devices
2021-06-22T22:16:35.4455340Z ==============================================================================
2021-06-22T22:16:35.4455708Z Task         : Azure IoT Edge
2021-06-22T22:16:35.4456026Z Description  : Build and deploy an Azure IoT Edge image
2021-06-22T22:16:35.4456309Z Version      : 2.4.8
2021-06-22T22:16:35.4456577Z Author       : Microsoft Corporation
2021-06-22T22:16:35.4456955Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2021-06-22T22:16:35.4457377Z ==============================================================================
2021-06-22T22:16:35.7288297Z Start deploying...
2021-06-22T22:16:35.7297942Z Deployment task is running in build pipeline? false
2021-06-22T22:16:35.7528947Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/EdgeSolution/config/deployment.json
2021-06-22T22:16:35.7529775Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/EdgeSolution/deployment.template.json
2021-06-22T22:16:35.7530781Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/EdgeSolution/modules/SampleModule/module.json
2021-06-22T22:16:35.7531538Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/EdgeSolution/undefined/deployment.json
2021-06-22T22:16:35.7532234Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/azuredeploy.json
2021-06-22T22:16:35.7532893Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/deployment.json
2021-06-22T22:16:35.7533541Z /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/release-pipeline.json
2021-06-22T22:16:35.7534402Z Checking if the following file is a valid json: /home/vsts/work/r1/a/_IoTEdge-DevOps-Testing/drop/EdgeSolution/config/deployment.json
2021-06-22T22:16:35.7534898Z Valid
2021-06-22T22:16:42.5320398Z Normalized deployment id is: azuredevops-deployment-qa
2021-06-22T22:17:06.1431550Z [command]/usr/bin/az --version
2021-06-22T22:17:06.6973438Z azure-cli                         2.24.2 *
2021-06-22T22:17:06.6974352Z 
2021-06-22T22:17:06.6975105Z core                              2.24.2 *
2021-06-22T22:17:06.6975963Z telemetry                          1.0.6
2021-06-22T22:17:06.6976434Z 
2021-06-22T22:17:06.6977056Z Extensions:
2021-06-22T22:17:06.6978151Z azure-devops                      0.18.0
2021-06-22T22:17:06.6978670Z 
2021-06-22T22:17:06.6979629Z Python location '/opt/az/bin/python3'
2021-06-22T22:17:06.6980784Z Extensions directory '/opt/az/azcliextensions'
2021-06-22T22:17:06.6981097Z 
2021-06-22T22:17:06.6981572Z Python (Linux) 3.6.10 (default, Jun  2 2021, 06:55:52) 
2021-06-22T22:17:06.6982060Z [GCC 7.5.0]
2021-06-22T22:17:06.6982289Z 
2021-06-22T22:17:06.6982726Z Legal docs and information: aka.ms/AzureCliLegal
2021-06-22T22:17:06.6983030Z 
2021-06-22T22:17:06.6983249Z 
2021-06-22T22:17:06.6984084Z WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'
2021-06-22T22:17:06.6984478Z 
2021-06-22T22:17:06.6984965Z Please let us know how we are doing: https://aka.ms/azureclihats
2021-06-22T22:17:06.6985882Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2021-06-22T22:17:06.6986315Z 
2021-06-22T22:17:20.2598956Z [command]/usr/bin/az iot edge deployment create --deployment-id azuredevops-deployment-qa --hub-name IoTEdge-DevOps-iot-hub --content /tmp/deployment_1624400202530.json --target-condition tags.environment='qa' --priority 0 --output none
2021-06-22T22:17:22.4414246Z Finished Deploying
2021-06-22T22:18:23.1807550Z ##[section]Finishing: Azure IoT Edge - Deploy to IoT Edge devices

However, when using a completely separate instance of Azure DevOps with the same exact codebase and deployment template, I noticed that this instance does not pull version 2.4.8, it instead pulls in version 2.4.7 and results in the exact same error mentioned in the original issue:

2021-06-22T13:23:49.8177967Z ##[section]Starting: Azure IoT Edge - Deploy to IoT Edge devices
2021-06-22T13:23:49.8187066Z ==============================================================================
2021-06-22T13:23:49.8187420Z Task         : Azure IoT Edge
2021-06-22T13:23:49.8187673Z Description  : Build and deploy an Azure IoT Edge image
2021-06-22T13:23:49.8187984Z Version      : 2.4.7
2021-06-22T13:23:49.8188197Z Author       : Microsoft Corporation
2021-06-22T13:23:49.8188565Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2021-06-22T13:23:49.8188921Z ==============================================================================
2021-06-22T13:23:50.4413363Z Start deploying...
2021-06-22T13:23:50.4414601Z Deployment task is running in build pipeline? false
2021-06-22T13:23:50.4481267Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/EdgeSolution/config/deployment.json
2021-06-22T13:23:50.4482032Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/EdgeSolution/deployment.template.json
2021-06-22T13:23:50.4482897Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/EdgeSolution/modules/SampleModule/module.json
2021-06-22T13:23:50.4483976Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/EdgeSolution/undefined/deployment.json
2021-06-22T13:23:50.4484633Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/azuredeploy.json
2021-06-22T13:23:50.4485262Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/deployment.json
2021-06-22T13:23:50.4485879Z /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/release-pipeline.json
2021-06-22T13:23:50.4487039Z Checking if the following file is a valid json: /home/vsts/work/r1/a/_IoTEdge-DevOps/drop/EdgeSolution/config/deployment.json
2021-06-22T13:23:50.4488670Z Valid
2021-06-22T13:23:54.1381404Z Normalized deployment id is: azuredevops-deployment-qa
2021-06-22T13:24:02.4409982Z [command]/usr/bin/az --version
2021-06-22T13:24:02.8756200Z azure-cli                         2.24.2 *
2021-06-22T13:24:02.8756842Z 
2021-06-22T13:24:02.8757371Z core                              2.24.2 *
2021-06-22T13:24:02.8757861Z telemetry                          1.0.6
2021-06-22T13:24:02.8758197Z 
2021-06-22T13:24:02.8758572Z Extensions:
2021-06-22T13:24:02.8759244Z azure-devops                      0.18.0
2021-06-22T13:24:02.8759579Z 
2021-06-22T13:24:02.8760168Z Python location '/opt/az/bin/python3'
2021-06-22T13:24:02.8760843Z Extensions directory '/opt/az/azcliextensions'
2021-06-22T13:24:02.8761169Z 
2021-06-22T13:24:02.8761630Z Python (Linux) 3.6.10 (default, Jun  2 2021, 06:55:52) 
2021-06-22T13:24:02.8762544Z [GCC 7.5.0]
2021-06-22T13:24:02.8764463Z 
2021-06-22T13:24:02.8764816Z Legal docs and information: aka.ms/AzureCliLegal
2021-06-22T13:24:02.8765009Z 
2021-06-22T13:24:02.8765109Z 
2021-06-22T13:24:02.8765781Z WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'
2021-06-22T13:24:02.8766052Z 
2021-06-22T13:24:02.8766408Z Please let us know how we are doing: https://aka.ms/azureclihats
2021-06-22T13:24:02.8767165Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2021-06-22T13:24:02.8767484Z 
2021-06-22T13:24:13.3268018Z [command]/usr/bin/az iot edge deployment create --deployment-id azuredevops-deployment-qa --hub-name IoTEdgeDevOps-iothub-IoTEdgeDevOps --content /tmp/deployment_1624368234137.json --target-condition tags.environment='qa' --priority 0 --output none
2021-06-22T13:24:13.7437998Z ERROR: The command failed with an unexpected error. Here is the traceback:
2021-06-22T13:24:13.7439536Z ERROR: 'IotHubResourceOperations' object has no attribute 'config'
2021-06-22T13:24:13.7440056Z Traceback (most recent call last):
2021-06-22T13:24:13.7440743Z   File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
2021-06-22T13:24:13.7441236Z     cmd_result = self.invocation.execute(args)
2021-06-22T13:24:13.7441944Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
2021-06-22T13:24:13.7442436Z     raise ex
2021-06-22T13:24:13.7443125Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
2021-06-22T13:24:13.7444060Z     results.append(self._run_job(expanded_arg, cmd_copy))
2021-06-22T13:24:13.7444814Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
2021-06-22T13:24:13.7445318Z     result = cmd_copy(params)
2021-06-22T13:24:13.7446003Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
2021-06-22T13:24:13.7446509Z     return self.handler(*args, **kwargs)
2021-06-22T13:24:13.7447212Z   File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
2021-06-22T13:24:13.7447723Z     return op(**command_args)
2021-06-22T13:24:13.7448436Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/operations/hub.py", line 1015, in iot_edge_deployment_create
2021-06-22T13:24:13.7448943Z     config_type=config_type,
2021-06-22T13:24:13.7449636Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/operations/hub.py", line 1067, in _iot_hub_configuration_create
2021-06-22T13:24:13.7450412Z     hub_name=hub_name, resource_group_name=resource_group_name, login=login
2021-06-22T13:24:13.7451164Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 136, in get_target
2021-06-22T13:24:13.7451715Z     target_iothub = self.find_iothub(hub_name=hub_name, rg=rg)
2021-06-22T13:24:13.7452455Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 72, in find_iothub
2021-06-22T13:24:13.7452954Z     self._initialize_client()
2021-06-22T13:24:13.7453695Z   File "/opt/az/azcliextensions/azure-iot/azext_iot/iothub/providers/discovery.py", line 35, in _initialize_client
2021-06-22T13:24:13.7454249Z     self.sub_id = self.client.config.subscription_id
2021-06-22T13:24:13.7454903Z AttributeError: 'IotHubResourceOperations' object has no attribute 'config'
2021-06-22T13:24:13.7702725Z To open an issue, please run: 'az feedback'
2021-06-22T13:24:14.0916146Z ##[error]Error: Error: The process '/usr/bin/az' failed with exit code 1
2021-06-22T13:25:14.8754598Z ##[section]Finishing: Azure IoT Edge - Deploy to IoT Edge devices

As such, it seems that this issue is very much related to a discrepancy in the 2.4.7 and 2.4.8 versions of the AzureIoTEdgeV2 task. When I attempt to configure the task version to use the latest release, I am only given the option to select "2.*". I have even tried removing and recreating steps but still end up getting the broken 2.4.7 version upon successive runs.

image

How exactly can I ensure that my Azure DevOps instance is receiving the 2.4.8 hotfix?

@github-actions
Copy link

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

@github-actions github-actions bot added the stale label Dec 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marianan marianan

Labels
Area: Release bug stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@YarekTyshchenko @toolboc @marianan @cindydeng1998 @AmrutaKawade @nadesu @rsmit3