Unable to locate executable file: 'iotedgedev'

[oliverchristen]

Type: Bug

Enter Task Name: AzureIoTEdgeV2

Environment

• Server - Azure Pipelines
• Agent - Hosted
  vmImage ubuntu 20.04.1
  agent version 2.179.0
  AzureIoTEdge task version 2.4.5

Issue Description

##[error]Error: Unable to locate executable file: 'iotedgedev'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

This happened from time to time and solves itself after some hours. But it happens again every few weeks.

All the required informations are found here:
https://developercommunity.visualstudio.com/content/problem/1282804/errorerror-unable-to-locate-executable-file-iotedg-1.html

If full log is required, I can provide it privately.

[Haeavar]

We have the same issue. It appears since yesterday without any changes.

AzureIoTEdge task version 2.4.5
Pool: Azure Pipelines
Image: ubuntu-18.04
Agent: Azure Pipelines 4

[oliverchristen]

It will disappear after some hours magically probably.
What you could try is to install iotedgedev manually

      - task: Bash@3
        inputs:
          targetType: 'inline'
          script: 'pip install -U iotedgedev'

[quality-leftovers]

+1. Not liking the idea of manually installing dependencies for devops tasks.
Pipeline is dead for at least 6 hours now.

Calling pip manually feels wrong. If we end up doing that we might also consider removing the devops task and using custom scripts instead.

[oliverchristen]

Totally agree. The suggestion is meant as a workaround.

But for deploying deploment manifests to IoT hub, we switched to custom azure CLI script. This is weirdly much faster than the IoT Edge pipeline task.

[holmeren]

I just stumbled upon this issue aswell. Did you just wait a few hours, and ran the pipeline again?

[quality-leftovers]

Our pipeline still failed today in the moring. Therefore we replaced the devops task with a custom script.

[holmeren]

Did the script include the 'pip install -U iotedgedev' mentioned above?

I'm not sure how to install iotedgedev

[oliverchristen]

@holmeren This happened to me about three times. In my case it disappeared every time after some hours, when I retried to run the pipeline.
The command pip install -U iotedgedev installs iotedgedev.

[oliverchristen]

@randomstring384540 could you share your custom script to replace the IoTEdge pipeline task Generate deployment manifest?

[oliverchristen]

@randomstring384540 Oh, I just saw, its logged in the pipeline run:
/usr/local/bin/iotedgedev genconfig --file /home/vsts/work/1/s/<path to deployment template>/deployment.template.json --platform amd64 --fail-on-validation-error

[holmeren]

After 14 hours and no success, i tried updating Agent Specification fromubunto-18.04 -> ubunto-20.04, and it seems to work :)

[anatolybolshakov]

Hi @holmeren @oliverchristen is this issue still reproducible for you?

[oliverchristen]

@anatolybolshakov not for me at the moment.

[kiranpradeep]

@anatolybolshakov Yes. It is reproducible for me if I don't specify a build agent. It goes away if I add below to YAML.

pool:
  vmImage: ubuntu-latest

[anatolybolshakov]

@microsoft/akvelon-ecosystem-team this seem to be related to some difference between ubuntu-18.04 and ubuntu-20.04 - do you know if this is some known issue?

[miketimofeev]

@anatolybolshakov it's not known. We will take a look

[miketimofeev]

@kiranpradeep @holmeren I'm unable to reproduce the issue on Ubuntu 18 Image version 20210330.1.
These steps from the task work correctly and I'm able to call iotedgedev:

azure-pipelines-tasks/Tasks/AzureIoTEdgeV2/util.ts

Lines 88 to 94 in 8c190dc

	if (tl.osType() === Constants.osTypeLinux) {
	cmds = [
	{ path: `sudo`, arg: `apt-get update`, execOption: Constants.execSyncSilentOption },
	{ path: `sudo`, arg: `apt-get install -y python3-setuptools`, execOption: Constants.execSyncSilentOption },
	{ path: `sudo`, arg: `pip install --upgrade cryptography`, execOption: Constants.execSyncSilentOption},
	{ path: `sudo`, arg: `pip3 install ${Constants.iotedgedev}~=${version}`, execOption: Constants.execSyncSilentOption },
	]

Could you please clarify, on which version do you see the issue? It can be found here

[kiranpradeep]

Version: 20210327.1. Last run was at 2021-04-07T07:20:58.0398479Z. Also to be clear, the error is only if I don't mention vmImage.

[miketimofeev]

@kiranpradeep it looks like if you don't specify the vmImage the pipeline will run on Ubuntu 16.04, which is not supported by the task I guess.
@anatolybolshakov do you happen to know if the task worked on Ubuntu16?

[kiranpradeep]

As on 2021-03-12T12:47:51.4740078Z, the task worked for Ubuntu 16.04.7. The Version field then was 20210302.0.

[karakal]

As of today we have the same problem. Our Build-Pipeline is not running anymore with any agent (I event tried the Windows one...)
Environment: ubuntu-18.04
Version: 20210330.1

[miketimofeev]

@karakal could you please share the logs?

[miketimofeev]

@kiranpradeep @anatolybolshakov it looks like some of the pip modules drop support for python 3.5, which is the default one on Ubuntu 16

[karakal]

@karakal could you please share the logs?

Here you find my log. I removed all private parts of the log, I hope the requested information can be find within....

azure-pipeline.log

[karakal]

@kiranpradeep @anatolybolshakov it looks like some of the pip modules drop support for python 3.5, which is the default one on Ubuntu 16

I tried the build on:

• Ubuntu 20.04
• Ubuntu 18.04
• Ubuntu 16.04
• even Windows-2019

The build failed with all agents.

[miketimofeev]

@karakal what version of task do you use?
@anatolybolshakov it seems that for ubuntu 16 we need a workaround regarding cryptolib installation — these steps work:

sudo apt install python3-pip
sudo pip3 install --upgrade pip==20.3.4
sudo pip3 install --upgrade cryptography
sudo pip3 install --upgrade iotedgedev
iotedgedev --version

[karakal]

@karakal what version of task do you use?

2.* I don't have any other choice?

[miketimofeev]

@karakal Sorry, I meant the exact version, found it from the logs — 2021-04-08T07:00:59.8075822Z Version : 2.4.5
.
I'm asking because there are significant changes between versions 2.4.5 and 2.4.6.

[Vic152]

Hello All, We noticed the same issue:

This is a critical issue with no known workaround. What is the ETA for the fix?

[Vic152]

@marianan I tried building with Ubuntu 18.04 with following configuration and it worked:

It is not a critical issue for us anymore as the workaround exists. Thanks for your help I will keep you posted, should this issue reoccur.

[marianan]

@YarekTyshchenko and @Vic152 thanks for your follow ups. The issue on Ubuntu 20.04 seems to originate from the python version (3.8) but we have yet to narrow down the root cause. At this point Ubuntu 16.04 and Ubuntu 18.04 are expected to both work well with the Azure IoT Edge V2 task 2.4.7.

[oliverchristen]

@marianan @cindydeng1998 For an unknown reason this workaround still works for me with vmImage 20.04.

[karakal]

@marianan @cindydeng1998 For an unknown reason this workaround still works for me with vmImage 20.04.

For me too. But it doesn't work on Ubuntu 18.04....

[stewartadam]

At this point Ubuntu 16.04 and Ubuntu 18.04 are expected to both work well with the Azure IoT Edge V2 task 2.4.7.
edit: updating with test results

@marianan I just tried ubuntu-18.04 agent with task v2.4.7 and it did successfully pull down 3.0.0. It fails on 20.04 because the requires-python metadata in setup.py on the recently released packages exclude it from installation on higher python versions -- i.e. running this works: pip3 install --ignore-requires-python iotedgedev~=3.0.0

@oliverchristen that workaround (manually running a pip install iotedgedev) works because iotedgedev=2.1 doesn't appear to restrict its supported Python versions in its metadata, so it will get pulled in on Python >3.7. The iotedge build task appears to only verify that iotedgedev is present, not its particular version -- the recent versions of the build tasks expect 3.0.0 to be installed. So it will proceed, although I don't know what issues the version discrepancy may cause going forward.

[marianan]

@stewartadam you are correct; Ubuntu 20.04 defaults to python 3.8 which isn't yet supported by iotedgedev.

[stewartadam]

Since specific version of Python and/or dependencies are required, can this task be moved to using a Docker container?

[marianan]

Thanks @stewartadam, your suggestion has been added to our backlog. As there are no recent symptoms of this issue, I am closing it.

[timstokman]

@marianan I just got this issue again today, so it doesn't seem fixed:

2021-05-11T13:12:28.8129340Z ##[section]Starting: Azure IoT Edge - Generate deployment manifest
2021-05-11T13:12:28.8140814Z ==============================================================================
2021-05-11T13:12:28.8141199Z Task         : Azure IoT Edge
2021-05-11T13:12:28.8141510Z Description  : Build and deploy an Azure IoT Edge image
2021-05-11T13:12:28.8141810Z Version      : 2.4.7
2021-05-11T13:12:28.8142073Z Author       : Microsoft Corporation
2021-05-11T13:12:28.8142482Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
2021-05-11T13:12:28.8142923Z ==============================================================================
2021-05-11T13:12:31.7402790Z Start generating deployment manifest...
2021-05-11T13:12:31.7418328Z /home/vsts/work/1/s/code/edgesolution /home/vsts/work/1/s
2021-05-11T13:12:53.4315960Z ##[error]Error: Unable to locate executable file: 'iotedgedev'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
2021-05-11T13:13:54.4662950Z ##[section]Finishing: Azure IoT Edge - Generate deployment manifest

I tried removing the version pin I had for the AzureIoTEdge task (to 2.4.5, as a workaround for this issue), but it seems I have to revert that again.

[stewartadam]

@timstokman can you confirm your agent pool vmImage setting?

[timstokman]

@stewartadam It's set to "ubuntu-latest"

[stewartadam]

@timstokman the comments above elaborate in more detail, but due to Python version compatibility issues the build task will function on 16.04 and 18.04 but not 20.04 (which is what is tagged as 'ubuntu-latest').

[oliverchristen]

@marianan since this issue still persists on the default agent, I would suggest to reopen this issue and only close it when the fix is released. What do you think?

[marianan]

Hi @oliverchristen , I suggest filing a new issue to track the work for Ubuntu 20.04 support and linking it here. The root cause for this issue has been resolved so I feel that it will be more clear if we start a new thread. Also, supporting Ubuntu 20.04 is already in our backlog but it doesn't hurt to have a tracking thread on GitHub for everyone's awareness. Does this sound like a plan?

[oliverchristen]

Thanks for the suggestion @marianan. But I don't see the necessity to start a new thread. The initial issue still persists since it's probably related to Ubuntu 20.04. It would also be nice to link the backlog item for the Ubuntu 20.04 support here. It's not yet, isn't it?
For me it only works with the workaround
pip install -U iotedgedev
which I don't understand why this works - but it does. I would rather remove this workaround.

[karakal]

I have the same problem and the same workaround "works". It also doesn't work anymore at all with 18.0.4. I would also prefer to keep this bug open and not start another thread anywhere... Oliver Christen ***@***.***> schrieb am Mi., 12. Mai 2021, 07:47:

…

Thanks for the suggestion @marianan <https://github.com/marianan>. But I don't see the necessity to start a new thread. The initial issue still persists since it's probably related to Ubuntu 20.04. It would also be nice to link the backlog item for the Ubuntu 20.04 support here. It's not yet, isn't it? For me it only works with the workaround pip install -U iotedgedev which I don't understand why this works - but it does. I would rather remove this workaround. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASVXJNHVE7VAZJF4DVAMBDTNII6VANCNFSM4VSZ37BQ> .

[EmileLombard]

Hi everyone, if anyone is still having this issue, this is how I solved it:

Enable debugging logs (set build variable system.debug = true)

From the logs I found these lines:
##[debug]which 'iotedgedev'
##[debug]not found
##[debug]IOTEDGEDEV_VERSION=undefined
##[debug]The specified iotedgedev version is: 3.0.0

and:
##[debug]ERROR: Could not find a version that satisfies the requirement iotedgedev~=3.0.0 (from versions: 0.79.0, 0.80.0, 0.81.0, 0.82.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 2.0.0, 2.0.1, 2.0.2, 2.1.0)

So I just set a build variable to the latest version specified in the logs, in this case: IOTEDGEDEV_VERSION=2.1.0

I then just ran the build pipline again and it solved the issue.

I was running on
pool:
vmImage: 'ubuntu-latest'