macOS setup fails to build for lack of "elftools" Python package

[snej]

Describe the bug
I'm just getting started, on macOS 10.15. I installed the latest GCC-ARM toolchain as directed.
However, building fails due to lack of the Python elftools package. This package should be added to requirements.txt, or at least mentioned on the 3rd-party-toolchains page if it's specific to using the GCC toolchain.

To Reproduce

1. Install zephyr on macOS, as directed in the Getting Started Guide.
2. Install the latest GCC-ARM toolchain and set up environment variables as directed here.
3. Continue following the Guide, up to running west build -b bbc_microbit samples/basic/blinky

Expected behavior
The "blinky" program builds.

Impact
Can't build anything, without digging through Python exception dumps looking for clues.

Screenshots or console output

$  west build -b bbc_microbit samples/basic/blinky
FATAL ERROR: extension command "build" couldn't be run
  Hint: could not import /Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/build.py
  See /var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/west-exc-ecpozvz2.txt for a traceback.

$ cat /var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/west-exc-ecpozvz2.txt
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/west/commands.py", line 369, in __call__
    mod = _commands_module_from_file(self.py_file)
  File "/usr/local/lib/python3.7/site-packages/west/commands.py", line 338, in _commands_module_from_file
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/build.py", line 15, in <module>
    from zephyr_ext_common import Forceable
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/zephyr_ext_common.py", line 17, in <module>
    from runners.core import RunnerConfig
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/runners/__init__.py", line 26, in <module>
    from runners import openocd
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/runners/openocd.py", line 8, in <module>
    from elftools.elf.elffile import ELFFile
ModuleNotFoundError: No module named 'elftools'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/west/app/main.py", line 334, in run_command
    self.run_extension(args.command, argv)
  File "/usr/local/lib/python3.7/site-packages/west/app/main.py", line 373, in run_extension
    command = self.extensions[name].factory()
  File "/usr/local/lib/python3.7/site-packages/west/commands.py", line 372, in __call__
    hint=f'could not import {self.py_file}') from ie
west.commands.ExtensionCommandError

Environment (please complete the following information):

• MacOS 10.15.3
• GCC-ARM toolchain, downloaded today
• Commit b96477d (HEAD -> master, origin/master)

[snej]

Also maybe worth noting is that installing elftools also failed, for lack of numpy. I realize this isn't your bug, rather a bug in the elftools package description that it doesn't mention that dependency, but you might want to work around it by explicitly adding numpy as a dependency of your own.

$  pip3 install elftools
Collecting elftools
  Downloading https://files.pythonhosted.org/packages/75/73/6d6103991bbc9bf0de609bfd74400178a2870ca487913cdd8ab84a99c718/elftools-0.1.0.dev0.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/local/opt/python/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/pip-install-yhygbg5s/elftools/setup.py'"'"'; __file__='"'"'/private/var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/pip-install-yhygbg5s/elftools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/pip-install-yhygbg5s/elftools/pip-egg-info
         cwd: /private/var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/pip-install-yhygbg5s/elftools/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/pip-install-yhygbg5s/elftools/setup.py", line 13, in <module>
        import numpy
    ModuleNotFoundError: No module named 'numpy'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[snej]

OK, even after installing elftools, the build fails with exactly the same error. WTF? I'm now officially blocked.

$  west build -b bbc_microbit samples/basic/blinky
FATAL ERROR: extension command "build" couldn't be run
  Hint: could not import /Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/build.py
  See /var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/west-exc-nqoyo8dw.txt for a traceback.

$  cat /var/folders/14/5kpdsdb13gqfmg00gx2_kdlr0000gn/T/west-exc-nqoyo8dw.txt
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/west/commands.py", line 369, in __call__
    mod = _commands_module_from_file(self.py_file)
  File "/usr/local/lib/python3.7/site-packages/west/commands.py", line 338, in _commands_module_from_file
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/build.py", line 15, in <module>
    from zephyr_ext_common import Forceable
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/zephyr_ext_common.py", line 17, in <module>
    from runners.core import RunnerConfig
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/runners/__init__.py", line 26, in <module>
    from runners import openocd
  File "/Users/snej/Couchbase/Embedded/zephyrproject/zephyr/scripts/west_commands/runners/openocd.py", line 8, in <module>
    from elftools.elf.elffile import ELFFile
ModuleNotFoundError: No module named 'elftools'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/west/app/main.py", line 334, in run_command
    self.run_extension(args.command, argv)
  File "/usr/local/lib/python3.7/site-packages/west/app/main.py", line 373, in run_extension
    command = self.extensions[name].factory()
  File "/usr/local/lib/python3.7/site-packages/west/commands.py", line 372, in __call__
    hint=f'could not import {self.py_file}') from ie
west.commands.ExtensionCommandError

$  pip3 install elftools
Requirement already satisfied: elftools in /usr/local/lib/python3.7/site-packages (0.1.0.dev0)
Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.7/site-packages (from elftools) (3.1.3)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/site-packages (from elftools) (1.18.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/site-packages (from matplotlib>=2.2.3->elftools) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib>=2.2.3->elftools) (1.1.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/site-packages (from matplotlib>=2.2.3->elftools) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib>=2.2.3->elftools) (2.4.6)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib>=2.2.3->elftools) (1.13.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib>=2.2.3->elftools) (42.0.2)

[snej]

I was able to work around this by adding numpy and elftools to the scripts/requirements.txt file and then repeating the pip3 install --user -r zephyr/scripts/requirements.txt command.

[carlescufi]

@snej I just went through the procedure myself and did not find any issues. The package required is actually called pyelftools which is indeed in our scripts/requirements.txt:
https://github.com/zephyrproject-rtos/zephyr/blob/master/scripts/requirements.txt#L13

I suspect your initial installation of requirements.txtdid not complete correctly, for whatever reason, and then you tried to install the elftools package which is not required by Zephyr (pyelftools is), and seems to have numpy as a dependency (numpy is not a Zephyr dependency).

@mbolivar can you think of another reason for this to fail based on the logs pasted by @snej?

EDIT:

I see that your brew python3 executable seems to be in /usr/local/opt/python
In my case I seem to have it elswhere:

$ ls -la `which python3`
lrwxr-xr-x  1 carles  admin  36 Jan  9 12:54 /usr/local/bin/python3 -> ../Cellar/python/3.7.6_1/bin/python3

EDIT 2: Actually, I do have a Python 3 there:

$ ls -la /usr/local/opt/python/bin/python3
lrwxr-xr-x  1 carles  staff  55 Dec 18 20:23 /usr/local/opt/python/bin/python3 -> ../Frameworks/Python.framework/Versions/3.7/bin/python3

I wonder if there's some collision with Catalina's built-in python3, which in my case is located in /usr/bin/python3.

[carlescufi]

Re-labeling as a question since this does not seem to be an actual issue with our requirements file.

[carlescufi]

I was able to work around this by adding numpy and elftools to the scripts/requirements.txt file and then repeating the pip3 install --user -r zephyr/scripts/requirements.txt command.

As described in my comment above, this is really not required. Could you please try removing those extra lines from scripts/requirements.txt, uninstalling numpy and elftools and seeing if you can still build? I tested with a clean environment and was able to build without those modules, which are certainly not required by Zephyr.

[snej]

I see that your brew python3 executable seems to be in /usr/local/opt/python

Hmm ... which python3 tells me it's in /usr/local/bin just like yours.
Aha: /usr/local/opt/python is a symlink to the actual python3 binary — as is /usr/local/bin/python3. They both point to the same place, /usr/local/Cellar/python/3.7.6_1/bin/python3. So this is a red herring.

[snej]

I suspect your initial installation of requirements.txtdid not complete correctly, for whatever reason

I know I got an error while installing HomeBrew packages, resolved it, and repeated the command. I don't recall an error running requirements.txt, but if I had gotten one I would have repeated the command to make sure it completed.

[carlescufi]

So this is a red herring.

Indeed, sorry for the noise.

I suspect your initial installation of requirements.txtdid not complete correctly, for whatever reason

I know I got an error while installing HomeBrew packages, resolved it, and repeated the command. I don't recall an error running requirements.txt, but if I had gotten one I would have repeated the command to make sure it completed.

I see, then I am out of ideas. I really don't know what happened there. I assume you cannot reproduce it either?

[snej]

I can't think of a way to reproduce it without first erasing my entire HomeBrew or Python3 setup (which I cannot do because I have plenty of other stuff dependent on those) or finding another Mac to install Zephyr on (I do have an iMac, but I can't upgrade it to Catalina as I have music software on it that isn't compatible.)

[carlescufi]

@snej are you using virtualenv or similar? I was googling for previous occurrences of this problem and found this:
#6866 (comment)

EDIT: Just to make sure, you didn't use --user when invoking pip3 right? I seem to recall that being another source of trouble on macOS.

[mbolivar]

I can't think of a way to reproduce it without first erasing my entire HomeBrew or Python3 setup (which I cannot do because I have plenty of other stuff dependent on those)

I would try to reproduce in a virtualenv. The package is definitely 'pyelftools', not 'elftools', and it sounds like there might be a lot going on in the homebrew environment that could be complicating things.

If it doesn't repro in a virtualenv, then I have no idea either about what happened the first time. I've set up macs from scratch before and
don't believe I've run into this, but they're not my primary dev environments.

[carlescufi]

@snej and @mbolivar

I actually managed to get a clean macOS machine and run the whole procedure locally. Everything worked as expected, with no issues, except for this (seemingly harmless) error message:

$ pip3 install -r zephyr/scripts/requirements.txt
[...all packages get installed, including pyelftools...]
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/LICENSE'
Consider using the `--user` option or check the permissions.

The error above appeared but all the actual python dependencies were indeed installed correctly. Googling a bit didn't really help much, so I don't know why this appeared.
I then proceeded to build a couple of samples (using GNU Arm Embedded) without issues. I used the Settings -> Security and Privacy -> Developer Tools method to allow unsigned binaries to run on the Terminal.

Since I now tried on a clean machine, running Catalina, and no Zephyr-attributable issues were found, I will close this issue.
I have the full logs of the installation if you are interested.

[EarthLord]

I had the same issue after a fresh install and solved it by pip3 install numpy pyelftools

[Dinduks]

If you have this issue, make sure your local python version is the same as the one west uses, then reinstall the requirements. In my case, my shell was using Python 3.9 (which means all dependencies were installed for 3.9) but west 3.10.

[TrailBee47]

I also fixed this issue after struggling .. basically for me I had python3.7 and python3.9 installed .. so zephyr was trying to use 3.9 and missed pyelftools. installing with pip3 shows success but eventually fail to compile the I did this "pip3.9 install pyelftools" and it worked :D