Trouble Installing develop branch code in MacOS

[DeepFriedDerp]

Description

Attempting to install latest commit of develop branch, have successfully installed main branch as of last commit.

Current behavior

python setup.py develop returns the following error

ld: warning: -pie being ignored. It is only used when linking a main executable
ld: warning: object file (pyhams/hams.cpython-310-darwin.so.p/meson-generated..__hamsmodule.c.o) was built for newer macOS version (12.0) than being linked (11.5)
ld: warning: object file (pyhams/hams.cpython-310-darwin.so.p/fc6b8e8fd2203a6e30e2ed9e6b5cf4f4e5e23faf.._.._f2py_src_fortranobject.c.o) was built for newer macOS version (12.0) than being linked (11.5)
ld: can't map file, errno=22 file '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework' for architecture x86_64
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /opt/anaconda3/envs/weis-env2/bin/ninja -C /Users/agmotes/gitRepos/WEIS-develop/pyHAMS/meson_build -v

Traceback (most recent call last):
File "/Users/agmotes/gitRepos/WEIS-develop/pyHAMS/setup.py", line 90, in
run_meson_build(staging_dir)
File "/Users/agmotes/gitRepos/WEIS-develop/pyHAMS/setup.py", line 60, in run_meson_build
raise OSError(
OSError: [Errno ['/opt/anaconda3/envs/weis-env2/bin/meson', 'compile', '-vC', 'meson_build']] The meson compile command failed! Check the log at meson_build/compile.log for more information.

Traceback (most recent call last):

File "/Users/agmotes/gitRepos/WEIS-develop/setup.py", line 136, in
subprocess.check_call([sys.executable, "setup.py", "develop"])
File "/opt/anaconda3/envs/weis-env2/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/anaconda3/envs/weis-env2/bin/python', 'setup.py', 'develop']' returned non-zero exit status 1.

Error message in meson_build.compile.log is

ld: warning: -pie being ignored. It is only used when linking a main executable
ld: warning: object file (pyhams/hams.cpython-310-darwin.so.p/meson-generated..__hamsmodule.c.o) was built for newer macOS version (12.0) than being linked (11.5)
ld: warning: object file (pyhams/hams.cpython-310-darwin.so.p/fc6b8e8fd2203a6e30e2ed9e6b5cf4f4e5e23faf.._.._f2py_src_fortranobject.c.o) was built for newer macOS version (12.0) than being linked (11.5)
ld: can't map file, errno=22 file '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Accelerate.framework' for architecture x86_64
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /opt/anaconda3/envs/weis-env2/bin/ninja -C /Users/agmotes/gitRepos/WEIS-develop/pyHAMS/meson_build -v

Code versions

• MacOS Monterey (12.6.2)
• Python 3.10.6

[gbarter]

Thanks for posting this. Do you have a mac with a new M1/2 processor or an older Intel processor?

[DeepFriedDerp]

Hi gbarter!

Its a 2019 MacBook Pro with a 2.4 GHz 8-Core Intel Core i9

[gbarter]

From what I can tell, I don't think this is a WEIS problem. It looks like you might have had a system update or Xcode update and conda needs to be refreshed. Here is what I would suggest:

sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install
conda update -n base conda

I would repeat that last command until you get a message that says it is fully up to date. I would then start a fresh new environment for WEIS instead of using the old one. Apologies that this means some annoying reinstallation.

[DeepFriedDerp]

@gbarter

I attempted to follow the suggested instructions. Everything worked out as expected.

Then I removed the old weis-env2 environment from the list of conda envs, and created a new environment called weis-env-develop, using the installation instructions fromt the develop branch readme.

When I get to the part where I cd into the repo and run setup.py, I get the following returned

(weis-env-develop) agmotes@ ~ % cd gitRepos/WEIS-develop

(weis-env-develop) agmotes@ WEIS-develop % ls
LICENSE OpenFAST README.md WISDEM dtqpy examples pyHAMS setup.py
MoorPy RAFT ROSCO docs environment.yml pCrunch pyproject.toml weis
(weis-env-develop) agmotes@ WEIS-develop % python setup.py develop
/Users/agmotes/gitRepos/WEIS-develop/setup.py:5: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import run_setup
/Users/agmotes/gitRepos/WEIS-develop/setup.py:7: DeprecationWarning:

numpy.distutils is deprecated since NumPy 1.23.0, as a result
of the deprecation of distutils itself. It will be removed for
Python >= 3.12. For older Python versions it will remain present.
It is recommended to use setuptools < 60.0 for those Python versions.
For more details, see:
https://numpy.org/devdocs/reference/distutils_status_migration.html

from numpy.distutils.command.build_ext import build_ext
['setup.py', 'develop']
['/opt/anaconda3/envs/weis-env-develop/bin/meson', 'setup', 'meson_build', '--prefix=/Users/agmotes/gitRepos/WEIS-develop/WISDEM/meson_build', '-Dpython.purelibdir=.', '-Dpython.platlibdir=.', '--buildtype', 'release']
['/opt/anaconda3/envs/weis-env-develop/bin/meson', 'compile', '-vC', 'meson_build']
ninja: Entering directory `/Users/agmotes/gitRepos/WEIS-develop/WISDEM/meson_build'
[0/1] /opt/anaconda3/envs/weis-env2/bin/meson --internal regenerate /Users/agmotes/gitRepos/WEIS-develop/WISDEM /Users/agmotes/gitRepos/WEIS-develop/WISDEM/meson_build
/bin/sh: /opt/anaconda3/envs/weis-env2/bin/meson: No such file or directory
FAILED: build.ninja
/opt/anaconda3/envs/weis-env2/bin/meson --internal regenerate /Users/agmotes/gitRepos/WEIS-develop/WISDEM /Users/agmotes/gitRepos/WEIS-develop/WISDEM/meson_build
ninja: error: rebuilding 'build.ninja': subcommand failed
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /opt/anaconda3/envs/weis-env-develop/bin/ninja -C /Users/agmotes/gitRepos/WEIS-develop/WISDEM/meson_build -v

Traceback (most recent call last):
File "/Users/agmotes/gitRepos/WEIS-develop/WISDEM/setup.py", line 90, in
run_meson_build(staging_dir)
File "/Users/agmotes/gitRepos/WEIS-develop/WISDEM/setup.py", line 61, in run_meson_build
raise OSError(
OSError: [Errno ['/opt/anaconda3/envs/weis-env-develop/bin/meson', 'compile', '-vC', 'meson_build']] The meson compile command failed! Check the log at meson_build/compile.log for more information.
Traceback (most recent call last):
File "/Users/agmotes/gitRepos/WEIS-develop/setup.py", line 136, in
subprocess.check_call([sys.executable, "setup.py", "develop"])
File "/opt/anaconda3/envs/weis-env-develop/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/anaconda3/envs/weis-env-develop/bin/python', 'setup.py', 'develop']' returned non-zero exit status 1.

It appears that a few packages are trying to look into the old weis-env2 conda environment directory that no longer exists, but I'm not sure I know how to fix that, any suggestions?

[gbarter]

Looks like you need to do conda install meson ninja. Not sure why those weren't installed otherwise.

[DeepFriedDerp]

Hi @gbarter,

I ran the conda install meson ninja command while in the weis-env-develop environment, and was told that these packages are already installed.

I did find an autogenerated file within the repo (/WISDEM/meson_build/build.ninja) that keeps referring to the weis-env2 directory within anaconda's environment directory (this was deleted per your instructions). It seems that something is cached either in the build files within WEIS, or in Anaconda, that keeps telling the ninja/meson to look for files in a deleted environment directory, even though I'm currently in a new environment.

I attempted to remove all anaconda environments, ran the conda clean --all command, and run the recommended conda update command. This appears to have completely broken anaconda beyond repair. I was forced to rm -rf anaconda3 in the anaconda parent directory and reinstall. I also deleted the copy I had of the WEIS develop branch repo and start with a fresh copy, to remove any potential build files that may point to that old deleted environment.

Once I had a fresh install of anaconda in place, created a fresh environment "weis-env-develop", and attempted to run python setup.py develop inside of the fresh copy of the WEIS repo, I keep getting the following errors.

Installed /Users/agmotes/gitRepos/WEIS-develop/WISDEM
Processing dependencies for WISDEM==3.8
Searching for certifi>=2017.4.17
Reading https://pypi.org/simple/certifi/
Download error on https://pypi.org/simple/certifi/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) -- Some packages may not be found!
Couldn't find index page for 'certifi' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) -- Some packages may not be found!
No local packages or working download links found for certifi>=2017.4.17
error: Could not find suitable distribution for Requirement.parse('certifi>=2017.4.17')
Traceback (most recent call last):
File "/Users/agmotes/gitRepos/WEIS-develop/setup.py", line 136, in
subprocess.check_call([sys.executable, "setup.py", "develop"])
File "/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/agmotes/opt/anaconda3/envs/weis-env-develop/bin/python', 'setup.py', 'develop']' returned non-zero exit status 1.

I attempted to install certifi manually, and was told that:

Requirement already satisfied: certifi in /Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages (2022.12.7)

I also attempted installing ruamel.yaml manually, and got the following response:

(weis-env-develop) agmotes@ WEIS-develop % pip install ruamel.yaml
Collecting ruamel.yaml
Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 kB 487.7 kB/s eta 0:00:00
Collecting ruamel.yaml.clib>=0.2.6
Downloading ruamel.yaml.clib-0.2.7-cp310-cp310-macosx_10_9_universal2.whl (141 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 141.3/141.3 kB 699.0 kB/s eta 0:00:00
Installing collected packages: ruamel.yaml.clib, ruamel.yaml
Successfully installed ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.7

Once that was complete, I attempted to run python setup.py develop again, and got the same error message quoted above from the previous attempt to compile.

I'm out of ideas at this point, haha.

[gbarter]

I really appreciate your patience and tenacity here. With @ptrbortolotti 's help, I was finally able to recreate your error on the Github CI servers, so I will work to resolve it. Definitely not your fault here.

[DeepFriedDerp]

@gbarter that's great news! I'll keep an eye out for the next commit 👍

[gbarter]

Okay, go ahead and do a git pull on the develop branch and give it another try. I would also recommend doing: rm -rf build */build */meson_build from the WEIS directory.

[DeepFriedDerp]

@gbarter

There were a few red-text errors during this new compile attempt, and while the compile did run through to the end, I'm getting runtime errors when trying to run example 7, I'll list what I have here:

Errors during compile

Installed /Users/agmotes/gitRepos/WEIS-develop/dtqpy
Processing dependencies for dtqpy==0.0.1
Searching for mat4py
Reading https://pypi.org/simple/mat4py/
Download error on https://pypi.org/simple/mat4py/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) -- Some packages may not be found!
Couldn't find index page for 'mat4py' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) -- Some packages may not be found!
No local packages or working download links found for mat4py

INFO: compile options: '-I/Users/agmotes/opt/anaconda3/envs/weis-env-develop/include/python3.10 -I/Users/agmotes/opt/anaconda3/envs/weis-env-develop/include/python3.10 -c'
extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -Werror=switch -Werror'
WARN: CCompilerOpt.dist_test[630] : CCompilerOpt._dist_test_spawn[764] : Command (x86_64-apple-darwin13.4.0-clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/agmotes/opt/anaconda3/envs/weis-env-develop/include -fPIC -O2 -isystem /Users/agmotes/opt/anaconda3/envs/weis-env-develop/include -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/agmotes/opt/anaconda3/envs/weis-env-develop/include -D_FORTIFY_SOURCE=2 -isystem /Users/agmotes/opt/anaconda3/envs/weis-env-develop/include -ftrapping-math -I/Users/agmotes/opt/anaconda3/envs/weis-env-develop/include/python3.10 -I/Users/agmotes/opt/anaconda3/envs/weis-env-develop/include/python3.10 -c /Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c -o /var/folders/c5/pckmh0894zv_8tbzlq4hbdph003hhf/T/tmp9b6o6ac8/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.o -MMD -MF /var/folders/c5/pckmh0894zv_8tbzlq4hbdph003hhf/T/tmp9b6o6ac8/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -Werror=switch -Werror) failed with exit status 1 output ->
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c:22:9: error: implicit declaration of function '_mm512_4fmadd_ps' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
b = _mm512_4fmadd_ps(b, b, b, b, b, NULL);
^
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c:22:9: note: did you mean '_mm512_fmadd_ps'?
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/clang/14.0.6/include/avx512fintrin.h:2720:1: note: '_mm512_fmadd_ps' declared here
_mm512_fmadd_ps(__m512 __A, __m512 __B, __m512 __C)
^
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c:22:7: error: assigning to '__m512' (vector of 16 'float' values) from incompatible type 'int'
b = _mm512_4fmadd_ps(b, b, b, b, b, NULL);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c:24:9: error: implicit declaration of function '_mm512_4dpwssd_epi32' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
a = _mm512_4dpwssd_epi32(a, a, a, a, a, NULL);
^
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c:24:9: note: did you mean '_mm512_dpwssd_epi32'?
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/clang/14.0.6/include/avx512vnniintrin.h:68:1: note: '_mm512_dpwssd_epi32' declared here
_mm512_dpwssd_epi32(__m512i __S, __m512i __A, __m512i __B)
^
/Users/agmotes/opt/anaconda3/envs/weis-env-develop/lib/python3.10/site-packages/numpy/distutils/checks/cpu_avx512_knm.c:24:7: error: assigning to '__m512i' (vector of 8 'long long' values) from incompatible type 'int'
a = _mm512_4dpwssd_epi32(a, a, a, a, a, NULL);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 errors generated.

WARN: CCompilerOpt.feature_test[1563] : testing failed

Installed /Users/agmotes/gitRepos/WEIS-develop
Processing dependencies for WEIS==1.1
Searching for certifi>=2017.4.17
Reading https://pypi.org/simple/certifi/
Download error on https://pypi.org/simple/certifi/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) -- Some packages may not be found!
Couldn't find index page for 'certifi' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) -- Some packages may not be found!
No local packages or working download links found for certifi>=2017.4.17
error: Could not find suitable distribution for Requirement.parse('certifi>=2017.4.17')

Errors during run of example 07_te_flaps

Using weis.aeroelasticse in ROSCO_toolbox...
Traceback (most recent call last):
File "/Users/agmotes/gitRepos/WEIS-develop/examples/07_te_flaps/dac_driver.py", line 1, in
from weis.glue_code.runWEIS import run_weis
File "/Users/agmotes/gitRepos/WEIS-develop/weis/glue_code/runWEIS.py", line 7, in
from weis.glue_code.glue_code import WindPark
File "/Users/agmotes/gitRepos/WEIS-develop/weis/glue_code/glue_code.py", line 14, in
from weis.aeroelasticse.openmdao_openfast import FASTLoadCases
File "/Users/agmotes/gitRepos/WEIS-develop/weis/aeroelasticse/openmdao_openfast.py", line 32, in
from weis.control.dtqp_wrapper import dtqp_wrapper
File "/Users/agmotes/gitRepos/WEIS-develop/weis/control/dtqp_wrapper.py", line 3, in
from dtqpy.src.DTQPy_oloc import DTQPy_oloc
File "/Users/agmotes/gitRepos/WEIS-develop/dtqpy/dtqpy/src/DTQPy_oloc.py", line 9, in
from mat4py import loadmat
ModuleNotFoundError: No module named 'mat4py'

[gbarter]

I'm not sure why mat4py wasn't picked up. Maybe there was a temporary blip in its availability or the network? You can try conda install mat4py or pip install mat4py

[DeepFriedDerp]

@gbarter its a really strange error, and I keep getting errors about not being able to download pytest and certifi, even though when I attempt to install them manually, I get a message that they are already installed.

Anyway, I attempted to run example 07_te_flaps, and it seems to have run successfully with no errors about missing dependencies, so I guess these errors are more like warnings?

Thanks for your help through all of this! I'll close this issue, and raise another if I run into any additional problems, since at this point any problems should be either runtime or on a different branch.