Issue Getting Example Program to Run with Apple M3

[cmaloney111]

Describe the bug
The bug occurs when attempting to run the provided example program from the NekRS README using mpirun -np 2 nekrs --setup turbPipe.par. Initially, issues related to OpenMP linkage arise, manifesting as "no such file" errors. After attempting a fix involving setting environment variables and paths, a new error surfaces. This error indicates a problem with dlopen() and a missing symbol _adj_logical_ in the file libturbPipe.so.

To Reproduce
Steps to reproduce the behavior:

1. Follow the steps outlined in the README for building the NekRS repository.
2. Execute the example program with these steps (from the README):

   cd <directory outside of installation/source folder>
   cp -a $NEKRS_HOME/examples/turbPipePeriodic .
   mpirun -np 2 nekrs --setup turbPipe.par

3. Initially encounter OpenMP-related errors:

   dyld[90248]: Library not loaded: @rpath/libomp.dylib
     Referenced from: <BEA5CF92-0C49-30E9-A331-3C68742198F6> ~/.local/nekrs/lib/libocca.so
     Reason: tried: '~/.local/nekrs/lib/libomp.dylib' (no such file), '~/.local/nekrs/lib/libomp.dylib' (no such file), 

4. Apply the suggested fix as shown below (installing llvm and libomp using Homebrew, and setting environment variables).
5. Run the example program again, resulting in a new error:

   dlopen(~/.local/nekrs/examples/turbPipePeriodic/.cache/nek5000/libturbPipe.so, 0x0006): symbol not found in flat namespace '_adj_logical_'

Expected behavior
The example program is expected to execute successfully.

Suggested Fix For First Problem (Does Not Fix Second Problem)
To resolve the initial OpenMP linkage issue:

1. Install LLVM and libomp using Homebrew:

   brew install llvm libomp

2. Add the following environment variables to $HOME/.bash_profile for Bash:

   export LDFLAGS="-L$HOME/opt/homebrew/opt/libomp/lib $LDFLAGS"
   export CPPFLAGS="-I$HOME/opt/homebrew/opt/libomp/include $CPPFLAGS"
   export DYLD_LIBRARY_PATH="$HOME/opt/homebrew/opt/libomp/lib:$DYLD_LIBRARY_PATH"

   Alternatively, if using Zsh, place these lines in ~/.zshrc:

   export LDFLAGS="-L/opt/homebrew/opt/libomp/lib $LDFLAGS"
   export CPPFLAGS="-I/opt/homebrew/opt/libomp/include $CPPFLAGS"
   export DYLD_LIBRARY_PATH="/opt/homebrew/opt/libomp/lib:$DYLD_LIBRARY_PATH"

Finally, run either one of the following depending on what you are using:

source $HOME/.bash_profile

source ~/.zshrc

Desktop:

• OS: macOS
• Version: 15.1 (Sequoia)

[stgeke]

What Fortran compiler are you using?

[stgeke]

Regardless of the issue reported here, LLVM + ARM is currently not functional:
#485

[cmaloney111]

Was using GFortran. Thanks for letting me know that LLVM + ARM is currently not functional. I'll see if I can do anything on my end to get it to work.

[cmaloney111]

Quick update. Same issue occurring with the following system configuration:

Apple clang version 16.0.0 (clang-1600.0.26.4)
Target: arm64-apple-darwin24.1.0
open-mpi: 5.0.6
gcc: 14.2.0
cmake: 3.30.5

[stgeke]

I assume you're referring to the symbol issue. Which compilers are recognized by CMake for CXX, CC, and FC?

[cmaloney111]

Yes, I am referring to the missing symbol _adj_logical_ error

CC: AppleClang 16.0.0 located at /opt/homebrew/bin/mpicc

CXX: AppleClang 16.0.0 located at /opt/homebrew/bin/mpic++

FC: GNU Fortran 14.2.0 located at /opt/homebrew/bin/mpif77

[stgeke]

You need to use the GNU compilers for CC and CXX.

…

On Wed, 20 Nov 2024 at 00:09, Cameron Maloney ***@***.***> wrote: CC: AppleClang 16.0.0 located at /opt/homebrew/bin/mpicc CXX: AppleClang 16.0.0 located at /opt/homebrew/bin/mpic++ FC: GNU Fortran 14.2.0 located at /opt/homebrew/bin/mpif77 — Reply to this email directly, view it on GitHub <#606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M76T5KXHV3QEQXNCZAT2BPALDAVCNFSM6AAAAABR7A336GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBWHEZTGNJZG4> . You are receiving this because you commented.Message ID: ***@***.***>

[cmaloney111]

Unfortunately, this does not fix the problem. I have built NekRS with GNU compilers for C, C++, and Fortran (with the mpi wrappers and after clearing the build/installation beforehand), as can be seen in the following screenshot:

And I still receive the following error when running the example program (turbPipePeriodic) from the README:

building nekInterface for lx1=8, lelt=3923 and lelg=7840 ... done (12.8478s)
loading nek ... The value of session_in is: turbPipe
The value of session_in is: turbPipe
Aborting in ~/nekRS/src/nekInterface/nekInterfaceAdapter.cpp:set_usr_handles
dlopen(~/nekRS/turbPipePeriodic/.cache/nek5000/libturbPipe.so, 0x0006): symbol not found in flat namespace '_adj_logical_'

[stgeke]

Please try again but delete .cache (in the case dir) first.

…

On Thu, 21 Nov 2024 at 11:56, Cameron Maloney ***@***.***> wrote: Unfortunately, this does not fix the problem. I have built NekRS with GNU compilers for C, C++, and Fortran (with the mpi wrappers and after clearing the build/installation beforehand), as can be seen in the following screenshot: Screenshot.2024-11-21.at.2.51.57.AM.png (view on web) <https://github.com/user-attachments/assets/88af37ab-43c0-4c64-9b94-9f6fffbc0f9d> And I still receive the following error when running the example program (turbPipePeriodic) from the README: building nekInterface for lx1=8, lelt=3923 and lelg=7840 ... done (12.8478s) loading nek ... The value of session_in is: turbPipe The value of session_in is: turbPipe Aborting in ~/nekRS/src/nekInterface/nekInterfaceAdapter.cpp:set_usr_handles dlopen(~/nekRS/turbPipePeriodic/.cache/nek5000/libturbPipe.so, 0x0006): symbol not found in flat namespace '_adj_logical_' — Reply to this email directly, view it on GitHub <#606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M77ENGO6S56EB4VUJW32BW36ZAVCNFSM6AAAAABR7A336GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJQG44TEOJYGI> . You are receiving this because you commented.Message ID: ***@***.***>

[cmaloney111]

Unfortunately, the same error as above is occurring.

On a separate note, I have to remove any reference to mtune in ~/.local/nekrs/nekrs.conf each time I run the build, but that's not a problem here. The same _adj_logical_ error is occurring.

Let me know if I can provide any other info to help.

[stgeke]

Not sure what's going wrong here. Unfortunately I don't have access to a M3 based Mac.

[cmaloney111]

Okay, I’ll keep trying. I appreciate the help and patience.

[stgeke]

Can you please run

>otool -Iv .cache/nek5000/*.so | grep _adj_logical_
1488:0x00000000001b8018 10075 _adj_logical_

in your case dir.

[cmaloney111]

This was the output (found in the libturbPipe.so file):

0x00000000001bc018 10046 _adj_logical_