Update to OpenFAST-dev

[rafmudaf]

Purpose

This pull request updates the git-subtree for OpenFAST to the latest dev branch.

Type of change

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (non-backwards-compatible fix or feature)
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Documentation update
• Maintenance update
• Other (please describe)

Testing

The input files for OpenFAST may have changed since the last update. The changes from v2.6 to v3.0 are listed here, but these may have been included in OpenFAST the last time it was updated in WEIS. There are currently no input file changes between v3.0 and the top of dev. So, running the existing WEIS test suite should verify whether this change is compatible.

Checklist

• I have run existing tests which pass locally with my changes
• I have added new tests or examples that prove my fix is effective or that my feature works
• I have added necessary documentation

[rafmudaf]

I am working on running the full test suite on my computer. While I fight with my compilers and environment, it would be helpful to let GitHub also run the tests and we'll see who wins.

[rafmudaf]

For context, I'm looking to update WEIS with my feature/python_driver branch. Then if everything looks good, I'll start working on integrating my branch to the main OpenFAST repo.

[dzalkind]

Hi @rafmudaf, thanks for updating this! I hate to be the bearer of bad news, but I have been having some minor issues running OpenFAST on eagle using the python library.

When I call OpenFAST from the python library in WEIS, it seems to initialize all the submodules and around the first timestep, I get the following error:

+   t=58  FairTen 1: 2.43677E+06, 2.43571E+06, 2.44251E+06                                         
+   t=59  FairTen 1: 2.44120E+06, 2.43677E+06, 2.43571E+06                                         
+   t=60  FairTen 1: 2.43587E+06, 2.44120E+06, 2.43677E+06                                         
    Fairlead tensions did not converge within TMaxIC=60 seconds.
  Time: 0 of 10.1 seconds.
INTEL MKL ERROR: /nopt/nrel/apps/compilers/intel/2020.1.217/mkl/lib/intel64/libmkl_avx512.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx512.so or libmkl_def.so.

When I run using the executable, it works without any issues.

Perhaps it's something about our/my eagle setup because it seems to be working fine on Linux in the CI. Do you have any ideas? I think my slurm script is pretty basic: submit.txt

[gbarter]

@dzalkind : Is this with a fresh checkout and build? Are you connecting to Eagle in a weird way- say through Visual Studio? Just trying to see if this is related to issues that consumed a lot of time for Pietro & Eliot a while back.

[dzalkind]

Yeah, everything is fresh. I built WEIS using VSCode, but I called the submit script using the terminal to avoid the usual MPI error issues that usually occur as soon as weis is imported. This issue seems to be happening right in the middle of an OpenFAST run.

I don't know if this issue is a show stopper. I have a workaround. I figured it would be a good time to bring it up so it's somewhat documented here and to see if Raf has any quick fix ideas. If not, I say we proceed and maybe switch the default WEIS to using the executable.

[rafmudaf]

Thanks for pointing this out @dzalkind. When you run on Eagle with the executable, where are you getting the executable? Is it also compiled with WEIS?

It looks like it's a problem with the Intel toolchain module not being loaded into the current environment.

[dzalkind]

Yes, it is also compiled with WEIS and lives in local/bin/.

I tried adding this to my submit script, with the same end result:

export CC=icc
export CXX=icpc
export FC=ifort

[rafmudaf]

Is the OpenFAST executable also compiled with Intel? You can check by passing the version flag: ./local/bin/openfast -v. If so, then it's interesting that it is not running into the same error.

[dzalkind]

It looks like it was compiled with intel:

 **************************************************************************************************
 OpenFAST

 Copyright (C) 2021 National Renewable Energy Laboratory
 Copyright (C) 2021 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

 OpenFAST-v0.2-2362-g6f47159d
 Compile Info:
  - Compiler: Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64,
  Version 19.1.1.217 Build 20200306
  - Architecture: 64 bit
  - Precision: single
  - OpenMP: Yes, number of threads: 36/36
  - Date: Oct 25 2021
  - Time: 10:48:56
 Execution Info:
  - Date: 10/25/2021
  - Time: 11:18:51-0600

[johnjasa]

I checked this today and I have the same issue locally. Not sure if I can help or provide more info. This is with me compiling using GCC though, maybe that introduces a different problem.

 FAST_InitializeAll:HydroDyn_Init:Waves_Init:VariousWaves_Init: The random number generator in use
 differs from the original code provided by NREL. This pRNG uses 33 seeds instead of the 2 in the
 HydroDyn input file.

  Time: 0 of 10.01 seconds.
INTEL MKL ERROR: /home/john/anaconda3/lib/libmkl_avx2.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
(weis) john@JJASA-35609S:/mnt/c/Users/jjasa/git/WEIS/examples/06_IEA-15-240-RWT$ ../../local/bin/openfast -v

 **************************************************************************************************
 OpenFAST

 Copyright (C) 2021 National Renewable Energy Laboratory
 Copyright (C) 2021 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

 OpenFAST-v0.2-2413-gb92cbb96
 Compile Info:
  - Compiler: GCC version 7.5.0
  - Architecture: 64 bit
  - Precision: single
  - OpenMP: Yes, number of threads: 12/12
  - Date: Oct 19 2021
  - Time: 09:27:40
 Execution Info:
  - Date: 10/25/2021
  - Time: 13:53:25-0600


  OpenFAST terminated normally.

[dzalkind]

To be clear, I was seeing this issue before this PR as well. I can't remember exactly when it started to break because my eagle use is intermittent, but I think it was around a month or so ago when there seemed to be some other changes with eagle. I hope this helps.

[gbarter]

Sounds like there was maybe an MKL upgrade or script change on Eagle and there is a directory or library missing in the LD_LIBRARY_PATH. I will look into it, but cannot promise a quick response

[nikhar-abbas]

Well, just to add to this, I'm able to run the current develop branch fine locally with intel compilers, but when I bump it to eagle and compile and run I get the same errors that @dzalkind is getting. I've spent some time diving into this and haven't solved anything, but maybe this info will help...

The modules I have loaded when I compile and run on eagle are:

Currently Loaded Modules:
  1) cmake/3.18.2   2) conda/4.9.2   3) comp-intel/2020.1.217   4) intel-mpi/2020.1.217   5) mkl/2020.1.217

I came across this post that sent me down a bit of a rabbit hole (though no solution, yet...).

I realized that locally I had the following conda-forge channels:

           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          file:///opt/intel/oneapi/conda_channel/osx-64
                          file:///opt/intel/oneapi/conda_channel/noarch
                          https://conda.anaconda.org/intel/osx-64
                          https://conda.anaconda.org/intel/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch

Based on the post linked above and the intel locally (where things work) I figured I should try adding the intel anaconda channel. This changes how some libraries get downloaded and linked with intel (e.g. numpy), so seems promising? Here is my conda channel list on eagle now:

           channel URLs : https://conda.anaconda.org/intel/linux-64
                          https://conda.anaconda.org/intel/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch

I rebuilt my conda environment with the above settings and modules, and then tried to run. Now I'm running into problems where it is just hanging up, and I can't figure out why.

Calling my submission script (basically the 02_control_opt example) with:
mpiexec -n $nC --bind-to core python weis_driver.py
results in it hanging up while doing
import openmdao.api as om
while importing run_weis from runWEIS.py.
I have tried this for openmdao 3.11.0 and 3.13.0 and gotten the same problems.

Perhaps even stranger, is that if I just call
python weis_driver.py (so no MPI),
It won't even print a print statement that I have on the first line in my weis_driver.py script.

So, hopefully this helps someplace. This feels like it might have just been backwards progress because now I'm getting errors even earlier in the calls to WEIS, but hopefully it helps sort out the higher-level issues of the linked libraries.

[dzalkind]

I'm not sure we should make this issue a show-stopper. I have it working if the executable is used. We can make that the default. What do people think?

Also, I'll be subtree-ing this branch of OpenFAST soon because it has better support for TMDs, which we are using in our benchmark case studies.

[johnjasa]

@dzalkind, I agree that it's not a show-stopper, I've been using the executable for my cases as well.

[dzalkind]

I documented the MKL issue in #153 and merged in WISDEM/develop. Once, this passes, I'm good to merge.