Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example 2 fails with RotorSE: True #240

Closed
mlidGWT opened this issue Oct 20, 2023 · 15 comments
Closed

Example 2 fails with RotorSE: True #240

mlidGWT opened this issue Oct 20, 2023 · 15 comments

Comments

@mlidGWT
Copy link

mlidGWT commented Oct 20, 2023

Description

Hi again,

I am trying to run WEIS in the format of example 02, where an openfast_file and openfast_dir are specified with the OpenFAST input files. I am able to both run example 02 and my own simulations with this format, until enabling RotorSE, at which point I get the errors below in the Current Behavior section (the errors are different for example 02 than for my own runs).

In my own directory, I am using the IEA 15MW RWT model, though it seems to be a slightly different 'vintage', with 60 blade stations and other numerical differences vs. those in example 02.

I am not trying to run an optimization, simply to use WEIS's DLC-generation ability paired with more control over the OpenFAST input files, so if my error is around some attempted optimization task, please advise how to turn that off. I have set driver: optimization: flag: False and recorder: flag: False in the modeling_options.yaml and am still getting the below error.

If enabling RotorSE means something in the WEIS files needs to be configured differently, it would be great if documentation around those changes could be added - perhaps to the README file in that example's folder - as they are not self-evident.

Thanks very much for your help - this issue is keeping me from making progress in my day job!
Best,
Meg

Steps to reproduce issue

In example 02:

  1. Set RotorSE: flag: True in `modeling_options.yaml'
  2. Run WEIS using python weis_driver.py

Current behavior

Example 02 error

Using weis.aeroelasticse in ROSCO_toolbox...
Loading wind turbine data for NREL's ROSCO tuning and simulation processeses
Loading FAST model: IEA-15-240-RWT-UMaineSemi.fst
Loading rotor performace data from text file: /Users/meglidrbauch/Software/WEIS/examples/01_aeroelasticse/OpenFAST_models/IEA-15-240-RWT/IEA-15-240-RWT/Cp_Ct_Cq.IEA15MW.txt
Traceback (most recent call last):
File "/Users/meglidrbauch/Software/WEIS/examples/02_control_opt/weis_driver.py", line 15, in
wt_opt, modeling_options, opt_options = run_weis(fname_wt_input, fname_modeling_options, fname_analysis_options)
File "/Users/meglidrbauch/Software/WEIS/weis/glue_code/runWEIS.py", line 154, in run_weis
wt_opt = yaml2openmdao(wt_opt, modeling_options, wt_init, opt_options)
File "/Users/meglidrbauch/Software/WEIS/WISDEM/wisdem/glue_code/gc_WT_InitModel.py", line 51, in yaml2openmdao
hub = wt_init["components"]["hub"]
KeyError: 'hub'

My directory error

Note: this error comes after updating the n_pitch_perf_surfaces parameter in modeling_options.yaml to 104 to match the pitch vector dimension in the Cp_Ct_Cq.txt file, and the n_tsr_perf_surfaces parameter to 72 for the same reason. I am unable to follow the proverbial thread in the error message to determine where the max_allowable_td_ratio should be found.

Using weis.aeroelasticse in ROSCO_toolbox...
Loading wind turbine data for NREL's ROSCO tuning and simulation processeses
Loading FAST model: IEA-15-240-RWT-GWTBaseline.fst
Loading rotor performace data from text file: /Users/meglidrbauch/Software/LoadsRuns/ModelLibrary/IEA15_GWTBaseline/IEA15_GWTB_Cp_Ct_Cq.txt
Traceback (most recent call last):
File "/Users/meglidrbauch/Software/LoadsRuns/SimulationLibrary/RunDir/weis_driver.py", line 15, in
wt_opt, modeling_options, opt_options = run_weis(fname_wt_input, fname_modeling_options, fname_analysis_options)
File "/Users/meglidrbauch/Software/WEIS/weis/glue_code/runWEIS.py", line 159, in run_weis
wt_opt = myopt.set_initial(wt_opt, wt_init)
File "/Users/meglidrbauch/Software/WEIS/WISDEM/wisdem/glue_code/gc_PoseOptimization.py", line 1685, in set_initial
wt_opt["tcons.max_allowable_td_ratio"] = blade_constr["tip_deflection"]["margin"]
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/problem.py", line 513, in setitem
self.set_val(name, value)
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/problem.py", line 544, in set_val
self.model.set_val(name, val, units=units, indices=indices)
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/system.py", line 5368, in set_val
raise KeyError(f'{model.msginfo}: Variable "{name}" not found.')
KeyError: ' : Variable "tcons.max_allowable_td_ratio" not found.'

Expected behavior

Simulations run as expected and I am able to run DLCs through WEIS with OpenFAST input files defined.

Code versions

  • Python 3.10
  • WEIS develop
@dzalkind
Copy link
Collaborator

Hi,

These 02 examples are designed to be run without most parts of WISDEM. They will not update an OpenFAST model except to run different load cases. Enabling RotorSE will result in unexpected behavior.

We are planning to release a series of software and documentation improvements over the next year with new ARPA-E funding, and we would be happy to discuss specific solutions for you and your team.

Best, Dan

@dzalkind
Copy link
Collaborator

Let's leave this issue open until we can throw an appropriate error for the situation posed.

@mlidGWT
Copy link
Author

mlidGWT commented Oct 23, 2023

Hi @dzalkind, thanks for the explanation - I didn't understand the relationship between the modules and was going off of the settings for previous WEIS runs.

I'm now encountering an issue where WEIS seems to run the first Case without any issue, but then does not proceed to the next one and seems to hang after printing the runtime for the completed Case, almost like it is expecting user input, but without any prompts about what to input. Do you know what might be causing this?

Thank you!
Meg

@dzalkind
Copy link
Collaborator

Hi Meg,

Can you please share the output log? It may also just be that WEIS finished its runs. There should be two case_matrix files (one .txt and one .yaml) where you run the OpenFAST simulations. There, you can confirm which cases have been set up.

Dan

@mlidGWT
Copy link
Author

mlidGWT commented Oct 23, 2023

Hi Dan,

I don't seem to have an output log that indicates what is happening in the backend, but the case matrix looks like all the cases are properly defined:
case_matrix.txt

Please advise how to enable the output log you are referring to - I don't see one in the WEIS yamls or the .fst file. (verbosity in the modeling_options seems to be for printing to screen, Echo in the .fst file seems to echo the inputs, and I take SumPrint to be a summary of the results, not the simulation state)

Meg

@dzalkind
Copy link
Collaborator

Hi Meg,

By the output log, I mean the terminal messages from the WEIS run. You can python <weis_driver.py> > output.log to save it to a file.

Dan

@mlidGWT
Copy link
Author

mlidGWT commented Oct 23, 2023

I'm not having success with that command. I've tried some variations, but as written I get:

zsh: parse error near `>'

If I try python weis_driver.py > output.log:

RuntimeWarning: /Users/meglidrbauch/Software/WEIS/weis/control/dac.py:734
invalid value encountered in scalar divide

which then hangs and I have to interrupt, leading to a long Traceback message.

@dzalkind
Copy link
Collaborator

Instead of writing to your terminal, it's writing to output.log. Is there anything in that file before you interrupted it?

@mlidGWT
Copy link
Author

mlidGWT commented Oct 23, 2023

Ah, I was expecting the file to be in the temp/ folder, but it was at the cd level. Now attached:

output.log

It was hanging after

OpenFAST terminated normally.

and required two CTRL+C interrupts before adding the final 4 lines of the file.

@dzalkind
Copy link
Collaborator

Are you still running with the same modeling options that we defined here?

Can you let it go for a while and try to figure out where it's hanging? Is there an error message that appears when you CTRL+C?

@mlidGWT
Copy link
Author

mlidGWT commented Oct 23, 2023

I am still running it with the OpenFAST executable, I now have verbosity: True, but I do not have it saving timeseries or iterations. Currently all of the *SE options in the WISDEM section are set to False, as well as BOS.

I have had it running since 5 ET, and so far the log only contains up until

OpenFAST terminated normally.

(I have refreshed it a couple times). I can try a single keyboard interrupt to see what else comes in, since it seems to take 2 interrupts to fully kill the run.

@mlidGWT
Copy link
Author

mlidGWT commented Oct 23, 2023

This time a single interrupt killed it, at which point the final 4 lines of the log file were printed, as in the version I shared.

The terminal error message after the interrupt:

RuntimeWarning: /Users/meglidrbauch/Software/WEIS/weis/control/dac.py:734
invalid value encountered in scalar divide^CTraceback (most recent call last):
File "/Users/meglidrbauch/Software/LoadsRuns/SimulationLibrary/6011_SBIR/weis_driver.py", line 15, in
wt_opt, modeling_options, opt_options = run_weis(fname_wt_input, fname_modeling_options, fname_analysis_options)
File "/Users/meglidrbauch/Software/WEIS/weis/glue_code/runWEIS.py", line 199, in run_weis
wt_opt.run_model()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/problem.py", line 629, in run_model
self.model.run_solve_nonlinear()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/system.py", line 4684, in run_solve_nonlinear
self._solve_nonlinear()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/group.py", line 2532, in _solve_nonlinear
self._nonlinear_solver._solve_with_cache_check()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/solvers/nonlinear/nonlinear_runonce.py", line 26, in _solve_with_cache_check
self.solve() # don't use caching
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/solvers/nonlinear/nonlinear_runonce.py", line 45, in solve
self._gs_iter()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/solvers/solver.py", line 800, in _gs_iter
subsys._solve_nonlinear()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/explicitcomponent.py", line 312, in _solve_nonlinear
self._compute_wrapper()
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/openmdao/core/explicitcomponent.py", line 283, in _compute_wrapper
self.compute(self._inputs, self._outputs,
File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/openmdao_openfast.py", line 611, in compute
summary_stats, extreme_table, DELs, Damage, case_list, case_name, chan_time, dlc_generator = self.run_FAST(inputs, discrete_inputs, fst_vt)
File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/openmdao_openfast.py", line 2107, in run_FAST
summary_stats, extreme_table, DELs, Damage, chan_time = fastBatch.run_serial()
File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/runFAST_pywrapper.py", line 361, in run_serial
_name, _ss, _et, _dl, _dam, _ct = evaluate(c)
File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/runFAST_pywrapper.py", line 480, in evaluate
return fast.execute()
File "/Users/meglidrbauch/Software/WEIS/weis/aeroelasticse/runFAST_pywrapper.py", line 265, in execute
case_name, sum_stats, extremes, dels, damage = self.la._process_output(output,
File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/analysis.py", line 178, in _process_output
extremes = output.extremes(output.channels)
File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/io/openfast.py", line 132, in extremes
idx_max = self.idxmaxs[i]
File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/io/openfast.py", line 19, in wrapper
return f(self, *args, **kwargs)
File "/Users/meglidrbauch/Software/WEIS/pCrunch/pCrunch/io/openfast.py", line 167, in idxmaxs
return np.argmax(self.data, axis=0)
File "<array_function internals>", line 200, in argmax
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 1242, in argmax
return _wrapfunc(a, 'argmax', axis=axis, out=out, **kwds)
File "/Users/meglidrbauch/miniconda3/envs/weis-env/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
KeyboardInterrupt

@mlidGWT
Copy link
Author

mlidGWT commented Oct 24, 2023

I let it run one more time before logging off (~5:30 PM ET) and just checked back on it (8:40 PM ET). It seems it wasn't hung, just taking an incredibly long time in between Cases. The first Case finished running in OpenFAST at 5:37 (~4 minute runtime on a 320s simulation), and OpenFAST was not called again until 6:18 PM for Case 2, finishing up about 4 minutes later as well. The 3rd Case started at 7:03 PM, the 4th at 7:48 PM, and the 5th at 8:30, which has completed but the 6th not yet begun. I will share the updated log file tomorrow once it is completed, but I don't see any messages in it that indicate why the delay between cases is so long. As far as I'm aware, all optimizations should be off and it should simply be reading and running the specified OpenFAST input files, though it's possible I've missed something somewhere.

@ptrbortolotti
Copy link
Collaborator

Hello Meg, thank you, that helps the debugging... Can you please check to make sure that your output files have a reasonable size (10-100 MB)? Is the number of output channel "ordinary" (100 or so), is your DT_Out in OpenFAST normal (100 Hz or so)? The alternative is that something went wrong with the compiling of pCrunch, but that would be a first. Also, is there a way for us to reproduce the issue locally? If not, feel free to send me an invite and we can do a screenshare
Pietro

@mlidGWT
Copy link
Author

mlidGWT commented Oct 24, 2023

Hi Pietro and Dan,

Thank you both for your patience and support, I'm a bit embarrassed to say that everything seems to be working, I just overscoped the outputs for what my computer is capable of. I had nodal outputs turned on for both AeroDyn and ElastoDyn, for all 3 blades with more than 20 nodes in each case. Each output file was 1.13GB! I'm trying again with some different settings. I didn't realize how much of an impact the nodal settings would have on run time, but it makes sense in hindsight. I might suggest these would be good warnings/notes to include in documentation around nodal outputs. :)

Thank you again for your support, and apologies if I wasted your time!
Meg

@mlidGWT mlidGWT closed this as completed Oct 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants