-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unstable simulations are not caught by buildkite, LES_driven_SCM actually breaking #574
Comments
I think for this you probably want some sort of state dump upon aborting (ex. simulation time, number of steps, run parameters). Then the CI can see if this abort state dump file exists for a particular run and raises an proper error exit code at the end. This will help with calibration as well because the error will have some context with it's state so you don't have to do the extra work of trying to figure out which parameters caused the abort to happen at which time. |
575: Add post-run tests, error if we do not run to t_max r=costachris a=charleskawczynski Closes #574. Co-authored-by: Charles Kawczynski <[email protected]>
Hmm, #575 adds a test on the ode integrator that we've reach |
581: Error on aborted simulations after solution export r=charleskawczynski a=charleskawczynski This PR moves the error on early aborted simulations to _after_ the solution is exported. This way we can more easily look at the solution / data to see what went wrong. [Discussed here](#574 (comment)). I'm hoping that this will shed light on #577. Co-authored-by: Charles Kawczynski <[email protected]>
581: Error on aborted simulations after solution export r=charleskawczynski a=charleskawczynski This PR moves the error on early aborted simulations to _after_ the solution is exported. This way we can more easily look at the solution / data to see what went wrong. [Discussed here](#574 (comment)). I'm hoping that this will shed light on #577. Co-authored-by: Charles Kawczynski <[email protected]>
I just realized that the MSE buildkite tests can pass without the simulation reaching the end time. This situation is frequently seen in the LES_driven_SCM test. Examples in closed PRs,
I attach an example of this behavior from the buildkite of PR #564. This issue has two components,
The text was updated successfully, but these errors were encountered: