Singular exception

[haakon-e]

When running simulations with a little bit of stochastic noise on the stoch_lognormal_closure branch, I came across the following exception:

I get a singular exception in the tridiagonal solver in the update_covariance_ED function in Turbulence_PrognosticTKE.jl file.

ERROR: LoadError: On worker 22:
SingularException(30)
Stacktrace:
  [1] checknonsingular
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/factorization.jl:19 [inlined]
  [2] #lu!#138
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/lu.jl:551
  [3] #lu#136
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/lu.jl:273
  [4] lu (repeats 2 times)
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/lu.jl:272 [inlined]
  [5] \
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:1136
  [6] tridiag_solve
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/src/turbulence_functions.jl:266
  [7] update_covariance_ED
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/src/Turbulence_PrognosticTKE.jl:2402
  [8] compute_covariance
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/src/Turbulence_PrognosticTKE.jl:1828
  [9] update
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/src/Turbulence_PrognosticTKE.jl:294
 [10] run
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/integration_tests/utils/main.jl:55
 [11] #main1d#833
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/integration_tests/utils/main.jl:110
 [12] main1d
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/integration_tests/utils/main.jl:105 [inlined]
 [13] #main#832
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/integration_tests/utils/main.jl:101 [inlined]
 [14] main
    @ /groups/esm/hervik/calibration/TurbulenceConvection.jl/integration_tests/utils/main.jl:101
 [15] run_SCM_handler
    @ /central/groups/esm/hervik/calibration/EnsembleKalmanProcesses.jl/examples/SCAMPy/src/helper_funcs.jl:220
 [16] run_SCM
    @ /central/groups/esm/hervik/calibration/EnsembleKalmanProcesses.jl/examples/SCAMPy/src/helper_funcs.jl:149
 [17] g_
    @ /central/groups/esm/hervik/calibration/EnsembleKalmanProcesses.jl/examples/SCAMPy/experiments/scm_pycles_pipeline/calibrate.jl:114

Since it's stochastic, I'm a bit unsure how I'll confidently reproduce the exact model configuration, but would be happy to investigate further.

[haakon-e]

related to #121. closing this in favor of investigating the other issue.