Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Omniscape fails on specific row / column: "median of empty array is undefined" #122

Open
slamander opened this issue Sep 29, 2022 · 24 comments

Comments

@slamander
Copy link

Cross-posting from gitter: there have two mentions of Omniscape failing on a specific row / column.

In my case, I receive this error: ERROR: ArgumentError: median of an empty array is undefined, Float64[] . The run will fail at different stages--sometimes quite far into a run, other times immediately--but always at the same row-column combination. I've tried to diagnose the error by changing the raster inputs or adjusting model parameters (e.g., solver, block size, r_cutoff [for source = resistance], and running conditional and non-conditional criteria).

Here's my error print-out. Let me know if additional information is needed.

Progress:   9%|█████                                             |  ETA: 2:57:07
Omniscape failed on the moving window centered on row 325 column 5198
ERROR: TaskFailedException
Stacktrace:
 [1] wait
   @ .\task.jl:322 [inlined]
 [2] threading_run(func::Function)
   @ Base.Threads .\threadingconstructs.jl:34
 [3] macro expansion
   @ .\threadingconstructs.jl:93 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [6] top-level scope
   @ REPL[5]:1

    nested task error: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] macro expansion
       @ C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [3] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48

    caused by: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] median!(v::Vector{Float64})
       @ Statistics C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Statistics\src\Statistics.jl:791
     [2] median
       @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Statistics\src\Statistics.jl:836 [inlined]
     [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:231
     [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:172
     [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:308
     [6] macro expansion
       @ C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
     [7] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [8] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48
@vlandau
Copy link
Member

vlandau commented Nov 10, 2022

Sorry for the delay in getting back to you on this. Is this still giving you problems?

It does make sense that the timing of the error being thrown is random since the order of the moving window solves is randomized for more accurate ETA estimates in the progress bar.

If possible, would you mind sharing your input files (and the associate ini file)? I'll try to take a look in the coming couple of weeks.

@slamander
Copy link
Author

slamander commented Mar 31, 2023

@vlandau, and my apologies for taking even longer to get back. Yes, this is still giving me issues, across several different input files. Here's the input data and ini file: https://drive.google.com/drive/folders/1kC4FVzw2HjlXVbLLot4a6LU0EPu-E-7s?usp=sharing.

@slamander
Copy link
Author

slamander commented Mar 31, 2023

By the way, here's the problem cells from two failed runs. The value of all of these cells are NA.
And here's the stack trace from one of them: trouble

Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
 [6] top-level scope
   @ /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_All.jl:7

    nested task error: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [3] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
    
    caused by: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] median!(v::Vector{Float64})
       @ Statistics /apps/julia/1.8.2/share/julia/stdlib/v1.8/Statistics/src/Statistics.jl:810
     [2] median
       @ /apps/julia/1.8.2/share/julia/stdlib/v1.8/Statistics/src/Statistics.jl:855 [inlined]
     [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:231
     [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:172
     [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:308
     [6] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
     [7] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [8] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [9] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_All.jl:7
Omniscape failed on the moving window centered on row 145 column 2049
Fri Mar 31 10:46:47 EDT 2023

@Daviser95
Copy link

Hi guys, I'm having the exact same error launching a Europe-wide Omniscape scan. The error appeared after adding a conditional layer in the .ini file. I've already checked that the conditional layer has the same size and resolution as the resistance layer. I tried to change some parameters (radius and block size) but the error keeps appearing.

Omniscape failed on the moving window centered on row 3323 column 2228
ERROR: TaskFailedException

nested task error: ArgumentError: median of an empty array is undefined, Float64[]
Stacktrace:
 [1] macro expansion
   @ C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
 [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
   @ Omniscape .\threadingconstructs.jl:163
 [3] #161#threadsfor_fun
   @ .\threadingconstructs.jl:130 [inlined]
 [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
   @ Base.Threads .\threadingconstructs.jl:108

caused by: ArgumentError: median of an empty array is undefined, Float64[]
Stacktrace:
 [1] median!(v::Vector{Float64})
   @ Statistics C:\Users\utente\AppData\Local\Programs\Julia-1.9.0\share\julia\stdlib\v1.9\Statistics\src\Statistics.jl:803
 [2] median
   @ C:\Users\utente\AppData\Local\Programs\Julia-1.9.0\share\julia\stdlib\v1.9\Statistics\src\Statistics.jl:848 [inlined]
 [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:219
 [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:172
 [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:308
 [6] macro expansion
   @ C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
 [7] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
   @ Omniscape .\threadingconstructs.jl:163
 [8] #161#threadsfor_fun
   @ .\threadingconstructs.jl:130 [inlined]
 [9] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
   @ Base.Threads .\threadingconstructs.jl:108

Stacktrace:
[1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
@ Base.Threads .\threadingconstructs.jl:120
[2] macro expansion
@ .\threadingconstructs.jl:168 [inlined]
[3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
@ Omniscape C:\Users\utente.julia\packages\Omniscape\9gHf2\src\main.jl:257
[4] run_omniscape(path::String)
@ Omniscape C:\Users\utente.julia\packages\Omniscape\9gHf2\src\main.jl:536
[5] top-level scope
@ REPL[6]:1

@vlandau
Copy link
Member

vlandau commented Jun 1, 2023

My best guess is at the column/row of one of your inputs, the resulting slice/chunks of the array is just empty (all NaN values). Would you be able to clip out a section (square, with side lengths equal to 2×radius + 1) of each of your input rasters centered on the failing row/column and share it with me?

@Daviser95
Copy link

Hi @vlandau and thanks for replying, you're probably right. I share the clipped input rasters with your options. However, if this is the problem, I don't know why using only the resistance raster doesn't cause problems, while this error only occurs when using a second layer as a condition.
Moreover, I noticed that the resolution of the two layers is not the same, while the coordinate system and the dimensions are, but I don't know if this difference can cause the error.

Immagine 2023-06-03 030331
Immagine 2023-06-03 030516

@ashton314
Copy link

Hi there! I'm a PhD student and I'm building a tool to help developers automatically track down where NaNs come from in computations, and I'd like to try it out on the problem you're having here to see if I can help. Would someone mind showing me how to fire it all off to reproduce the problems that you're having here? I think I have the .ini and the .tiff files already—just wondering how to set it up properly.

@Daviser95
Copy link

Hi @ashton314! If you already have .ini and .tiff file, I think you just need to donwload Omniscape on Julia and then run the .ini file. You can get information on this page https://docs.circuitscape.org/Omniscape.jl/stable/usage/#Running-Omniscape

@slamander
Copy link
Author

@vlandau I also thought there could be NaNs on the failing pixels, but after investigating the failures I found no NaNs. Does it make sense that identical jobs fail at different locations/durations? Sometimes the job will near completion (>90%), but others will fail early--all failing on different pixels. Unlike @Daviser95, some jobs will fail whether or not I have conditional criteria (it's not very predictable). Here's an output from the last job I ran without conditional criteria:

Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
 [6] top-level scope
   @ /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_Americas.jl:7

    nested task error: 
Progress:  56%|████████████████████████████�                     |  ETA: 6:19:09�[K
Progress:  56%|████████████████████████████�                     |  ETA: 6:19:09�[KAssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [3] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
    
    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SuiteSparse.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/core.jl:621
      [2] macro expansion
        @ ./timing.jl:382 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/raster/advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/raster/advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:332
      [7] macro expansion
        @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape ./threadingconstructs.jl:84
      [9] #161#threadsfor_fun
        @ ./threadingconstructs.jl:51 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_Americas.jl:7

Progress:  56%|████████████████████████████�                     |  ETA: 6:19:08�[KOmniscape failed on the moving window centered on row 2542 column 7157
Omniscape failed on the moving window centered on row 1424 column 1983
Omniscape failed on the moving window centered on row 1593 column 4102
Wed Jul 12 21:26:43 EDT 2023

@vlandau
Copy link
Member

vlandau commented Jul 20, 2023

It does make sense that it would occur at different times because the order in which the windows are solved is random (this helps the ETA on the progress bar to be more accurate). I assume in your case there are multiple moving window locations that will cause failure, but which one you run into first is random due to the random solve order.

@geomenke
Copy link

geomenke commented Feb 2, 2024

I regularly get the same issue as @slamander and @Daviser95. Omniscape will fail on specific row / columns. These are not No Data or Nan pixels however. I have clipped out a small area centered on these pixels and run OS on the subset and it finishes successfully. What are the potential reasons for this type of error?
Here is the stacktrace of the error.

moving window centered on row 6461 column 8703
Progress:  92%|███████████████████████████████████████████████   |  ETA: 4:55:16ERROR: TaskFailedException

    nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:194
     [3] #161#threadsfor_fun
       @ .\threadingconstructs.jl:161 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads .\threadingconstructs.jl:139

    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SparseArrays.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\core.jl:621
      [2] macro expansion
        @ .\timing.jl:393 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\utils.jl:332
      [7] macro expansion
        @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape .\threadingconstructs.jl:194
      [9] #161#threadsfor_fun
        @ .\threadingconstructs.jl:161 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads .\threadingconstructs.jl:139
Stacktrace:
 [1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads .\threadingconstructs.jl:151
 [2] macro expansion
   @ .\threadingconstructs.jl:199 [inlined]
 [3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [4] run_omniscape(path::String)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [5] top-level scope
   @ REPL[3]:1

@vlandau
Copy link
Member

vlandau commented Feb 2, 2024

@geomenke this is a different error than the empty array error mentioned above (but I see now one of the stack traces had an AssertionError too), coming directly from Circuitscape, but fortunately, we just made a patch release to Circuitscape that may fix that issue. Would you update to Circuitscape 5.13.2 (you'll need Julia 1.9) and try again?

@geomenke
Copy link

geomenke commented Feb 2, 2024

@vlandau I am already at Julia 1.9.2 so what is the best way to update to Circuitscape 5.13.2? Do I upgrade Omniscape? Can that be done via the command line?

@vlandau
Copy link
Member

vlandau commented Feb 2, 2024

Oh great! To update, in a Julia terminal, enter the package interface by typing the ] key, then run update Circuitscape and you should see 5.13.2 get installed.

@geomenke
Copy link

geomenke commented Feb 2, 2024

It did not specifically list Circuitscape as being updated, just things like updating registry etc.
How can I check which Circuitscape version I have?

@vlandau
Copy link
Member

vlandau commented Feb 2, 2024

In the same package prompt/interface you should be able to run status Circuitscape

@geomenke
Copy link

geomenke commented Feb 2, 2024

I see that I have an outdated and restricted by compatibility contraints version of Omniscape and that Circuitscape did not update after running update. Running update Omniscape did not update Omniscape. Do I need to run using Pkg; Pkg.add(PackageSpec(name = "Omniscape", rev = "main))
image

@vlandau
Copy link
Member

vlandau commented Feb 2, 2024

You shouldn't need to do that. There are some solutions in this thread that might work for you: https://discourse.julialang.org/t/how-to-force-an-update/73071/12

@geomenke
Copy link

geomenke commented Feb 7, 2024

I updated to Circuitscape v5.13.2 and Omniscape v0.6.1, but it failed again with a similar stacktrace as before: Any recommendations?

Progress:  81%|█████████████████████████████████████████         |  ETA: 17:26:09Omniscape failed on the moving window centered on row 11263 column 10839
Progress:  97%|█████████████████████████████████████████████████ |  ETA: 2:58:35ERROR: TaskFailedException

    nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:194
     [3] #161#threadsfor_fun
       @ .\threadingconstructs.jl:161 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads .\threadingconstructs.jl:139

    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SparseArrays.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\core.jl:621
      [2] macro expansion
        @ .\timing.jl:393 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\utils.jl:332
      [7] macro expansion
        @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape .\threadingconstructs.jl:194
      [9] #161#threadsfor_fun
        @ .\threadingconstructs.jl:161 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads .\threadingconstructs.jl:139
Stacktrace:
 [1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads .\threadingconstructs.jl:151
 [2] macro expansion
   @ .\threadingconstructs.jl:199 [inlined]
 [3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [4] run_omniscape(path::String)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [5] top-level scope
   @ REPL[14]:1

@ptfreeman-csp
Copy link

@slamander or @vlandau - we are also running into this issue using conditional files to incorporate climate connectivity into our models. We've exported all of our input layers at the same resolution and CRS but we're running into the same error with the latest version of Omniscape. Was this issue ever resolved?

@vlandau
Copy link
Member

vlandau commented Aug 27, 2024

@ptfreeman-csp hmm, no I'm not sure why this is happening, but I guess either something is wrong with the input, or Omniscape's processing of the input. Do you have an exact column and row that it failed on? Taking a look around that area/coordinate of all of the inputs might shed some light about what could be going on.

@ptfreeman-csp
Copy link

ptfreeman-csp commented Aug 27, 2024 via email

@ptfreeman-csp
Copy link

ptfreeman-csp commented Aug 27, 2024 via email

@vlandau
Copy link
Member

vlandau commented Aug 27, 2024

Oh that second point is very strange. Glad it's working now though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants