Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading time from multiple forcing netCDF files (> 2Gb) very slow #238

Closed
verseve opened this issue Dec 7, 2022 · 1 comment · Fixed by #240
Closed

Reading time from multiple forcing netCDF files (> 2Gb) very slow #238

verseve opened this issue Dec 7, 2022 · 1 comment · Fixed by #240
Assignees
Labels
bug Something isn't working

Comments

@verseve
Copy link
Member

verseve commented Dec 7, 2022

Reading the time variable from multiple forcing netCDF files (e.g.: "inmaps-er_*.nc">2 Gb) is very slow. During model initialization it hangs here:

nctimes = reader.dataset["time"][:]

A possible solution is to read the time variable in a for loop once, and to store nctimes (as part of NCReader). I did a small test (for loop) and this gives a significant speedup. @ahweerts did report the issue. Bit strange that until now no one else ran into this issue (Wflow runs have been made at the global scale, including multiple forcing files). @visr : perhaps you have some additional info/insights on this?

@verseve verseve added the bug Something isn't working label Dec 7, 2022
@verseve verseve self-assigned this Dec 7, 2022
@verseve
Copy link
Member Author

verseve commented Dec 7, 2022

Not able to reproduce this issue with NCDatasets v0.12.3, with v0.12.4 Wflow initialization hangs, need to check differences between these two versions...

Reported the following issue: Alexander-Barth/NCDatasets.jl#194

@verseve verseve linked a pull request Dec 14, 2022 that will close this issue
@visr visr closed this as completed in #240 Dec 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant