Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce dependence on external data servers #625

Closed
juliasloan25 opened this issue May 22, 2024 · 4 comments
Closed

reduce dependence on external data servers #625

juliasloan25 opened this issue May 22, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@juliasloan25
Copy link
Member

juliasloan25 commented May 22, 2024

Describe the bug

We download some of our data directly from external servers every time we run a simulation. This causes problems, e.g. when those servers are down we can't download data and can't run our simulations. We can address this by fully switching to use ClimaArtifacts.

SNOTEL data bug description

Suddenly I get a docs failure on two separate branches that don't make any changes in docs/. It comes from docs/tutorials/standalone/Snow/data_tutorial.jl and seems to be due to a dimension mismatch when creating a dataframe.

ERROR: LoadError: LoadError: DimensionMismatch: Length of nms doesn't match length of x.
in expression starting at /home/runner/work/ClimaLand.jl/ClimaLand.jl/docs/src/generated/standalone/Snow/data_tutorial.md:2
when executing the following code block from inputfile `~/work/ClimaLand.jl/ClimaLand.jl/docs/tutorials/standalone/Snow/data_tutorial.jl`

metadata = DataTools.snotel_metadata();
metacols = ["id", "state", "elev", "lat", "lon"]
DataFrames.rename!(metadata, Symbol.(metacols));

Failing Github actions:

To Reproduce

Ideally a minimal reproducible example:

# some code here

Or even better, add it as a unit test, and open pull request.

Project

If not using the `examples` project: ``` paste your Project.toml here. ``` ``` paste your Manifest.toml here. ```

System details

Any relevant system information:

  • Julia version
  • operating system
  • modules loaded on cluster (module list)

Related issues / PRs

Please add any relevant links.

@juliasloan25 juliasloan25 added the bug Something isn't working label May 22, 2024
@Sbozzolo
Copy link
Member

There were probably server issues. I meged the PR.

I think that the real issue is that we should try to avoid depending on an external server if the data we use is always the same.

@juliasloan25 juliasloan25 changed the title docs for snow data tutorial fails reduce dependence on external data servers May 23, 2024
@juliasloan25
Copy link
Member Author

There were probably server issues. I meged the PR.

I think that the real issue is that we should try to avoid depending on an external server if the data we use is always the same.

Agreed, I updated this issue to highlight that

@juliasloan25
Copy link
Member Author

will be resolved as we move to ClimaArtifacts - see #580

@juliasloan25 juliasloan25 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 30, 2024
@Sbozzolo
Copy link
Member

My understanding is that part of the Snow code is supposed to grab and process whatever data the user from Snotel wants.

I also think that we had issues with the MODIS servers at some point. I am not sure where we access them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants