remove use of OBS and use CMIP instead in examples/recipe_ncl.yml

[valeriupredoi]

Description

As mentioned in #2301 it's good to have the basic examples recipes (Python, NCL, Julia, R) not use any OBS data since it can't be automatically downloaded (yes, it can be downloaded and cmorized by the user but we shouldn't expect brand new users who we target those recipes to to do it just yet). I checked the other examples and only the NCL one needed OBS data.

I am asking two NCL-ers to review this (even though I've not changed any NCL code) and Sarah since she may want to comment from the what's best for users type of perspective 🍺

• Closes Have NCL example recipe that doesn't use any OBS/OBS6 data #2301
• Link to documentation:

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Tests run successfully
• 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

• 🧪 Recipe runs successfully
• 🧪 Recipe is well documented
• 🧪 Figure(s) and data look as expected from literature
• 🛠 Provenance information has been added

[valeriupredoi]

@esmvalbot please run examples/recipe_ncl.yml

[esmvalbot]

Since @valeriupredoi asked, ESMValBot will run recipe examples/recipe_ncl.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe examples/recipe_ncl.yml: exit is 1, output has been generated here

[valeriupredoi]

@remi-kazeroni @bouweandela @zklaus you guys want to fix the bot not to install Julia outside the conda/mamba env please? Or I can do that too, up to you!

[SarahAlidoost]

@valeriupredoi thanks for tagging me here. I understood issue #2301. In my opinion, users might be confused about the purpose of having additional_datasets in this recipe as the dataset CanESM2 could be included in the datasets section.

So I suggest removing the section additional_datasets or adding another variable to the recipe because the documentation says "this is useful if specific datasets need to be used only by a specific diagnostic or variable. For example, this can be a good way to add observational datasets, which are usually variable-specific."

Hope it helps.

[valeriupredoi]

@SarahAlidoost very good point, cheers! 👍

[remi-kazeroni]

@esmvalbot please run examples/recipe_ncl.yml

[esmvalbot]

Since @remi-kazeroni asked, ESMValBot will run recipe examples/recipe_ncl.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe examples/recipe_ncl.yml: exit is 1, output has been generated here

[remi-kazeroni]

@esmvalbot please run examples/recipe_ncl.yml

[esmvalbot]

Since @remi-kazeroni asked, ESMValBot will run recipe examples/recipe_ncl.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe examples/recipe_ncl.yml: exit is 1, output has been generated here

[remi-kazeroni]

@esmvalbot please run examples/recipe_ncl.yml

[esmvalbot]

Since @remi-kazeroni asked, ESMValBot will run recipe examples/recipe_ncl.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe examples/recipe_ncl.yml: exit is 1, output has been generated here

[remi-kazeroni]

The Julia installation is failing but I'm not sure what is going wrong with the bot. The bot repository would need a bit of cleanup regarding the handling of Julia but I don't see anything really odd there.

Note that the VM where the bot runs has been upgraded recently and Bouwe has made changes to the bot accordingly. On the (rather old) test branch, the bot seems to run fine or at least the Julia installation is successful (see #1882). Could it be that you created this branch at a time we had a problem with Julia? But even then, the tests are all green. Will have another look tomorrow...

[valeriupredoi]

@remi-kazeroni not a worry, mate, I need to change the recipe again anyways, to include the (very useful) advice I got from @SarahAlidoost ie not have any reference datasets - just a plain old Joe recipe with a couple datasets in datasets and that's it - can any of the NCL literate people help here please - you or @schlunma or @axel-lauer bitte schoen - I don't speak that archaic language 😁

[bouweandela]

@remi-kazeroni My Julia problems are usually solved by deleting ~/.julia. I suspect this happens because Julia compiles some of its packages against the Python version from your conda environment. If you create a new conda environment with a newer Python version, the compiled Julia packages in the ~/.julia directory are broken (see #2060).

[remi-kazeroni]

@esmvalbot please run examples/recipe_ncl.yml

[esmvalbot]

Since @remi-kazeroni asked, ESMValBot will run recipe examples/recipe_ncl.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe examples/recipe_ncl.yml: exit is 1, output has been generated here

[bouweandela]

@esmvalbot please run examples/recipe_ncl.yml

[esmvalbot]

Since @bouweandela asked, ESMValBot will run recipe examples/recipe_ncl.yml as soon as possible, output will be generated here

[zklaus]

@remi-kazeroni My Julia problems are usually solved by deleting ~/.julia. I suspect this happens because Julia compiles some of its packages against the Python version from your conda environment. If you create a new conda environment with a newer Python version, the compiled Julia packages in the ~/.julia directory are broken (see #2060).

That used to be very effective, but I am not sure it is true anymore. There was a change in the conda-forge Julia package in which this directory was practically moved into the conda-forge environment. In principal a good thing, but we'll have to check if our scripts still work correctly with that.

[bouweandela]

I have just disabled the Julia installation in the bot installation script for now.

[remi-kazeroni]

Regarding the bot tests: the last esmvaltool run examples/recipe_ncl.yml never finished...

I also don't think the issue comes from the .julia because I deleted it from the VM before starting this run and still got the same error message. The bot installation of Julia seems to work fine on PRs using old branches like #1882 so it's probably due to some later changes in the handling of the Julia installation.
Disabling the Julia installation is a good compromise until we find a clean solution. Maybe we could continue in a PR dedicated for bot testing in #2511 and stop high-jacking @valeriupredoi's PR 😆

[valeriupredoi]

be my guest and keep hijacking this one Remi, the only ransom I'd ask for is to fix the NCL example diag to remove the use of additional_datasets as Sarah was suggesting - I am illiterate in NCL 😁

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe examples/recipe_ncl.yml: exit is 1, output has been generated here

[remi-kazeroni]

On top of the Julia installation issue, we faced a known issue that NCL diagnostics may hang forever as described in #2059. So I killed the last run after 3h30...

[valeriupredoi]

I needs halp here NCL peeps, pls 👀 🍺

[axel-lauer]

Just tested this and the output looks as expected.

[valeriupredoi]

well, we made a donkey out of a horse here guys, I wanted to implement the changes @SarahAlidoost suggested ie remove additional_datasets as well as the OBS datasets, but I didn't know how to edit the NCL script to not complain about the lack of additional_datasets - anyways, my objective is now met (no OBS requirements in an example recipe) but the NCL recipe is now a bit wonky from a user's perspective but heyho, we'll change it again down the line if we get confused users 🤷‍♂️