Fix parsing in diag validation

[FranziskaWinterstein]

Description

When using the validation.py diagnostic I ran into problems when using aliases and the parsing of the variable ancestor_2 in function _get_provenance_record.
The code changes make use of the alias of the datasets and now finds the proper files if the alias is set manually.

---

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

• 🧪 Recipe runs successfully
• 🧪 Recipe is well documented
• 🧪 Figure(s) and data look as expected from literature
• 🛠 Provenance information has been added

---

[schlunma]

Thanks @FranziskaWinterstein for this PR! I have a couple of small suggetions for the code to simplify it.

Before I test existing recipes: can you describe quickly what is going wrong with apply_supermeans? At first glance it looks like your function is doing exactly the same, so I wonder what the problem is.

[FranziskaWinterstein]

Thank you for taking the time and your code updates are just perfect.

Regarding the apply_supermeans. I think the problem is, that the preprocessor function climate_statistics (in preprocessor/_time.py) somehow changed its output. Because when the cubes ctrl and exper are later given to plot_ctrl_exper and plot_zonal_cubes I receive an error. I attached the log file of a run, where this happens:
log.txt

[schlunma]

You are right, there have been some changes in climate_statistics recently, which lead to a couple of problems (e.g., ESMValGroup/ESMValCore#2223). However, this should be fixed by a PR from 10 Oct (ESMValGroup/ESMValCore#2191). Could you please update your version of ESMValCore and try again please? Since your log talks about the coordinate time_weights I guess you are using an older version, in the new the coordinate is called _time_weights_.

[FranziskaWinterstein]

Okay, I updated the Core, but now the preprocessor fails completely (main_log_debug.txt) That is however as far as I can go, today. I will look into it again next week, but maybe you have a suggestion what the problem could be already.

[schlunma]

Yes I do! The error is caused by a new feature of iris. Unfortuntately you have to reinstall your environment to solve this properly (i.e., get the latest iris version). Sorry, there have been quite some changes recently.

[FranziskaWinterstein]

@schlunma Yes, you were right. I needed to reinstall conda anyway. With the current ESMValCore release as well as the current development ESMValCore everything went fine with the supermeans function. So I will revoke the changes regarding supermeans.

[FranziskaWinterstein]

Ready to merge, I think.

[schlunma]

I just tested this with the existing recipes recipe_validation.yml and recipe_validation_CMIP6.yml. Unfortunately, both failed with the following error:

Traceback (most recent call last):
  File "/home/b/b309141/repos/ESMValTool/esmvaltool/diag_scripts/validation.py", line 418, in <module>
    main(config)
  File "/home/b/b309141/repos/ESMValTool/esmvaltool/diag_scripts/validation.py", line 383, in main
    plot_ctrl_exper_seasons(ctrl_seasons, exper_seasons, cfg, plot_key)
  File "/home/b/b309141/repos/ESMValTool/esmvaltool/diag_scripts/validation.py", line 353, in plot_ctrl_exper_seasons
    plot_zonal_cubes(control_season, experiment_season, cfg, plot_info)
  File "/home/b/b309141/repos/ESMValTool/esmvaltool/diag_scripts/validation.py", line 223, in plot_zonal_cubes
    save_plotted_cubes(
  File "/home/b/b309141/repos/ESMValTool/esmvaltool/diag_scripts/validation.py", line 103, in save_plotted_cubes
    _get_provenance_record(cfg, save_path,
  File "/home/b/b309141/repos/ESMValTool/esmvaltool/diag_scripts/validation.py", line 43, in _get_provenance_record
    ancestor_1 = [
                 ^
IndexError: list index out of range

The reason for this is that the code that retrieves the ancestors is really not optimal. It uses lots of split('_'), which fails if aliases look different (they may or may not contain _). I tried to fix this in 90fc8dc, but this is still not really nice. This diagnostic would need a lot of changes to make it more robust (for example, explicitly pass aliases and ancestors to the functions instead of relying on weird strings that somehow include this information).

@FranziskaWinterstein Please let me know if this also works for your use case; our two validation recipes run successfully and produce consistent resuls 👍

[FranziskaWinterstein]

@schlunma Yes, my recipe output is identical.

I was also confused that the names are retrieved from the output filenames, which fails easily, if something changes there.

[schlunma]

Great, then I think this can be merged!

@valeriupredoi, as the author of this diagnostic, could you please do a final check and merge? Thanks!!

[valeriupredoi]

looking good, many thanks! 🍺

[valeriupredoi]

guys, bear in mind this diagnostic predates most all the bits that you mention, it's ye olde, but still going strong if you don't need to do fancy stuff 😁

@esmvalbot please run recipe_validation.yml

[valeriupredoi]

@esmvalbot please run recipe_validation.yml

[esmvalbot]

Since @valeriupredoi asked, ESMValBot will run recipe recipe_validation.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is happy to report recipe recipe_validation.yml ran OK, output has been generated here