[PR]: CDAT Migration: Refactor `aerosol_aeronet` set #788

tomvothecoder · 2024-02-14T23:12:43Z

Description

Closes CDAT Migration Phase 2: Refactor aerosol_aeronet set #672

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

If applicable:

New and existing unit tests pass with my changes (locally and CI/CD build)
I have added tests that prove my fix is effective or that my feature works
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

- Add results png for main and dev branches

tomvothecoder

Hi @chengzhuzhang, this PR is ready for review.

This set involved minimal work and will probably have been the easiest to refactor. The main branch doesn't save the .nc or .json files, so we have to do an eye-check for the .png plot files (included in PR). As far as I can tell, the plots are identical.

Thanks!

e3sm_diags/driver/aerosol_aeronet_driver.py

tomvothecoder · 2024-02-16T18:48:04Z

e3sm_diags/plot/cartopy/zonal_mean_2d_stratosphere_plot.py

@@ -1,15 +0,0 @@
-import xarray as xr


This file is not needed anymore (zonal_mean_2d_plot.py is used directly by this set)

tomvothecoder · 2024-02-16T18:49:14Z

e3sm_diags/plot/cartopy/zonal_mean_xy_plot.py

@@ -1,138 +0,0 @@
-from __future__ import print_function


This file is note needed anymore. I think this file was accidentally left behind in #752

e3sm_diags/driver/zonal_mean_2d_driver.py

chengzhuzhang · 2024-02-16T20:29:17Z

This is nice progress! I will review early next week.

chengzhuzhang

Great work. I think this set is ready to ship!

Refer to the PR for more information because the changelog is massive. Update build workflow to run on `cdat-migration-fy24` branch CDAT Migration Phase 2: Add CDAT regression test notebook template and fix GH Actions build (#743) - Add Makefile for quick access to multiple Python-based commands such as linting, testing, cleaning up cache and build files - Fix some lingering unit tests failure - Update `xcdat=0.6.0rc1` to `xcdat >=0.6.0` in `ci.yml`, `dev.yml` and `dev-nompi.yml` - Add `xskillscore` to `ci.yml` - Fix `pre-commit` issues CDAT Migration Phase 2: Regression testing for `lat_lon`, `lat_lon_land`, and `lat_lon_river` (#744) - Add Makefile that simplifies common development commands (building and installing, testing, etc.) - Write unit tests to cover all new code for utility functions - `dataset_xr.py`, `metrics.py`, `climo_xr.py`, `io.py`, `regrid.py` - Metrics comparison for `cdat-migration-fy24` `lat_lon` and `main` branch of `lat_lon` -- `NET_FLUX_SRF` and `RESTOM` have the highest spatial average diffs - Test run with 3D variables (`_run_3d_diags()`) - Fix Python 3.9 bug with using pipe command to represent Union -- doesn't work with `from __future__ import annotations` still - Fix subsetting syntax bug using ilev - Fix regridding bug where a single plev is passed and xCDAT does not allow generating bounds for coordinates of len <= 1 -- add conditional that just ignores adding new bounds for regridded output datasets, fix related tests - Fix accidentally calling save plots and metrics twice in `_get_metrics_by_region()` - Fix failing integration tests pass in CI/CD - Refactor `test_diags.py` -- replace unittest with pytest - Refactor `test_all_sets.py` -- replace unittest with pytest - Test climatology datasets -- tested with 3d variables using `test_all_sets.py` CDAT Migration Phase 2: Refactor utilities and CoreParameter methods for reusability across diagnostic sets (#746) - Move driver type annotations to `type_annotations.py` - Move `lat_lon_driver._save_data_metrics_and_plots()` to `io.py` - Update `_save_data_metrics_and_plots` args to accept `plot_func` callable - Update `metrics.spatial_avg` to return an optionally `xr.DataArray` with `as_list=False` - Move `parameter` arg to the top in `lat_lon_plot.plot` - Move `_set_param_output_attrs` and `_set_name_yr_attrs` from `lat_lon_driver` to `CoreParameter` class Regression testing for lat_lon variables `NET_FLUX_SRF` and `RESTOM` (#754) Update regression test notebook to show validation of all vars Add `subset_and_align_datasets()` to regrid.py (#776) Add template run scripts CDAT Migration Phase: Refactor `cosp_histogram` set (#748) - Refactor `cosp_histogram_driver.py` and `cosp_histogram_plot.py` - `formulas_cosp.py` (new file) - Includes refactored, Xarray-based `cosp_histogram_standard()` and `cosp_bin_sum()` functions - I wrote a lot of new code in `formulas_cosp.py` to clean up `derivations.py` and the old equivalent functions in `utils.py` - `derivations.py` - Cleaned up portions of `DERIVED_VARIABLES` dictionary - Removed unnecessary `OrderedDict` usage for `cosp_histogram` related variables (we should do this for the rest of the variables in in #716) - Remove unnecessary `convert_units()` function calls - Move cloud levels passed to derived variable formulas to `formulas_cosp.CLOUD_BIN_SUM_MAP` - `utils.py` - Delete deprecated, CDAT-based `cosp_histogram` functions - `dataset_xr.py` - Add `dataset_xr.Dataset._open_climo_dataset()` method with a catch for dataset quality issues where "time" is a scalar variable that does not match the "time" dimension array length, drops this variable and replaces it with the correct coordinate - Update `_get_dataset_with_derivation_func()` to handle derivation functions that require the `xr.Dataset` and `target_var_key` args (e.g., `cosp_histogram_standardize()` and `cosp_bin_sum()`) - `io.py` - Update `_write_vars_to_netcdf()` to write test, ref, and diff variables to individual netCDF (required for easy comparison to CDAT-based code that does the same thing) - Add `cdat_migration_regression_test_netcdf.ipynb` validation notebook template for comparing `.nc` files CDAT Migration Phase 2: Refactor `zonal_mean_2d()` and `zonal_mean_2d_stratosphere()` sets (#774) Refactor 654 zonal mean xy (#752) Co-authored-by: Tom Vo <[email protected]> CDAT Migration - Update run script output directory to NERSC public webserver (#793) [PR]: CDAT Migration: Refactor `aerosol_aeronet` set (#788) CDAT Migration: Test `lat_lon` set with run script and debug any issues (#794) CDAT Migration: Refactor `polar` set (#749) Co-authored-by: Tom Vo <[email protected]> Align order of calls to `_set_param_output_attrs` CDAT Migration: Refactor `meridional_mean_2d` set (#795) CDAT Migration: Refactor `aerosol_budget` (#800) Add `acme.py` changes from PR #712 (#814) * Add `acme.py` changes from PR #712 * Replace unnecessary lambda call Refactor area_mean_time_series and add ccb slice flag feature (#750) Co-authored-by: Tom Vo <[email protected]> [Refactor]: Validate fix in PR #750 for #759 (#815) CDAT Migration Phase 2: Refactor `diurnal_cycle` set (#819) CDAT Migration: Refactor annual_cycle_zonal_mean set (#798) * Refactor `annual_cycle_zonal_mean` set * Address PR review comments * Add lat lon regression testing * Add debugging scripts * Update `_open_climo_dataset()` to decode times as workaround to misaligned time coords - Update `annual_cycle_zonal_mean_plot.py` to convert time coordinates to month integers * Fix unit tests * Remove old plotter * Add script to debug decode_times=True and ncclimo file * Update plotter time values to month integers * Fix slow `.load()` and multiprocessing issue - Due to incorrectly updating `keep_bnds` logic - Add `_encode_time_coords()` to workaround cftime issue `ValueError: "months since" units only allowed for "360_day" calendar` * Update `_encode_time_coords()` docstring * Add AODVIS debug script * update AODVIS obs datasets; regression test results --------- Co-authored-by: Tom Vo <[email protected]> CDAT Migration Phase 2: Refactor `qbo` set (#826) CDAT Migration Phase 2: Refactor tc_analysis set (#829) * start tc_analysis_refactor * update driver * update plotting * Clean up plotter - Remove unused variables - Make `plot_info` a constant called `PLOT_INFO`, which is now a dict of dicts - Reorder functions for top-down readability * Remove unused notebook --------- Co-authored-by: tomvothecoder <[email protected]> CDAT Migration Phase 2: Refactor `enso_diags` set (#832) CDAT Migration Phase 2: Refactor `streamflow` set (#837) [Bug]: CDAT Migration Phase 2: enso_diags plot fixes (#841) [Refactor]: CDAT Migration Phase 3: testing and documentation update (#846) CDAT Migration Phase 3 - Port QBO Wavelet feature to Xarray/xCDAT codebase (#860) CDAT Migration Phase 2: Refactor arm_diags set (#842) Add performance benchmark material (#864) Add function to add CF axis attr to Z axis if missing for downstream xCDAT operations (#865) CDAT Migration Phase 3: Add Convective Precipitation Fraction in lat-lon (#875) CDAT Migration Phase 3: Fix LHFLX name and add catch for non-existent or empty TE stitch file (#876) Add support for time series datasets via glob and fix `enso_diags` set (#866) Add fix for checking `is_time_series()` property based on `data_type` attr (#881) CDAT migration: Fix African easterly wave density plots in TC analysis and convert H20LNZ units to ppm/volume (#882) CDAT Migration: Update `mp_partition_driver.py` to use Dataset from `dataset_xr.py` (#883) CDAT Migration - Port JJB tropical subseasonal diags to Xarray/xCDAT (#887) CDAT Migration: Prepare branch for merge to `main` (#885) [Refactor]: CDAT Migration - Update dependencies and remove Dataset._add_cf_attrs_to_z_axes() (#891) CDAT Migration Phase 2: Refactor core utilities and `lat_lon` set (#677) Refer to the PR for more information because the changelog is massive. Update build workflow to run on `cdat-migration-fy24` branch CDAT Migration Phase 2: Add CDAT regression test notebook template and fix GH Actions build (#743) - Add Makefile for quick access to multiple Python-based commands such as linting, testing, cleaning up cache and build files - Fix some lingering unit tests failure - Update `xcdat=0.6.0rc1` to `xcdat >=0.6.0` in `ci.yml`, `dev.yml` and `dev-nompi.yml` - Add `xskillscore` to `ci.yml` - Fix `pre-commit` issues CDAT Migration Phase 2: Regression testing for `lat_lon`, `lat_lon_land`, and `lat_lon_river` (#744) - Add Makefile that simplifies common development commands (building and installing, testing, etc.) - Write unit tests to cover all new code for utility functions - `dataset_xr.py`, `metrics.py`, `climo_xr.py`, `io.py`, `regrid.py` - Metrics comparison for `cdat-migration-fy24` `lat_lon` and `main` branch of `lat_lon` -- `NET_FLUX_SRF` and `RESTOM` have the highest spatial average diffs - Test run with 3D variables (`_run_3d_diags()`) - Fix Python 3.9 bug with using pipe command to represent Union -- doesn't work with `from __future__ import annotations` still - Fix subsetting syntax bug using ilev - Fix regridding bug where a single plev is passed and xCDAT does not allow generating bounds for coordinates of len <= 1 -- add conditional that just ignores adding new bounds for regridded output datasets, fix related tests - Fix accidentally calling save plots and metrics twice in `_get_metrics_by_region()` - Fix failing integration tests pass in CI/CD - Refactor `test_diags.py` -- replace unittest with pytest - Refactor `test_all_sets.py` -- replace unittest with pytest - Test climatology datasets -- tested with 3d variables using `test_all_sets.py` CDAT Migration Phase 2: Refactor utilities and CoreParameter methods for reusability across diagnostic sets (#746) - Move driver type annotations to `type_annotations.py` - Move `lat_lon_driver._save_data_metrics_and_plots()` to `io.py` - Update `_save_data_metrics_and_plots` args to accept `plot_func` callable - Update `metrics.spatial_avg` to return an optionally `xr.DataArray` with `as_list=False` - Move `parameter` arg to the top in `lat_lon_plot.plot` - Move `_set_param_output_attrs` and `_set_name_yr_attrs` from `lat_lon_driver` to `CoreParameter` class CDAT Migration Phase 2: Refactor `zonal_mean_2d()` and `zonal_mean_2d_stratosphere()` sets (#774) CDAT Migration Phase 2: Refactor `qbo` set (#826)

tomvothecoder added the cdat-migration-fy24 CDAT Migration FY24 Task label Feb 14, 2024

tomvothecoder added this to the FY24 Q2 (01/01/24 - 03/31/24) milestone Feb 14, 2024

tomvothecoder self-assigned this Feb 14, 2024

tomvothecoder changed the title ~~Add initial refactored driver~~ [PR]: CDAT Migration: Refactor aerosol_aeronet set Feb 14, 2024

tomvothecoder linked an issue Feb 14, 2024 that may be closed by this pull request

CDAT Migration Phase 2: Refactor aerosol_aeronet set #672

Closed

tomvothecoder added 2 commits February 16, 2024 09:39

Add initial refactored driver

97da49f

Add initial refactored aerosol_aeronet_plot.py

c03e727

tomvothecoder force-pushed the refactor/672-aerosol-aeronet branch from f971c29 to c03e727 Compare February 16, 2024 17:39

tomvothecoder added 3 commits February 16, 2024 10:30

Fix obj ref and color map panel config

2f3a8ad

- Add results png for main and dev branches

Revert change to lat_lon_plot.py

8e5d13c

Remove nested dir in main results

190a8e2

tomvothecoder commented Feb 16, 2024

View reviewed changes

e3sm_diags/driver/aerosol_aeronet_driver.py Show resolved Hide resolved

tomvothecoder commented Feb 16, 2024

View reviewed changes

e3sm_diags/driver/aerosol_aeronet_driver.py Show resolved Hide resolved

tomvothecoder commented Feb 16, 2024

View reviewed changes

e3sm_diags/driver/aerosol_aeronet_driver.py Show resolved Hide resolved

Organize plot modules and deleted deprecated ones

0953ec5

tomvothecoder commented Feb 16, 2024

View reviewed changes

e3sm_diags/driver/zonal_mean_2d_driver.py Show resolved Hide resolved

Undo rename of SET_DIR

bd96c4d

chengzhuzhang self-requested a review February 16, 2024 20:28

chengzhuzhang approved these changes Feb 27, 2024

View reviewed changes

tomvothecoder merged commit c2c7b49 into cdat-migration-fy24 Feb 27, 2024
4 checks passed

tomvothecoder deleted the refactor/672-aerosol-aeronet branch February 27, 2024 21:43

tomvothecoder mentioned this pull request Mar 14, 2024

CDAT Migration Phase 2: Refactor aerosol_aeronet set #672

Closed

tomvothecoder added a commit that referenced this pull request May 1, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

c147d50

tomvothecoder mentioned this pull request Jul 3, 2024

CDAT Migration Phase 2: Refactor aerosol_budget set #673

Closed

tomvothecoder added a commit that referenced this pull request Jul 15, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

a16072f

tomvothecoder added a commit that referenced this pull request Jul 15, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

03edd3a

tomvothecoder added a commit that referenced this pull request Aug 21, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

8d76992

tomvothecoder added a commit that referenced this pull request Oct 1, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

0e3d3dd

tomvothecoder added a commit that referenced this pull request Oct 25, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

1a2de1c

tomvothecoder added a commit that referenced this pull request Oct 29, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

05b881e

tomvothecoder added a commit that referenced this pull request Oct 29, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

daeaa71

tomvothecoder added a commit that referenced this pull request Dec 4, 2024

[PR]: CDAT Migration: Refactor aerosol_aeronet set (#788)

8bf49f3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PR]: CDAT Migration: Refactor `aerosol_aeronet` set #788

[PR]: CDAT Migration: Refactor `aerosol_aeronet` set #788

tomvothecoder commented Feb 14, 2024 •

edited

Loading

tomvothecoder left a comment

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

chengzhuzhang commented Feb 16, 2024

chengzhuzhang left a comment

[PR]: CDAT Migration: Refactor aerosol_aeronet set #788

[PR]: CDAT Migration: Refactor aerosol_aeronet set #788

Conversation

tomvothecoder commented Feb 14, 2024 • edited Loading

Description

Checklist

tomvothecoder left a comment

Choose a reason for hiding this comment

tomvothecoder Feb 16, 2024

Choose a reason for hiding this comment

tomvothecoder Feb 16, 2024

Choose a reason for hiding this comment

chengzhuzhang commented Feb 16, 2024

chengzhuzhang left a comment

Choose a reason for hiding this comment

[PR]: CDAT Migration: Refactor `aerosol_aeronet` set #788

[PR]: CDAT Migration: Refactor `aerosol_aeronet` set #788

tomvothecoder commented Feb 14, 2024 •

edited

Loading