Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update API for CaseClass #30

Merged
merged 15 commits into from
Oct 9, 2020
Merged

Commits on Sep 25, 2020

  1. Update API for CaseClass

    This first commit is done in anticipation of supporting time series files in
    addition to history files, and also using intake-esm catalogs behind the
    scenes.
    
    1. start_date & end_date are not part of the class constructor
       -- basically all the __init__ routine should do is set up a catalog
       -- _find_hist_files() also doesn't rely on these variables either; this
          routine is essentially building the catalog, and we don't restrict the
          files listed until we are ready to call open_mfdataset()
    2. _open_history_files() -> gen_dataset(); this routine is where start_date and
       end_date are specified, and it returns a dataset rather than updating a
       class member variable. Logic to apply start_date and end_date at this stage
       is much simpler than doing it in _find_hist_files() (or I figured out a much
       easier way to do it). This routine also now expects a list of variable names
       to include in the dataset.
    3. compare_fields_at_lat_lon expects array of DataArrays, not array of cases
    4. Added get_varnames_from_metadata_list() to utils/utils.py because I need
       this function to generate the list of variable names for gen_dataset() in
       several notebooks.
    
    I also created a script that submits jobs to the slurm queue to re-run
    notebooks, though it does not work for the trend_maps or plot_suite_00[34]
    notebooks (I think because of the reliance on dask and NCAR_jobqueue).
    mnlevy1981 committed Sep 25, 2020
    Configuration menu
    Copy the full SHA
    361cd79 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2020

  1. Configuration menu
    Copy the full SHA
    390ecc2 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2020

  1. Configuration menu
    Copy the full SHA
    5309e40 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2020

  1. Add ability to read data from time series

    gen_dataset() first reads time series from campaign, then reads any additional
    data from history files (from archive or run directory). There's an API change
    where, instead of expecting start_date and end_date (strings formatted as
    'YYYY-MM'), the function wants integers start_year and end_year. Figuring out
    better default values will be necessary to extend this to other CESM runs but
    it's a good start.
    
    I split plot_suite_004.ipynb into 4-year segments, as it was choking on 8 years
    of output. I also added plot_suite_map notebooks for 004 for years 0006 and
    0007 (we're one month shy of 0008).
    mnlevy1981 committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    4c43880 View commit details
    Browse the repository at this point in the history
  2. CaseClass doesn't need yaml

    I had left the "import yaml" statement in from when I toyed with the idea of
    passing the metadata yaml file to __init__
    mnlevy1981 committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    3ee0d33 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2020

  1. Improve xr.concat call

    Concatenating time series dataset with history file dataset now works as
    expected
    mnlevy1981 committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    b3d38bb View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2020

  1. clean up variable names and logic in CaseClass

    Also reran plot_suite notebooks in an environment that will match previous
    master (note that this changes plots in Sanity Check). Next commit will be
    updated trend_maps, and the plan is to squash-merge this PR to remove the
    commits that changed the plots and then changed them back.
    mnlevy1981 committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    eed1035 View commit details
    Browse the repository at this point in the history
  2. Re-run trend_maps

    Re-ran in an environment that matches the previous master to minimize diffs in
    the PR.
    mnlevy1981 committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    ebcdf70 View commit details
    Browse the repository at this point in the history
  3. run plot_suite_maps_0001_004

    I didn't realize this notebook hadn't been run in previous commits
    mnlevy1981 committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    f9657d8 View commit details
    Browse the repository at this point in the history
  4. Reran plot_suite_maps notebooks

    Some of these were plotting the wrong year; I refactored all of them to be more
    clear about what is being plotted.
    mnlevy1981 committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    3ae8225 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2020

  1. New: compare time series to history files

    For now the comparisons are done based on variables in diag_metadata.yaml.
    Required a flag to suppress some CaseClass output.
    mnlevy1981 committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    cf9c15a View commit details
    Browse the repository at this point in the history
  2. Re-run notebooks with xarray 0.16.1

    All plot_suite_maps now match master
    mnlevy1981 committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    53c1f9f View commit details
    Browse the repository at this point in the history
  3. Code clean-up following review

    * run_notebook.sh launches a single notebook via slurm, and run_all.sh is now
      run_all.py (python-script) that calls run_notebook.sh
    * _find_timeseries_files() now includes pop.h.nyear1 files
    * gen_dataset():
      1. no longer assume time_bound is the name of the variable (look at bounds
         property of ds["time"]
      2. if bounds are not decoded, still update start_year before looking for
         history files (via decoding time, which may not be best solution)
    * use list.extend() instead of += operator
    mnlevy1981 committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    17d34a3 View commit details
    Browse the repository at this point in the history
  4. Clean up run_notebooks.sh

    mnlevy1981 committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    c013454 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e2c58b7 View commit details
    Browse the repository at this point in the history