FATES LUH2 data curation tool

tools/luh2/README.md

@@ -0,0 +1,55 @@
+# FATES LUH2 data tool README
+
+## Purpose
+
+This tool takes the raw Land Use Harmonization (https://luh.umd.edu/), or LUH2, data files as
+input and prepares them for use with FATES.  The tool concatenates the various raw data sets into
+a single file and provides the ability to regrid the source data resolution to a target
+resolution that the user designates.  The output data is then usable by FATES, mediated through
+a host land model (currently either CTSM or E3SM).
+
+For more information on how FATES utilizes this information see https://github.com/NGEET/fates/pull/1040.
+
+## Installation
+
+This tool requires the usage of conda with python3.  See https://docs.conda.io/en/latest/miniconda.html#installing
+for information on installing conda on your system.  To install the conda environment necessary to run the tool
+execute the following commands:
+
+conda env create -f conda-luh2.yml
+
+This will create a conda environment named "luh2".  To activate this environment run:
+
+conda activate luh2
+
+For more information on creating conda environments see
+https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file
+
+Note that it is planned that a subset of host land model (hlm) and hlm supported machines will incoporate this tool into the surface dataset workflow.
+As such, if you are working on one of these machines, the output from this tool may be precomputed and available for the grid resolution of interest.
+
+## Usage
+
+After activating the "luh2" environment the tool can be run from the command line with the following minimum required inputs:
+
+python luh2.py -l <raw-luh2-datafile> -s <luh2-static-datafile> -r <regrid-targetfile> -w <regridder-output> -o <outputfile>
+
+The description of the minimum required input arguments is as follows:
+- raw-luh2-datafile: this is one of three raw luh2 datafiles, either states, transitions, or management.  This is the data to be regridded and used by FATES.
+- luh2-static-datafile: supplementary 0.25 deg resolution static data used in the construction of the raw luh2 datafiles.  This is utilized to help set the gridcell mask for the output file.
+- regrid-targetfile: host land model surface data file intended to be used in conjunction with the fates run at a specific grid resolution.  This is used as the regridder target resolution.
+- regridder-output: the path and filename to write out the regridding weights file or to use an existing regridding weights file.
+- outputfile: the path and filename to which the output is written
+
+The tool is intended to be run three times, sequentially, to concatenate the raw states, transitions, and management data into a single file.  After the first run of
+the tool, a merge option should also be included in the argument list pointing to the most recent output file.  This will ensure that the previous regridding run
+will be merged into the current run as well as reusing the previously output regridding weights file (to help reduce duplicate computation).
+The luh2.sh file in this directory provides an example shell script in using the python tool in this sequential manner.  The python tool itself provides additional
+help by passing the `--help` option argument to the command line call.
+
+## Description of directory contents
+
+- luh2.py: main luh2 python script
+- luh2mod.py: python module source file for the functions called in luh2.py
+- luh2.sh: example bash shell script file demonstrating how to call luh2.py
+- conda-luh2.yml: conda enviroment yaml file which defines the minimum set of package dependencies for luh2.py

tools/luh2/conda-luh2.yml

@@ -0,0 +1,11 @@
+# This yaml file is intended for users who wish to utilize the luh2.py tool on their own machines.
+# The file is not yet tested regularly to determine if the latest versions of the dependencies will
+# always work.  This regular testing is expected to be implemented in the future.
+name: luh2
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - xesmf
+  # xarray which is autodownloaded as xesmf dependency, uses scipy, which needs netcdf4 to open datasets
+  - netcdf4

tools/luh2/luh2.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+
+# LUH2 python script
+# Usage: python luh2.py -l <raw-luh2-datafile> -s <luh2-static-datafile> \
+#                       -r <regrid-targetfile> -w <regridder-output> -o <outputfile>
+
+import argparse, os, sys
+from luh2mod import ImportData, SetMaskLUH2, SetMaskSurfData
+from luh2mod import RegridConservative, RegridLoop, CorrectStateSum
+
+# Add version checking here in case environment.yml not used
+def main():
+
+    # Add argument parser - subfunction? Seperate common module?
+    # input_files and range should be the only arguments
+    # Allow variable input files (state and/or transitions and/or management)
+    args = CommandLineArgs()
+
+    # Import and prep the LUH2 datasets and regrid target
+    ds_luh2 = ImportData(args.luh2_file,args.begin,args.end)
+    ds_regrid_target = ImportData(args.regridder_target_file,args.begin,args.end)
+
+    # Import the LUH2 static data to use for masking
+    ds_luh2_static = ImportData(args.luh2_static_file)
+
+    # Create new variable where the ice water fraction is inverted w
+    ds_luh2_static["landfrac"] = 1 - ds_luh2_static.icwtr
+
+    # Mask all LUH2 input data using the ice/water fraction for the LUH2 static data
+    ds_luh2 = SetMaskLUH2(ds_luh2, ds_luh2_static)
+    ds_luh2_static = SetMaskLUH2(ds_luh2_static, ds_luh2_static)
+
+    # Mask the regrid target
+    ds_regrid_target = SetMaskSurfData(ds_regrid_target)
+
+    # Determine if we are saving a new regridder or using an old one
+    # TO DO: add check to handle if the user enters the full path
+    # TO DO: check if its possible to enter nothing with the argument
+    regrid_reuse = False
+    # If we are merging files together, we assume that the weights file
+    # being supplied exists on file
+    if (not isinstance(args.luh2_merge_file,type(None))):
+        regrid_reuse = True
+
+    # Regrid the luh2 data to the target grid
+    # TO DO: provide a check for the save argument based on the input arguments
+    regrid_luh2,regridder_luh2 = RegridConservative(ds_luh2, ds_regrid_target,
+                                                    args.regridder_weights, regrid_reuse)
+
+    # Regrid the inverted ice/water fraction data to the target grid
+    regrid_land_fraction = regridder_luh2(ds_luh2_static)
+
+    # Adjust the luh2 data by the land fraction
+    # TO DO: determine if this is necessary for the transitions and management data
+    regrid_luh2 = regrid_luh2 / regrid_land_fraction.landfrac
+
+    # Correct the state sum (checks if argument passed is state file in the function)
+    regrid_luh2 = CorrectStateSum(regrid_luh2)
+
+    # Add additional required variables for the host land model
+    # Add 'YEAR' as a variable.
+    # This is an old requirement of the HLM and should simply be a copy of the `time` dimension
+    # If we are merging, we might not need to do this, so check to see if its there already
+    if (not "YEAR" in list(regrid_luh2.variables)):
+        regrid_luh2["YEAR"] = regrid_luh2.time
+        regrid_luh2["LONGXY"] = ds_regrid_target["LONGXY"] # TO DO: double check if this is strictly necessary
+        regrid_luh2["LATIXY"] = ds_regrid_target["LATIXY"] # TO DO: double check if this is strictly necessary
+
+    # Rename the dimensions for the output.  This needs to happen after the "LONGXY/LATIXY" assignment
+    if (not 'lsmlat' in list(regrid_luh2.dims)):
+        regrid_luh2 = regrid_luh2.rename_dims({'lat':'lsmlat','lon':'lsmlon'})
+
+    # Merge existing regrided luh2 file with merge input target
+    # TO DO: check that the grid resolution 
+    # We could do this with an append during the write phase instead of the merge
+    if (not(isinstance(args.luh2_merge_file,type(None)))):
+        ds_luh2_merge = ImportData(args.luh2_merge_file,args.begin,args.end,merge_flag=True)
+        #ds_luh2_merge = ds_luh2_merge.merge(regrid_luh2)
+        regrid_luh2 = regrid_luh2.merge(ds_luh2_merge)
+
+    # Write the files
+    # TO DO: add check to handle if the user enters the full path
+    output_file = os.path.join(os.getcwd(),args.output)
+    print("generating output: {}".format(output_file))
+    regrid_luh2.to_netcdf(output_file)
+
+def CommandLineArgs():
+
+    parser = argparse.ArgumentParser(description="placeholder desc")
+
+    # Required input luh2 datafile
+    # TO DO: using the checking function to report back if invalid file input
+    parser.add_argument("-l","--luh2_file",
+                        required=True,
+                        help = "luh2 raw states, transitions, or management data file")
+
+    # Required static luh2 data to get the ice/water fraction for masking
+    parser.add_argument("-s", "--luh2_static_file",
+                        required=True,
+                        help = "luh2 static data file")
+
+    # File to use as regridder target (e.g. a surface dataset)
+    parser.add_argument("-r","--regridder_target_file",
+                        required=True,
+                        help = "target file with desired resolution to regrid luh2 data to")
+
+    # Filename to use or save for the regridder weights
+    parser.add_argument("-w", "--regridder_weights",
+                        default = 'regridder.nc',
+                        help = "filename of regridder weights to write to or reuse (if -m option used)")
+
+    # Optional input to subset the time range of the data
+    # TODO: add support for parsing the input and checking against the allowable date range
+    parser.add_argument("-b","--begin",
+                        type = int,
+                        default = None,
+                        help = "beginning of date range of interest")
+    parser.add_argument("-e","--end",
+                        type = int,
+                        default = None,
+                        help = "ending of date range to slice")
+
+    # Optional output argument
+    parser.add_argument("-o","--output",
+                        default = 'LUH2_timeseries.nc',
+                        help = "output filename")
+
+    # Optional merge argument to enable merging of other files
+    parser.add_argument("-m", "--luh2_merge_file",
+                        default = None,
+                        help = "previous luh2 output filename to merge into current run output")
+
+    args = parser.parse_args()
+
+    return(args)
+
+if __name__ == "__main__":
+    main()

tools/luh2/luh2.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+# WARNING: This script generates intermediate copies of the LUH2
+# data which at its peak takes up approximately 42G of space.
+#
+# Note that this script must be run with the luh2 conda environment
+# It requires a single argument that points to the full path location
+# of the luh2 data and the dataset to regrid against
+
+# LUH2 data names
+DATA_LOC=$1
+STATIC_LOC=$2
+TARGET_LOC=$3
+OUTPUT_LOC=$4
+STATES_FILE=states.nc
+TRANSITIONS_FILE=transitions.nc
+MANAGE_FILE=management.nc
+STATIC_FILE=staticData_quarterdeg.nc
+REGRID_TARGET_FILE=surfdata_4x5_16pfts_Irrig_CMIP6_simyr2000_c170824.nc
+
+START=1850
+END=2015
+
+# Save files
+REGRID_SAVE=regridder.nc
+OUTPUT_FILE=LUH2_historical_0850_2015_4x5.nc
+
+# Combine strings
+STATES=${DATA_LOC}/${STATES_FILE}
+TRANSITIONS=${DATA_LOC}/${TRANSITIONS_FILE}
+MANAGE=${DATA_LOC}/${MANAGE_FILE}
+STATIC=${STATIC_LOC}/${STATIC_FILE}
+REGRID_TARGET=${TARGET_LOC}/${REGRID_TARGET_FILE}
+REGRIDDER=${OUTPUT_LOC}/${REGRID_SAVE}
+
+# Comment this out if the user already has the modified datasets available
+
+# Regrid the luh2 data against a target surface data set and then remove the states_modified file
+echo "starting storage"
+du -h ${OUTPUT_LOC}
+python luh2.py  -b ${START} -e ${END} -l ${STATES} -s ${STATIC} -r ${REGRID_TARGET} -w ${REGRIDDER} -o ${OUTPUT_LOC}/states_regrid.nc
+echo -e"storage status:\n"
+du -h ${OUTPUT_LOC}
+
+# Regrid the luh2 transitions data using the saved regridder weights file and merge into previous regrid output
+python luh2.py  -b ${START} -e ${END} -l ${TRANSITIONS} -s ${STATIC} -r ${REGRID_TARGET} -w ${REGRIDDER} \
+                -m ${OUTPUT_LOC}/states_regrid.nc -o ${OUTPUT_LOC}/states_trans_regrid.nc
+echo -e"storage status:\n"
+du -h ${OUTPUT_LOC}
+rm ${DATA_LOC}/states_regrid.nc
+
+# Regrid the luh2 management data using the saved regridder file and merge into previous regrid output
+python luh2.py  -b ${START} -e ${END} -l ${MANAGE} -s ${STATIC} -r ${REGRID_TARGET} -w ${REGRIDDER} \
+                -m ${OUTPUT_LOC}/states_trans_regrid.nc -o ${OUTPUT_LOC}/${OUTPUT_FILE}
+echo -e"storage status:\n"
+du -h ${OUTPUT_LOC}
+rm ${OUTPUT_LOC}/states_trans_regrid.nc
+rm ${REGRIDDER}