Merge pull request #10 from sister-jpl/sister-dev

Sister dev

.gitignore

@@ -127,3 +127,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# Pycharm .idea
+.idea

README.md

@@ -3,61 +3,74 @@
 ## Description
 
 The L2A spectral resample PGE takes as input surface reflectance and uncertainty images and spectrally resamples the data
-to 10nm spectral spacing. Spectral resampling is performed in a two-step process, first bands are aggregated and averaged to the closest resolution to the target resolution (10 nm). For example DESIS data, which has an average spectral spacing of 2.55 nm, is aggregated and averaged every 4 bands. Next a piecewise cubic interpolator is used to interpolate the spectra to the target wavelength spacing. Output range for all sensors except DESIS is 400-2500 nm, while the DESIS output range is 400-990 nm.
+to 10nm spectral spacing. Spectral resampling is performed in a two-step process, first bands are aggregated and averaged 
+to the closest resolution to the target resolution (10 nm). For example DESIS data, which has an average spectral spacing 
+of 2.55 nm, is aggregated and averaged every 4 bands. Next a piecewise cubic interpolator is used to interpolate the spectra 
+to the target wavelength spacing. Output range for all sensors except DESIS is 400-2500 nm, while the DESIS output range 
+is 400-990 nm.
 
 ![DESIS spectral resampling example](./figures/spectral_resample_example.png)
 
 ## PGE Arguments
 
-In addition to required MAAP job submission arguments the L2A spectral resampling PGE also takes the following argument(s):
+The L2A spectral resampling PGE takes the following argument(s):
 
 
-|Argument| Type |  Description | Default|
-|---|---|---|---|
-| reflectance_dataset| file |L2A reflectance dataset| -|
-| uncertainty_dataset| file |L2A uncertainty dataset| -|
-| crid| config | Composite release identifier| '000'|
+| Argument            | Description                          | Default |
+|---------------------|--------------------------------------|---------|
+| reflectance_dataset | L2A reflectance dataset              | -       |
+| uncertainty_dataset | L2A uncertainty dataset              | -       |
+| crid                | Composite release identifier         | '000'   |
+| experimental        | Designates outputs as "experimental" | 'True'  |
 
 ## Outputs
 
 The outputs of the L2A spectral resampling PGE use the following naming convention:
 
-    SISTER_<SENSOR>_L2A_RSRFL_<YYYYMMDDTHHMMSS>_CRID<_SUBPRODUCT>
+    (EXPERIMENTAL-)SISTER_<SENSOR>_L2A_RSRFL_<YYYYMMDDTHHMMSS>_<CRID>_<SUBPRODUCT>
 
-and produce the following data products:
-
-|Product description |  Units | Example filename |
-|---|---|---|
-| ENVI 10nm Resampled reflectance datacube | % | SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001.bin|
-| ENVI 10nm Resampled reflectance header file  | - | SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001.hdr|
-| Resampled reflectance metadata  | - | SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001.met.json|
-| False color reflectance quicklook  | - |  SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001.png |
-| ENVI 10nm Resampled uncertainty datacube | % | SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001_RSUNC.bin|
-| ENVI 10nm Resampled uncertainty header file  | - |SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001_RSUNC.hdr|
-| ENVI 10nm Resampled uncertainty metedata | - |SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001_RSUNC.met.json|
-| PGE runconfig| - |  SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001.runconfig.json |
-| PGE log| - |  SISTER\_AVNG\_L2A\_RSRFL\_20220502T180901\_001.log |
-
-## Algorithm registration
-
-This algorithm can be registered using the algorirthm_config.yml file found in this repository:
-
-	from maap.maap import MAAP
-	import IPython
-
-	maap = MAAP(maap_host="sister-api.imgspec.org")
-
-	resample_alg_yaml = './sister-resample/algorithm_config.yaml'
-	maap.register_algorithm_from_yaml_file(file_path= resample_alg_yaml)
-
-## Example
-
-	resample_job_response = maap.submitJob(
-	    algo_id="sister-resample",
-	    version="2.0.0",
-	    reflectance_dataset= 'SISTER_AVNG_L2A_RFL_20220502T180901_001',
-	    crid = '001'
-	    publish_to_cmr=False,
-	    cmr_metadata={},
-	    queue="sister-job_worker-16gb",
-	    identifier='SISTER_AVNG_L2A_RSRFL_20170827T175432_001"
+Note that the "EXPERIMENTAL-" prefix is optional and is only added when the "experimental" flag is set to True.
+
+The following data products are produced:
+
+| Product description                             | Units | Example filename                                              |
+|-------------------------------------------------|-------|---------------------------------------------------------------|
+| ENVI 10nm Resampled reflectance datacube        | %     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000.bin            |
+| ENVI 10nm Resampled reflectance header file     | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000.hdr            |
+| Resampled reflectance metadata (STAC formatted) | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000.json           |
+| False color reflectance quicklook               | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000.png            |
+| ENVI 10nm Resampled uncertainty datacube        | %     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000_UNC.bin        |
+| ENVI 10nm Resampled uncertainty header file     | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000_UNC.hdr        |
+| Resampled uncertainty metedata (STAC formatted) | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000_UNC.json       |
+| PGE runconfig                                   | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000.runconfig.json |
+| PGE log                                         | -     | SISTER\_AVCL\_L2A\_RSRFL\_20110513T175417\_000.log            |
+
+Metadata files are [STAC formatted](https://stacspec.org/en) and compatible with tools in the [STAC ecosystem](https://stacindex.org/ecosystem).
+
+## Executing the Algorithm
+
+This algorithm requires [Anaconda Python](https://www.anaconda.com/download)
+
+To install and run the code, first clone the repository and execute the install script:
+
+    git clone https://github.com/sister-jpl/sister-resample.git
+    cd sister-resample
+    ./install.sh
+    cd ..
+
+Then, create a working directory and enter it:
+
+    mkdir WORK_DIR
+    cd WORK_DIR
+
+Copy input files to the work directory. For each "dataset" input, create a folder with the dataset name, then download 
+the data file(s) and STAC JSON file into the folder.  For example, the reflectance dataset input would look like this:
+
+    WORK_DIR/SISTER_AVCL_L2A_RFL_20110513T175417_000/SISTER_AVCL_L2A_RFL_20110513T175417_000.bin
+    WORK_DIR/SISTER_AVCL_L2A_RFL_20110513T175417_000/SISTER_AVCL_L2A_RFL_20110513T175417_000.hdr
+    WORK_DIR/SISTER_AVCL_L2A_RFL_20110513T175417_000/SISTER_AVCL_L2A_RFL_20110513T175417_000.json
+
+Finally, run the code 
+
+    ../sister-resample/pge_run.sh --reflectance_dataset SISTER_AVCL_L2A_RFL_20110513T175417_000 --uncertainty_dataset SISTER_AVCL_L2A_RFL_20110513T175417_000_UNC
+

algorithm_config.yaml

@@ -1,8 +1,8 @@
 # DO NOT DELETE
 # THIS CONFIG IS AUTO-GENERATED BY ADE UI
-algorithm_name: sister-resample
-algorithm_version: 2.0.1
-repository_url: https://github.com/EnSpec/sister-resample.git
+algorithm_name: ogc-sister-resample
+algorithm_version: 3.0.0
+repository_url: https://github.com/sister-jpl/sister-resample.git
 docker_container_url: mas.dit.maap-project.org/root/maap-workspaces/base_images/vanilla:dit
 
 # fill out these fields
@@ -15,18 +15,17 @@ run_command: sister-resample/pge_run.sh
 # set a storage value in GB or MB or KB, e.g. "100GB", "20MB", "10KB"
 disk_space: 50GB
 inputs:
-
-  file:
+  directory:
     - name: reflectance_dataset
       description:
       required: True
-
     - name: uncertainty_dataset
       description:
       required: True
-
   config:
     - name: crid
       data_type:
       default: '000'
       description:
+    - name: experimental
+      default: 'True'

generate_runconfig.py

@@ -6,34 +6,38 @@
 Author: Adam Chlus
 """
 
-import sys
+import argparse
 import json
 
+
 def main():
-    '''
+    """
         This function takes as input the path to an inputs.json file and exports a run config json
         containing the arguments needed to run the L1 preprocess PGE.
 
-    '''
-
-    inputs_json  = sys.argv[1]
-
-    with open(inputs_json, 'r') as in_file:
-        inputs =json.load(in_file)
-
-    run_config = {"inputs":{}}
+    """
 
-    for file_dict in inputs["file"]:
-        for key,value in file_dict.items():
-            run_config["inputs"][key] = value
+    parser = argparse.ArgumentParser(description='Parse inputs to create runconfig.json')
+    parser.add_argument('--reflectance_dataset', help='Path to reflectance dataset')
+    parser.add_argument('--uncertainty_dataset', help='Path to uncertainty dataset')
+    parser.add_argument('--crid', help='CRID value', default="000")
+    parser.add_argument('--experimental', help='If true then designates data as experiemntal', default="True")
+    args = parser.parse_args()
 
-    run_config["inputs"].update(inputs["config"])
+    run_config = {
+        "inputs": {
+            "reflectance_dataset": args.reflectance_dataset,
+            "uncertainty_dataset": args.uncertainty_dataset,
+            "crid": args.crid
+        }
+    }
+    run_config["inputs"]["experimental"] = True if args.experimental.lower() == "true" else False
 
     config_file = 'runconfig.json'
 
     with open(config_file, 'w') as outfile:
-        json.dump(run_config,outfile,indent=3)
+        json.dump(run_config, outfile, indent=4)
 
 
-if __name__=='__main__':
+if __name__ == '__main__':
     main()

install.sh

@@ -5,6 +5,8 @@ pge_dir=$(cd "$(dirname "$0")" ; pwd -P)
 # Need to do custom install to prevent dependency errors
 conda create -y --name sister python=3.8
 source activate sister
+pip install pystac==1.8.4
+pip install spectral==0.23.1
 
 git clone https://github.com/EnSpec/hytools.git -b 1.5.0
 cd hytools

pge_run.sh

@@ -5,7 +5,7 @@ source activate sister
 pge_dir=$(cd "$(dirname "$0")" ; pwd -P)
 
 echo "Creating runconfig"
-python ${pge_dir}/generate_runconfig.py inputs.json
+python ${pge_dir}/generate_runconfig.py "${@:1}"
 
 echo "Running L2a Spectral Resample PGE"
-python ${pge_dir}/spectral_resample.py runconfig.json
+python ${pge_dir}/spectral_resample.py runconfig.json