Feature 2085 convert R script (pntnc2ascii.r) to python

[hsoh-u]

Expected Differences

Note: There is a warning with older netCDF4 package.

• warning with netCDF4 1.5.3
• no warning with netCDF4 1.5.6

print_nc2ascii.py:78: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
  self.use_var_id = nc_group.use_var_id.lower() == "true"

Two options were added

• --add-header: the header is added (default, no header)

• --use-comma: allows to generate comma separated text file (default: disabled)

• Do these changes introduce new tools, command line arguments, or configuration file options? [No]
  
  If yes, please describe:
  

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
  
  If yes, please describe:
  

Pull Request Testing

• Describe testing already performed for these changes:
  

python3 /d1/personal/hsoh/git/features/feature_2085_R_to_python_pntnc2ascii_R/MET/scripts/python/print_nc2ascii.py /d1/projects/MET/MET_regression/develop/NB20221110/MET-develop/test_output/ascii2nc/edr_hourly.20130827.nc | less

python3 /d1/personal/hsoh/git/features/feature_2085_R_to_python_pntnc2ascii_R/MET/scripts/python/print_nc2ascii.py /d1/projects/MET/MET_regression/develop/NB20221110/MET-develop/test_output/ascii2nc/edr_hourly.20130827.nc | less

python3 /d1/personal/hsoh/git/features/feature_2085_R_to_python_pntnc2ascii_R/MET/scripts/python/print_nc2ascii.py /d1/projects/MET/MET_regression/develop/NB20221110/MET-develop/test_output/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc | less

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]

• Do these changes include sufficient testing updates? [No]

This is a standalone utility

• Will this PR result in changes to the test suite? [No]
  
  If yes, describe the new output and/or changes to the existing output:
  

• Please complete this pull request review by [Fill in date].
  

Pull Request Checklist

See the METplus Workflow for details.

• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Development with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

@DanielAdriaansen, I'm wondering if you're planning to review this PR from Howard?

I see we're both listed but perhaps we're both thinking the other is going to review it ;)

[DanielAdriaansen]

I have a few comments and questions:

1. I ran the Rscript and compared with the Python script and noticed that the precision printed is different. From the e-mail chain "Questions for printing out MET point observation as text by python script", I thought we had agreed to align them but maybe not. For example:
   Rscript:
   MEDEDR N571UA 20130827_130000 31.9179 -96.2782 10973.41 200 NA 36002.00 1.00 0.05
   Python:
   MEDEDR N571UA 20130827_130000 31.9179 -96.2782 10973.4102 200 NA 36002.0000 1.00 0.05

The "elv" and "height" columns look like they need to have some formatting applied to the print statement. Could you add formatting to these columns @hsoh-u?

2. In the issue, @JohnHalleyGotway says:

However, I'd recommend re-evaluating the final location for this. Does it really belong in the METplus repo? Or perhaps the METdatadb repo after it's been named to METdataio? Or perhaps it does belong in MET, but should be placed in a new "met/scripts/utility" directory?

Do we need to change the location of the new print_nc2ascii.py script?

3. I recommend renaming the script from print_nc2ascii.py to print_pointnc2ascii.py.

4. In the issue description, @JohnHalleyGotway mentions:

Note that the majority of this work is likely already done in read_met_point_obs.py.

What is the difference between read_met_point_obs.py and print_nc2ascii.py?

5. Is there any current documentation about pntnc2ascii.R? If there is not, should we add documentation about this highly useful utility script somewhere?

6. As a final step, we should remove the pntnc2ascii.R script, as noted in the issue.

[hsoh-u]

I have a few comments and questions:

1. I ran the Rscript and compared with the Python script and noticed that the precision printed is different. From the e-mail chain "Questions for printing out MET point observation as text by python script", I thought we had agreed to align them but maybe not. For example:
   Rscript:
   `MEDEDR N571UA 20130827_130000 31.9179  -96.2782 10973.41 200 NA 36002.00 1.00 0.05`
   Python:
   `MEDEDR N571UA 20130827_130000  31.9179 -96.2782 10973.4102 200  NA 36002.0000 1.00 0.05`

The "elv" and "height" columns look like they need to have some formatting applied to the print statement. Could you add formatting to these columns @hsoh-u?

2. In the issue, @JohnHalleyGotway says:

However, I'd recommend re-evaluating the final location for this. Does it really belong in the METplus repo? Or perhaps the METdatadb repo after it's been named to METdataio? Or perhaps it does belong in MET, but should be placed in a new "met/scripts/utility" directory?

Do we need to change the location of the new print_nc2ascii.py script?

3. I recommend renaming the script from `print_nc2ascii.py` to `print_pointnc2ascii.py`.

4. In the issue description, @JohnHalleyGotway mentions:

Note that the majority of this work is likely already done in read_met_point_obs.py.

What is the difference between read_met_point_obs.py and print_nc2ascii.py?

5. Is there any current documentation about `pntnc2ascii.R`? If there is not, should we add documentation about this highly useful utility script somewhere?

6. As a final step, we should remove the `pntnc2ascii.R` script, as noted in the issue.

• The formatting was not enhanced for this release. It will be implemented later.
• I will rename it and move to met/scripts/utility
• read_met_point_obs.py is an example of the customized python embedding for the point observation. The base class of the read_met_point_obs.py is the met_point_obs.py. met_point_obs.py has capability to read the MET's point observation NetCDF for the testing purpose. Two cases - one as the MET point obs NetCDF and other which reads the same data by the python script - process the same input. They must produce the same output for MET tools. Some codes at met_point_obs.py are copied to print_nc2ascii.py.

[hsoh-u]

• Renamed and moved scripts/utility/print_pointnc2ascii.py
  • will be installed at MET_BASE/shared/met/utility/print_pointnc2ascii.py
• The columns are aligned
• -o "out_file_name" (or --out="out_file_name") option is added

[DanielAdriaansen]

Just a minor request.

[DanielAdriaansen]

@JohnHalleyGotway you can view the documentation I added here: https://met.readthedocs.io/en/feature_2085_r_to_python_pntnc2ascii_r/Users_Guide/reformat_point.html#point-netcdf-to-ascii-python-utility

What do you think?

[DanielAdriaansen]

Thanks for doing this Howard, this tool will be a great addition for users that need to inspect the point NetCDF files MET makes. I approve of the changes.