Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature 2202 python embedding logs #2209

Merged
merged 10 commits into from
Aug 2, 2022
Merged

Conversation

hsoh-u
Copy link
Collaborator

@hsoh-u hsoh-u commented Jul 25, 2022

Expected Differences

The main goals are 1) avoiding segmentation fault with python embedding and 2) giving better log message to fix the problems at the python script. Initially it supports the numpy array only. The generating python list from the customized script caused the seg fault. Now the python list is supported. And more log messages were added to help to fix the problems at the python script at MET. And the base python script (met_point_obs.py) checks the generated python point data and gives messages.

MET allows the python list with numpy data type but MET processes them without checking the numpy data type (for example, unsigned data types). The safe way will be not allowing numpy data type members into the python list. But it's allowed for the convenience of customers. The side effect can happen (for example, unsigned numpy data types). The ==INFO_PYTHON== is given for this case.

Here are added log messages )from MET and from the base python scriupt:
'''
-- when the python input does not exists:
ERROR :
ERROR : straight_python_point_data -> The header is empty. Please check if python input exists
ERROR :

-- when a header or an obs member is not filled
==ERROR_PYTHON==
==ERROR_PYTHON== hdr_vld_table is empty
==ERROR_PYTHON==

ERROR :
ERROR : straight_python_point_data -> The hdr_vld_table is empty. Please check if python input is processed properly
ERROR :

-- when the first member at the integer array is the float number (note: checking the first member only):
==ERROR_PYTHON==
==ERROR_PYTHON== Not supported data type (<class 'float'>) for obs_vid[0] (int only)
==ERROR_PYTHON==
OR
==ERROR_PYTHON==
==ERROR_PYTHON== Not supported data type (<class 'str'>) for obs_vid[0] (int only)
==ERROR_PYTHON==

ERROR :
ERROR : Only int type is supported at python list. Please check the data type
ERROR :
ERROR :
ERROR : set_array_from_python(T *) -> Only int type is supported at python list for obs_vid. Check the data type from python and consider using numpy for python embedding
ERROR :

-- when the numpy member is added to python list (checking the first member only.
==INFO_PYTHON== Recommend using numpy instead of python list for obs_val (<class 'numpy.float32'>) to avoid side effect

'''

  • Do these changes introduce new tools, command line arguments, or configuration file options? [No]

    If yes, please describe:

  • Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]

    If yes, please describe:

Pull Request Testing

  • Describe testing already performed for these changes:

/d1/personal/hsoh/git/features/feature_2202_python_embedding_logs/MET/src/tools/other/plot_point_obs/plot_point_obs 'PYTHON_NUMPY=/d1/personal/hsoh/git/features/feature_2202_python_embedding_logs/MET/src/tools/other/plot_point_obs/read_doe_arm_obs_bad_float_at_int.py missing_python_input.nc' sgp30ebbrE15.b1.20190617.000000.ps -title "DOE ARM Data" -plot_grid G130 -v 4

/d1/personal/hsoh/git/features/feature_2202_python_embedding_logs/MET/src/tools/other/plot_point_obs/plot_point_obs 'PYTHON_NUMPY=/d1/personal/hsoh/git/features/feature_2202_python_embedding_logs/MET/src/tools/other/plot_point_obs/read_doe_arm_obs_bad_float_at_int.py /d1/personal//hsoh/data/MET-2202/sgp30ebbrE15.b1.20190617.000000.nc' sgp30ebbrE15.b1.20190617.000000.ps -title "DOE ARM Data" -plot_grid G130 -v 4

  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

  • Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]

  • Do these changes include sufficient testing updates? [No]

  • Will this PR result in changes to the test suite? [No]

    If yes, describe the new output and/or changes to the existing output:

  • Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

  • Review the source issue metadata (required labels, projects, and milestone).
  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s)
    Select: Organization level software support Project or Repository level development cycle Project
    Select: Milestone as the version that will include these changes
  • After submitting the PR, select Linked issues with the original issue number.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

@hsoh-u hsoh-u linked an issue Jul 25, 2022 that may be closed by this pull request
20 tasks
@JohnHalleyGotway JohnHalleyGotway added this to the MET 11.0.0 milestone Jul 28, 2022
Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve of this PR.

After merging the latest from develop into this feature branch, the latest GHA run produces no diffs. Thanks for tweaking the docs, correcting that typo, clarifying the expected data type in the comments of met_point_obs.py, and removing the commented-out development code. Those are all great changes.

@hsoh-u please proceed with the merge when you see fit and remember to close the original GitHub issue.

@JohnHalleyGotway JohnHalleyGotway removed the request for review from georgemccabe August 2, 2022 23:23
@hsoh-u hsoh-u merged commit 1afadcc into develop Aug 2, 2022
@hsoh-u hsoh-u deleted the feature_2202_python_embedding_logs branch August 11, 2022 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

Add more error checking for python embedding of point observations
2 participants