Importing metplus modules to external python scripts

[mkavulich]

Hi, I have come up with a potentially interesting use case for METplus tools, and I was curious to get some feedback from the METplus developers on my implementation, whether that feedback is "that looks fine", "there's a better way to do that", "please don't open that can of worms", or some other reaction.

The use case mainly boils down to how we currently use METplus in the context of the UFS SRW Application. Because we want to avoid duplication of information while allowing users as much flexibility as possible, we allow users to use METplus wildcards for filenames; e.g. for reading in forecast files hrrr.t12z.wrfprsf01.grib2, hrrr.t12z.wrfprsf01.grib2, etc., we simply have FCST_FN_TEMPLATE: 'hrrr.t{init?fmt=%H}z.wrfprsf{lead?fmt=%HH}.grib2' in our SRW configuration file. However, in addition to this we have a desire to check that all of our files needed for verification are present prior to running METplus (to avoid unnecessary core hours, confusing errors, etc.). This means we need some way to include the same filename wildcard logic that is used within METplus. Previously we have simply implemented the regex that we need to check the relevant filenames, but I found that I can skip this duplication of effort and potential for errors by simply importing the string_template_substitution module directly:

sys.path.append(os.environ['METPLUS_ROOT'])
from metplus.util import string_template_substitution as sts

...

        fn = sts.do_string_sub(tmpl=filename_template,init=initdate,lead=leadsec,valid=validdate,shift=time_lag)
        fp = os.path.join(base_dir, fn)

        if os.path.isfile(fp):
            if verbose:
                print(f"Found file (fp) for the current forecast hour (fhr; relative to the cycle date cdate):\n  fhr = \"{fhr}\"\n  cdate = \"{cdate}\"\n  fp = \"{fp}\"")

...
...
etc.

I realize this is undocumented and likely not an intended use of METplus (directly accessing internal functions like this) but I found that this dramatically decreases the complexity of our scripts, and ensures self-consistency with the handling of filenames in our workflow. Since I can't find any references to other people attempting this sort of thing, I thought it made sense to open a discussion here so developers and others might offer feedback.

Let me know if you'd like more information, I could post the full script or even a full example use case if that would be helpful.

Thanks!

[JohnHalleyGotway]

@mkavulich, big picture, I certainly don't have any objection to accessing functionality provided by any of the METplus Python components. We'd like the METplus wrappers to work well out of the box, but providing them as modules that can be loaded facilitates advanced usage like what you're doing. The only potential downside would be if future changes to the METplus string template substitution logic broke the logic in SRW. But you can protect against that using automated testing to figure out as soon as something changes. And I certainly don't expect any changes that'd break the logic.

That being said, I want to call your attention to some work that's on @georgemccabe's plate. #2524 is intended to check for the amount of valid input files and only run if a configurable threshold is met. Folks at NOAA had been doing this upstream with shell scripts, and @georgemccabe has been working to replicate that logic directly through METplus wrappers. However @georgemccabe just went on parental leave, so I don't have any guarantee on whether the complete set of changes will make it into the next release. But I just wanted to put it on your radar. So while using string template substitution directly is great, you may find that validating input files can also be done directly by METplus in the future.

[georgemccabe]

Hi @mkavulich,

I agree with @JohnHalleyGotway's response above.

It sounds like the solution outlined in issue #2524 would allow you to configure METplus to perform the available input file checks and prevent a timely/costly run if insufficient files were found. However, since this issue is not marked as required for the official release, it is unclear if this enhancement will make it into the next release. It is good to know that this functionality would be useful to your project as well. This helps to prioritize this work.

Regarding the import of the metplus python module in your scripts:

As you have discovered, it is certainly possible to import metplus into your own python script. You can even install the module into your python environment by running pip install . in the METplus directory of the GitHub repository, which would remove the need for the sys.path.append line in your script. We currently do not have any documentation describing how to do this because we don't currently have a clear need to support it officially. Others have included similar logic in their scripts in the past, but in each instance we have come to the conclusion that the desired behavior could be accomplished just as easily by running the METplus wrappers directly via run_metplus.py. That said, I am excited that you are experimenting with using the METplus wrappers utilities in this way. I think there is a lot of useful functionality available that could be utilized. As @JohnHalleyGotway mentioned, there is a risk of logic/function names/etc changing from under you, but I don't foresee any major changes that would break the filename template substitution logic.

A few comments on the code snippet you shared:

• The metplus.util module is set up so that you can import any function directly from it, so you should be able to do from metplus.util import do_string_sub then call do_string_sub directly instead of sts.do_string_sub. Doing this would ensure that the function would be imported successfully in your script even if metplus.util was reorganized in a later version of the wrappers.
• The shift argument that you passed to do_string_sub defines a value that would replace a tag called {shift} since nearly any keyword can be passed to the function. It does not actually define a time shift to apply other filename template tags using the Shift Keyword functionality, e.g. {valid?fmt=%H?shift=3H}. Maybe I am not understanding what you script is doing with that value.

I hope this lengthy response is helpful and answers your questions. Feel free to post your full python script here if you'd like so other users can utilize the same behavior until issue #2524 is completed.