-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
additional dependencies including on NCEPLIBS-bufr #169
Comments
When I link in NCEPLIBS-bufr I get:
|
I found this: https://nixdoc.net/man-pages/IRIX/man3/f90/asnqfile.3.html The function w3fi33() is mention in this Office Note 184 from 1978: https://www.lib.ncep.noaa.gov/ncepofficenotes/files/013BA20B.pdf dbn_alerts is surely related to this NOAA page: https://www.nco.ncep.noaa.gov/pmb/docs/dbnet/doc_2.95/161.html |
Here's some of the functions that call functions from NCEPLIBS-bufr:
|
Also tagging @jbathegit and @jack-woollen . Perhaps they know the history of why w3emc, our GRIB1 library, calls functions from NCEPLIBS-bufr... |
The bacio dependency was added by Kyle. Please see following:
#129
…On Tue, Jan 31, 2023 at 12:41 PM Edward Hartnett ***@***.***> wrote:
Also tagging @jbathegit <https://github.com/jbathegit> and @jack-woollen
<https://github.com/jack-woollen> . Perhaps they know the history of why
w3emc, our GRIB1 library, calls functions from NCEPLIBS-bufr...
—
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKWSMFDYM5LTPJRW7JJOSLTWVFFDZANCNFSM6AAAAAAUMWPUFM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Yes, we knew about bacio. But bufr is new. It's not considered in the build system. And, even more disturbingly, we have a bunch of other mystery functions... |
Sorry, I have no idea why those NCEPLIBS-bufr routines are apparently being called from within w3emc. My guess would be that the w3emc routines in question are reading BUFR files, which I guess then begs the question of why the application codes which call those w3emc routines aren't just directly calling NCEPLIBS-bufr themselves? |
Well we will simply have to add NCEPLIBS-bufr as a dependency of w3emc, and all that is entailed by that. Which is a lot But that still leaves all the missing functions. Where are they coming from? |
I would strongly suggest not doing that. |
I remember why the iw3unp29 calls bufrlib. It was created in the mid-1990s as a prelude to switching the ON29 database to BUFR. It was a look alike routine which mimicked a library routine which decoded ON29 data, so programs like prepdata could transition to using the BUFR database without further ado. I think w2miscan is a similar story. The simple fix, from the nceplibs point of view, is to delete it from the library and move it into whatever apps actually still use it. |
@aerorahul I would love to not introduce a dependency here on NCEPLIBS-bufr. But I am not introducing it, I just discovered it. ;-) How can I take these functions out? I have no idea where the might be used elsewhere. When you say:
I don't actually know what you mean. Who are you referring to, and what do I tell them? Even if the plan is to remove all these functions eventually, I can't do it immediately and I need to do a w3emc release. Suggestions welcome. |
@jack-woollen thanks for the history, but it's unclear how to achieve your suggestion. If I take out the code, perhaps some applications will then break. |
@edwardhartnett @jbathegit @aerorahul @ilianagenkova Bits and pieces. There are 3 subroutines in w3emc lib which reference the bufrlib. |
@jack-woollen, thanks for tagging me here! |
@ilianagenkova Only the three members of w3emc mentioned are at issue. We'd like to retire them if possible. AFAIK they were only ever used in prepobs and bufr-dump. Subroutine w3miscan is called from prepssmi in bufr-dump. I don't think that's used anymore since the ssmi data is long gone. Can you check that? Thanks! An alternative would be to include w3miscan in the prepssmi code. I did that already to process reanalysis data dating back to the 90's and early 2000's. It's not a big change. |
@jack-woollen, I confirm that w3miscan is the only w3emc code that obsproc/prepobs/bufr-dump is looking for (in the source code). No production logs show we actually evoke it at the moment, and there is no ssmi data in the prepbufr now. If one is to argue we might need it for reanalysis, then I would consider including it in the prepobs repo. @edwardhartnett , is there a version of w3emc already, with Apologies for the slow reply! |
@ilianagenkova If you want to test it, here's a branch based on current develop with those three files removed: https://github.com/AlexanderRichert-NOAA/NCEPLIBS-w3emc/tree/rm_bufr |
Here's what I suggest we do. In the next release of w3emc we introduce two new options, ENABLE_BUFR and ENABLE_EXTRA_FUNCTIONS. Both of these will be ON by default. When turned off, they will disable the functions dependent on bufr, and the functions dependent on all the mystery function calls. This can then be used by groups to test whether these functions are ever really used. When and if we get adequate confirmation, we can change the default to OFF for these enables, and in a future release, even remove the functions entirely. This way we can investigate these issues without breaking anything, and we can do the w3emc release we need. Any objections or comments? @aerorahul and @AlexanderRichert-NOAA what do you think? |
I think @edwardhartnett 's plan is sound. I know the routines that are required, that look very similar in name to those listed in @ilianagenkova 's latest comment include: The prepobs module is very involved and convoluted. It may not be straightforward to remove some of these w3emc dependencies. |
Hi All, this sounds fine to me as well, and I have a related question for @jack-woollen and @ilianagenkova: At the moment, I'm working to increase the scope of the CI test code coverage that we have for NCEPLIBS-bufr, and there's a library routine Could one of you please help here by pointing me to a "prepfits" file, presumably somewhere on WCOSS2? I know where to find prepbufr files, but again I don't know anything about "prepfits" files nor where to find one of those that I can use to cook up a CI test for (Thanks in advance! ;-) |
@edwardhartnett What would you think of setting the default to OFF, so that way it's only turned on by those who really need it? It might help discourage anyone from newly using those functions, but I don't know how much of a possibility that is... |
@jbathegit A quick search turns out a number of prepfits codes used in verify jobs. The prepfits format holds interpolated background values for t, q, w, ps, used in making fit plots. /lfs/h1/ops/prod/packages/verf_gridtobs.v4.5.2/sorc/verf_gridtobs_prepfits_pm.fd/prepfits.f |
Thanks for that information Jack, but what I need is an actual example of a prepfits file. I tried searching through all of the job output in /lfs/h1/ops/prod/output/20230227 for the string 'fits', and I found a bunch of matched filenames such as, e.g., /lfs/h1/ops/prod/com/hrrr/v4.1/hrrr.20230227/hrrrdas/hrrr_d02.t15z.fits.tm00.mem0000, but those are ASCII files, not BUFR files. I presume actual prepfits files are BUFR, correct? Would you or anyone else know where I can find an actual such file? What's even more puzzling here is that many of the subdirectories in your sorc list seem to contain inline copies of
And in fact some of these sorc packages seem to contain an inline copy of the entire library, in places where there's a libsrc/bufr subdirectory!? Or am I just missing something here? If so, I'd really appreciate if you or someone could chime in to help me make sense of what I'm seeing. |
@ShelleyMelchior-NOAA, thanks for pointing out the risk with some routines being replaced with newer, similarly named ones! @jbathegit , I see the prepfits routine in VSDB's code , if that helps, can look more into it later today. |
Sorry, I don't know what you mean by "VSDB's code"(?) What I'm looking for is an example of an actual prepfits file, which I presume is in BUFR format(?) |
VSDB : https://github.com/yangfanglin/gfs_verif.git , see here @yangfanglin, can you point us to a sample prepfits file? thanks! |
@jbathegit -- I chased down some prepfits information for you from Perry. I will email you outside of this github issue. |
@AlexanderRichert-NOAA I would be happy to set default to OFF in future releases, but let's do at least one release which does not break anything, so people have a chance to test it out. ;-) |
I haven't tested my code with Alex's code yet, I'm lacking experience with cmake and pointing to non-default library. @aerorahul , just emailed with one more question. |
and it was answered. |
Obsproc/prepobs/bufr-dump global runs with Alex's branch, https://github.com/AlexanderRichert-NOAA/NCEPLIBS-w3emc/tree/rm_bufr), completed successfully and the generated observation files match the production files. @aerorahul , thanks for helping! |
Regional obsproc runs tested fine too. |
OK, I think in the next release we should turn off the BUILD_WITH_BUFR and BUILD_WITH_EXTRA_DEPS by default. This is all now handled in the cmake build so I will close this issue. |
When I try to build with shared library support I see:
At least some of these are in NCEPLIBS-bufr.
I am investigating...
The text was updated successfully, but these errors were encountered: