Add GP ORTs #946

dustinswales · 2021-12-06T17:09:55Z

Attention

This PR was merged with the wrong fv3atm submodule pointer hash. The hash referenced in this branch (cf0a73180b2d9ac55ebfce4785a7270d205423db) resides/d in @dustinswales's fv3atm fork. The correct fv3atm hash in the NOAA-EMC repository would have been 86d4bb3.

PR Checklist

Ths PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.
This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
are specified below.
Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.
New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsiblity to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

This PR contains new operational regression tests for RRTMGP in physics prototype 7.

Issue(s) addressed

See NCAR/ccpp-physics#782 for description of issues.

Testing

Testing at commit b707db5 on Cheyenne against baseline develop-20211203 shows the following:

GNU: all jobs ran to completion. The following two tests failed comparison (as expected):

control_rrtmgp
control_rrtmgp_debug

INTEL: All non-RRTMGP tests ran to completion and passed. The following test ran to completion but failed comparison (as expected):

control_rrtmgp_debug

The following jobs compiled but failed at runtime:

cpld_control_p7_rrtmgp
control_p7_rrtmgp
regional_RRTMGP
control_rrtmgp
control_rrtmgp_c192

These were tested on Intel and gnu on Hera.
This PR contains new tests to be added to tests/rt.conf.

Dependencies

waiting on RRTMGP cleanup from ORTs NCAR/ccpp-physics#782
waiting on Changes to FV3 for ccpp-physics #782 NOAA-EMC/fv3atm#428

DeniseWorthen · 2021-12-06T17:48:25Z

Please update to the last commit for FV3 and ufs-weather.

…r-model into add_GPorts

dustinswales · 2021-12-06T18:07:05Z

@DeniseWorthen
Should be all up-to-date now

DeniseWorthen · 2021-12-06T18:43:33Z

tests/tests/control_p7_rrtmgp

+
+export FV3_RUN=control_run.IN
+export CCPP_SUITE=FV3_GFS_v16_p7_rrtmgp
+export NEW_DIAGTABLE=diag_table_gfsv16_merra2


This needs to be consistent with the newest control_p7 test (NEW_DIAGTABLE-> DIAGTABLE)

DeniseWorthen · 2021-12-06T19:37:26Z

Could you update your PR information for the CCPP-physics PR? It is showing as closed.

Also, is there an issue in either FV3 or CCPP that you can list in the 'issues addressed' section?

dustinswales · 2021-12-06T19:46:31Z

@DeniseWorthen
I updated the ccpp-physics link in the PR.
There are no open issues to reference.

MinsukJi-NOAA · 2021-12-06T20:07:46Z

I am not able to build control_p7_rrtmgp and cpld_control_p7_rrtmgp with the following errors:

1403:/scratch2/NCEPDEV/stmp1/Minsuk.Ji/PR946_2/FV3/ccpp/data/GFS_typedefs.F90(2109): error #6459: This field name has been defined more than once within the same structure.   [SCMPSW]
1406:/scratch2/NCEPDEV/stmp1/Minsuk.Ji/PR946_2/FV3/ccpp/data/GFS_typedefs.F90(2109): error #6227: This symbol has multiple POINTER statement/attribute declarations which is not allowed.   [SCMPSW]
1409:/scratch2/NCEPDEV/stmp1/Minsuk.Ji/PR946_2/FV3/ccpp/data/GFS_typedefs.F90(2028): error #7367: The data value NULL() can only be assigned to a Fortran POINTER.
1412:/scratch2/NCEPDEV/stmp1/Minsuk.Ji/PR946_2/FV3/ccpp/data/GFS_typedefs.F90(2109): error #7367: The data value NULL() can only be assigned to a Fortran POINTER.
1415:/scratch2/NCEPDEV/stmp1/Minsuk.Ji/PR946_2/FV3/ccpp/data/GFS_typedefs.F90(7570): error #6158: The structure-name is invalid or is missing.   [INTERSTITIAL]
1419:make[2]: *** [FV3/ccpp/physics/CMakeFiles/ccpp_physics.dir/__/data/GFS_typedefs.F90.o] Error 1
1422:make[1]: *** [FV3/ccpp/physics/CMakeFiles/ccpp_physics.dir/all] Error 2
1424:make: *** [all] Error 2

dustinswales · 2021-12-06T20:23:30Z

@MinsukJi-NOAA
Sorry, there were multiple declarations/allocations of Interstitial%scmpsw in GFS_typedefs.F90.
(Not sure why the merge didn't catch this, but should be good to go now)

MinsukJi-NOAA · 2021-12-06T21:08:49Z

@dustinswales Could you grab the ORT results for control_p7_rrtmgp (/scratch2/NCEPDEV/stmp1/Minsuk.Ji/PR946_2/tests/OpnReqTests_hera.intel.log) and commit? ORT for cpld_control_p7_rrtmgp cannot be run at this time due to wave component.

dustinswales · 2021-12-06T21:14:34Z

@MinsukJi-NOAA
NP. done

DeniseWorthen · 2021-12-06T21:16:58Z

Please update BL_DATE in rt.sh to 20211206.

climbfuji · 2021-12-08T13:39:54Z

The weird thing is that the same setup runs in DEBUG mode, but not in PROD or REPRO mode. I'll use the parallel debugger tomorrow morning to see why it is failing in that place.
…

When I run this through allinea ddt, I get weird values for gptS, see attached screenshot.

RobertPincus · 2021-12-08T13:44:34Z

@DomHeinzeller That is certainly an error. It suggests corrupted memory, I guess.

BrianCurtis-NOAA · 2021-12-08T14:09:47Z

@DomHeinzeller is -fsanitize=address set in the compiler options? that may help? Although I've seen weird integers like that even when the build passes with sanitize.

…mg_debug on Jet w/ Intel

climbfuji · 2021-12-08T14:36:21Z

@DomHeinzeller is -fsanitize=address set in the compiler options? that may help? Although I've seen weird integers like that even when the build passes with sanitize.

We don't have this compiler flag yet, will try manually.

This is the screenshot for exactly the same run, same place, when compiled in DEBUG mode. The values look good.

DeniseWorthen · 2021-12-08T16:02:14Z

@dustinswales would you please merge Dom's PR to update the rt.conf? Thanks.

Turn off tests in rt.conf

climbfuji · 2021-12-08T17:13:13Z

@dustinswales I have a little more info. The repro mode is apparently broken due to previous build system updates (recall that we don't really test the repro mode ... a flaw). I run in prod mode, I get more information on the variables. I stopped the model in DDT before the offending line. The screenshot shows that jeta(2) has one bogus (for reasons I do not know yet).

Also look at variables fmajor and k ...

RobertPincus · 2021-12-08T17:22:28Z

@DomHeinzeller @dustinswales Variable jeta is set on line 104 of the same file. Tracing up you can see there should be no way to get a negative value. I don't know what to make of the debugger not being able to read fmajor

DomHeinzeller · 2021-12-08T17:25:48Z

This probably happens because something else is writing into the variables ... I guess

…

On Dec 8, 2021, at 10:22 AM, Robert Pincus ***@***.***> wrote: @DomHeinzeller <https://github.com/DomHeinzeller> @dustinswales <https://github.com/dustinswales> Variable jeta is set on line 104 of the same file. Tracing up you can see there should be no way to get a negative value. I don't know what to make of the debugger not being able to read fmajor — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#946 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN7FF5AA4BM4SAUQ6OPB2Q3UP6H57ANCNFSM5JPEPSQQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

RobertPincus · 2021-12-08T17:28:41Z

I guess too, but who? And why would someone do such a thing 😢

RobertPincus · 2021-12-08T17:29:57Z

@DomHeinzeller In more seriousness, these variables are all internal to RRTMGP. The big tables like kmajor are private components of class structures. The smaller variables like jeta are internal to the subroutines.

BrianCurtis-NOAA · 2021-12-08T18:06:57Z

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: RT
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/796090531/20211208161511/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_187200
Test control_stochy 026 failed in check_result failed
Test control_stochy 026 failed in run_test failed
Please make changes and add the following label back:
jet-intel-RT

DeniseWorthen · 2021-12-08T18:57:24Z

@dustinswales Logs are not being posted from any of the RTs because they can't get pushed to your fork. I'll collect them in once place for you to commit.

The cheyenne.intel baseline created but I think I'm not sure the verify ran or where it is located. I may just run it manually if I can't find it.

DeniseWorthen · 2021-12-08T19:03:17Z

The jet.intel failure of the control_stochy test was another case of the atmf000.nc file not comparing (same file as the control_csawmg_debug failure seen earlier).

DeniseWorthen · 2021-12-08T21:31:22Z

@dustinswales Please commit the logs in /scratch2/NCEPDEV/stmp1/Denise.Worthen/PR946

DusanJovic-NOAA · 2021-12-17T01:17:12Z

I am getting build error on my laptop when compiling mo_gas_optics_kernel:

[1/1075] Generating Fortran dyndep file FV3/ccpp/physics/CMakeFiles/ccpp_physics.dir/Fortran.dd
ninja: build stopped: multiple rules generate FV3/ccpp/physics/mo_gas_optics_kernels.mod.

in CMakeLists.txt I see two mo_gas_optics_kernels.F90 being compiled, both define same module (mo_gas_optics_kernels)

$ grep mo_gas_optics_kernels FV3/ccpp/physics/CMakeLists.txt 
                       ${LOCAL_CURRENT_SOURCE_DIR}/physics/rte-rrtmgp/rrtmgp/kernels-openacc/mo_gas_optics_kernels.F90
                       ${LOCAL_CURRENT_SOURCE_DIR}/physics/rte-rrtmgp/rrtmgp/kernels/mo_gas_optics_kernels.F90

Could this be the reason for the crash on Cheyenne.

climbfuji · 2021-12-17T03:19:17Z

I am getting build error on my laptop when compiling mo_gas_optics_kernel:
[1/1075] Generating Fortran dyndep file FV3/ccpp/physics/CMakeFiles/ccpp_physics.dir/Fortran.dd
ninja: build stopped: multiple rules generate FV3/ccpp/physics/mo_gas_optics_kernels.mod.
in CMakeLists.txt I see two mo_gas_optics_kernels.F90 being compiled, both define same module (mo_gas_optics_kernels)
$ grep mo_gas_optics_kernels FV3/ccpp/physics/CMakeLists.txt 
                       ${LOCAL_CURRENT_SOURCE_DIR}/physics/rte-rrtmgp/rrtmgp/kernels-openacc/mo_gas_optics_kernels.F90
                       ${LOCAL_CURRENT_SOURCE_DIR}/physics/rte-rrtmgp/rrtmgp/kernels/mo_gas_optics_kernels.F90
Could this be the reason for the crash on Cheyenne.

Definitely, this could be the problem. Need to figure out why this happens. Thanks for pointing this out!

dustinswales added 5 commits December 3, 2021 23:19

Add P7 regression tests with RRTMGP enabled

e7b1625

Updated .gitmodules

953a20b

Updated FV3.

f316f42

Updated ccpp-physics branch name.

bbd2318

Fixed error in rte-rrtmgp submodule hash.

7a0b1f5

dustinswales added 2 commits December 6, 2021 18:05

Updated FV3

9f107c1

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

a355c2e

…r-model into add_GPorts

DeniseWorthen reviewed Dec 6, 2021

View reviewed changes

Updated p7 GP test

9601a0f

DeniseWorthen requested review from junwang-noaa and MinsukJi-NOAA December 6, 2021 19:21

Removed redundant definition/allocation of scmpsw.

db81114

Updated ORT log

b707db5

DeniseWorthen added the Baseline Updates Current baselines will be updated. label Dec 6, 2021

DeniseWorthen assigned DeniseWorthen and MinsukJi-NOAA Dec 6, 2021

Update baseline date in rt.sh

2f6e510

DeniseWorthen added hera-gnu-BL labels Dec 6, 2021

MinsukJi-NOAA added the run-ci label Dec 6, 2021

github-actions bot removed the run-ci label Dec 6, 2021

Turn of RRTMGP tests on Cheyenne w/ Intel, turn off test control_csaw…

f89c78c

…mg_debug on Jet w/ Intel

Merge pull request #11 from climbfuji/add_GPorts_dom

80cd716

Turn off tests in rt.conf

DeniseWorthen added cheyenne-intel-BL labels Dec 8, 2021

BrianCurtis-NOAA removed cheyenne-intel-BL labels Dec 8, 2021

dustinswales added 2 commits December 8, 2021 21:44

Update logs

2c8c68a

Reverted .gitmodules.

267e524

junwang-noaa approved these changes Dec 9, 2021

View reviewed changes

junwang-noaa requested review from climbfuji and DeniseWorthen December 9, 2021 17:02

MinsukJi-NOAA approved these changes Dec 9, 2021

View reviewed changes

junwang-noaa merged commit f20ac76 into ufs-community:develop Dec 9, 2021

dustinswales deleted the add_GPorts branch February 25, 2022 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GP ORTs #946

Add GP ORTs #946

dustinswales commented Dec 6, 2021 •

edited by climbfuji

Loading

DeniseWorthen commented Dec 6, 2021

dustinswales commented Dec 6, 2021

DeniseWorthen Dec 6, 2021

dustinswales Dec 6, 2021

DeniseWorthen commented Dec 6, 2021

dustinswales commented Dec 6, 2021

MinsukJi-NOAA commented Dec 6, 2021 •

edited

Loading

dustinswales commented Dec 6, 2021

MinsukJi-NOAA commented Dec 6, 2021

dustinswales commented Dec 6, 2021

DeniseWorthen commented Dec 6, 2021

climbfuji commented Dec 8, 2021

RobertPincus commented Dec 8, 2021

BrianCurtis-NOAA commented Dec 8, 2021

climbfuji commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

climbfuji commented Dec 8, 2021 •

edited

Loading

RobertPincus commented Dec 8, 2021

DomHeinzeller commented Dec 8, 2021 via email

RobertPincus commented Dec 8, 2021

RobertPincus commented Dec 8, 2021

BrianCurtis-NOAA commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

DusanJovic-NOAA commented Dec 17, 2021

climbfuji commented Dec 17, 2021

Add GP ORTs #946

Add GP ORTs #946

Conversation

dustinswales commented Dec 6, 2021 • edited by climbfuji Loading

Attention

PR Checklist

Instructions: All subsequent sections of text should be filled in as appropriate.

Description

Issue(s) addressed

Testing

Dependencies

DeniseWorthen commented Dec 6, 2021

dustinswales commented Dec 6, 2021

DeniseWorthen Dec 6, 2021

Choose a reason for hiding this comment

dustinswales Dec 6, 2021

Choose a reason for hiding this comment

DeniseWorthen commented Dec 6, 2021

dustinswales commented Dec 6, 2021

MinsukJi-NOAA commented Dec 6, 2021 • edited Loading

dustinswales commented Dec 6, 2021

MinsukJi-NOAA commented Dec 6, 2021

dustinswales commented Dec 6, 2021

DeniseWorthen commented Dec 6, 2021

climbfuji commented Dec 8, 2021

RobertPincus commented Dec 8, 2021

BrianCurtis-NOAA commented Dec 8, 2021

climbfuji commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

climbfuji commented Dec 8, 2021 • edited Loading

RobertPincus commented Dec 8, 2021

DomHeinzeller commented Dec 8, 2021 via email

RobertPincus commented Dec 8, 2021

RobertPincus commented Dec 8, 2021

BrianCurtis-NOAA commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

DeniseWorthen commented Dec 8, 2021

DusanJovic-NOAA commented Dec 17, 2021

climbfuji commented Dec 17, 2021

dustinswales commented Dec 6, 2021 •

edited by climbfuji

Loading

MinsukJi-NOAA commented Dec 6, 2021 •

edited

Loading

climbfuji commented Dec 8, 2021 •

edited

Loading