Reorganize code for MPAS dycore. Isolate FV3 and MPAS pieces. #129

dustinswales · 2024-07-19T17:36:50Z

This pull request contains changes to generalize atmospheric component of the UFS Weather Model:

The reorganization of the repository is to achieve the following goals:

Move dynamical core specific code into subdirectories.
Move common code (e.g. physics) into shareable, cross-dycore, higher-level directories.
This is outlined in Figure 1 of the "Tiger Team" report.

Refactor the CMakeLists to build an atmospheric component with either an FV3
or MPAS dynamical core.
Add CMakeLists.txt to build MPAS core_atmosphere component. The CMake recipes from the "MPAS atmospheric core" were adapted to work in the UFS build system.
Modify CCPP physics CmakeLists.txt to build independent MPAS/FV3 couplings to the physics.
Create "stub" mpas infrastructure. This includes stubs for the MPAS atmospheric component driver and overlaying NUOPC cap.

With these changes, the MPAS dynamical core build within UFS.
Next steps, build atmospheric driver for MPAS, which includes creating new typedefs for coupling between the CCPP and MPAS.

@junwang-noaa @DusanJovic-NOAA

… in the GFS physics suite (NOAA-EMC#851) * Output updraft value of convective cloud condensate in CCPP * update xkinv1 to 0.15 * add check if progsigma is true * address seg fault if progsigma=F --------- Co-authored-by: Grant Firl <[email protected]>

* Land surface upgrades for HR4

* Update diagnose_cplFields routine to use FieldBundleWrite. Needs esmf v8.6.0 * Fixed bug in aux2d dimensions for GFS meta file. * Pass return code from diagnose_cplFields back to caller * Skip 'cpl_scalars' field when dumping export state * fix fhzero for GEFS * fix cpl_scalars (NCAR#6) * fix issues w/ cplscalars * error out of all 3 spatial indices are not present * add check for scalar_id = 0 * modify for timeslices and times (NCAR#7)

…ture for CCPP. More CMake modifications

scrasmussen · 2024-08-07T15:39:20Z

CMakeLists.txt

+endif()
+
+if (MPAS)
+  add_library(${DYCORE_TARGET}


Naming the add_library target ${DYCORE_TARGET} means we can't have both MPAS and FV3 build at the same time. At least my CMake v3.26.3 was complaining. Maybe not a big deal for now since we are just spinning up but might be good to give them unique names for the future?

Great point. Giving them unique names will also keep the hope alive of "piggy backing" the other dycore in a diagnostic mode. I will make this change.

@scrasmussen I added new variables for each dycore target, but didn't go as far as implementing them. I need to think about this a bit more.

scrasmussen · 2024-08-07T15:46:24Z

This is a great start and is looking pretty good!

Right now I am mainly just trying to build this PR and am having a build issue. I tried building the feature/mpas-dycore-reorg on Derecho with the instructions from the slides.

$ CMAKE_FLAGS="-DAPP=ATMMPAS -DCCPP_SUITES=FV3_HRRR_gf" 
$ export CMAKE_FLAGS
$ ./build.sh

@dustinswales, I'm getting the following error, did you change the MPAS submodule at all? Thanks!

-- Configuring done (50.5s)
CMake Error at FV3/mpas/MPAS-Model/src/core_atmosphere/CMakeLists.txt:154 (add_library):
  Cannot find source file:

    physics/ccpp_kinds.F

dustinswales · 2024-08-07T16:51:23Z

This is a great start and is looking pretty good!

Right now I am mainly just trying to build this PR and am having a build issue. I tried building the feature/mpas-dycore-reorg on Derecho with the instructions from the slides.
$ CMAKE_FLAGS="-DAPP=ATMMPAS -DCCPP_SUITES=FV3_HRRR_gf" 
$ export CMAKE_FLAGS
$ ./build.sh
@dustinswales, I'm getting the following error, did you change the MPAS submodule at all? Thanks!
-- Configuring done (50.5s)
CMake Error at FV3/mpas/MPAS-Model/src/core_atmosphere/CMakeLists.txt:154 (add_library):
  Cannot find source file:

    physics/ccpp_kinds.F

Sorry, I forgot to include the changes needed to MPAS..
Should be good to go now.

…ow also includes NOAA-EMC#752) (NOAA-EMC#798) * Convert GFS_grid DDT: use contiguous arrays instead of blocked data structures * Add GCC-based CI build --------- Co-authored-by: Alex Richert <[email protected]> Co-authored-by: Dustin Swales <[email protected]>

scrasmussen · 2024-08-08T17:57:02Z

Sorry, I forgot to include the changes needed to MPAS.. Should be good to go now.

Nice! This builds for me now. Just had to make slight changes to work with the GNU software stack on Derecho.

I changed 0.5 to 0.5_RKIND for lines 4776 and 5153 to fix Error: Type mismatch in argument 'ua' at (1); passed REAL(8) to REAL(4) errors. That is for the FV3/mpas/MPAS-Model/src/core_atmosphere/dynamics/mpas_atm_time_integration.F.

* atmos_cube_sphere: heat_source is allocated when SKEB is true and d_con is zero

CMakeLists.txt

dustinswales · 2024-08-13T17:13:10Z

@DusanJovic-NOAA Thanks for the comments. I incorporated your suggestions, and grouped all the FV3 and MPAS pieces together.

CMakeLists.txt

ccpp/data/GFS_typedefs.meta

* send CCPP ebu_smoke to UPP ebb * upp: remove GSD_NC synonyms and rename GSD_NC fields instead * update upp hash: g2, g2tmpl, etc. * upp: correct pressure levels for hafs-ar postxconfig files * bugfix: fixed_sfc2_type defaults to fixed_sfc1_type * Add aerosol fix from Li Pan. --------- Co-authored-by: Wen Meng <[email protected]>

mkavulich · 2024-08-23T17:42:46Z

ccpp/CMakeLists.txt

@@ -1,8 +1,49 @@
 cmake_minimum_required(VERSION 3.0)


Minor point, can we update this required version to 3.19 since other CMakeLists.txt require it?

…-EMC#837) * Add namelist parameter, ncrement_file_on_native_grid

…e refactor and C3/SAS/MYNN fix) (NOAA-EMC#865) * Host side changes for h2o photochemistry scheme * A fix for the issue to run C3 or SAS convection with the prognostic area fraction closure, and MYNN PBL: tendency_of_vertically_diffused_tracer_concentration from MYNN PBL --------- Co-authored-by: Dustin Swales <[email protected]> Co-authored-by: Lisa Bengtsson <[email protected]>

* add fire_behavior smoke coupling and flags and export variables for fire behavior * added: inst_pres_levels, inst_geop_levels, inst_zonal_wind_levels, inst_merid_wind_levels, inst_surface_roughness, inst_temp_height2m, inst_spec_humid_height2m, inst_pres_height_surface, mean_prec_rate, inst_rainfall_amount * add fire behavior tendencies to physics * add hflx_fire and evap_fire to FV3 imports * added surface emissions fire_smoke imports and initialized the variables and fsmoke tracer index --------- Co-authored-by: masih <[email protected]> Co-authored-by: Grant Firl <[email protected]>

* Add support for IntelLLVM compiler: only sub modules (atmos_cubed_sphere, ccpp/physics and upp) are updated. * Update atmos_cubed_sphere, ccpp/physics and upp

…ture/mpas-dycore-reorg

dustinswales · 2024-10-02T15:27:28Z

@scrasmussen @mkavulich @grantfirl
I'm going to merge this and open a PR into NOAA-EMC:fv3atm from NCAR:feature/mpas-dycore, instead of dustinswales:feature/mpas-dycore-reorg.

junwang-noaa · 2024-10-02T16:15:10Z

CMakeLists.txt

+    endif()
+    if(MULTI_GASES)
+      list(APPEND _ufsatm_defs_private MULTI_GASES)
+    endif()


I am not sure why these if statements here are under the MOVING_NEST if statement. May need further check when merging in EMC's repo

@junwang-noaa Thanks. I'm running the UFS RTs now to ensure everything is working before opening PRs into the UFS repositories. (I will make sure to move this statement out of the MOVING_NEST logic.)

dustinswales and others added 7 commits July 19, 2024 17:30

Reorganize code for MPAS dycore. Isolate FV3 and MPAS pieces.

eb63f9f

Added more build step pieces, up until the MPAS dycore build

1301e07

Land surface upgrades for HR4 (NOAA-EMC#855)

927261d

* Land surface upgrades for HR4

Some more changes. Almost building.

587ea55

MPAS building in UFSATM! Some more reorganization. Initial infrastruc…

b4f6463

…ture for CCPP. More CMake modifications

dustinswales requested review from grantfirl, VanderleiVargas-NOAA and scrasmussen August 5, 2024 18:24

Move ccpp/drivers into common directory (revert prioir move)

424e86b

scrasmussen reviewed Aug 7, 2024

View reviewed changes

Forgot to link to personal for of MPAS submodule

8168a66

SKEB fix when using hydrostatic option (NOAA-EMC#862)

66bded5

* atmos_cube_sphere: heat_source is allocated when SKEB is true and d_con is zero

DusanJovic-NOAA reviewed Aug 13, 2024

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

DusanJovic-NOAA reviewed Aug 13, 2024

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

Address reviewers comments

b56c486

junwang-noaa reviewed Aug 13, 2024

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

junwang-noaa reviewed Aug 13, 2024

View reviewed changes

ccpp/data/GFS_typedefs.meta Outdated Show resolved Hide resolved

dustinswales and others added 2 commits August 13, 2024 17:50

Addres comments

64afb90

mkavulich reviewed Aug 23, 2024

View reviewed changes

DavidNew-NOAA and others added 3 commits August 27, 2024 09:32

Add ability to read increment files on native cubed sphere grid (NOAA…

70b3065

…-EMC#837) * Add namelist parameter, ncrement_file_on_native_grid

Move MPAS to ufs-community reporitory

5dec7c2

dustinswales and others added 4 commits September 3, 2024 17:19

Revert unneeded change

1b2c665

Support IntelLLVM compiler (NOAA-EMC#861)

1aba87c

* Add support for IntelLLVM compiler: only sub modules (atmos_cubed_sphere, ccpp/physics and upp) are updated. * Update atmos_cubed_sphere, ccpp/physics and upp

Merge branch 'develop' of https://github.com/NOAA-EMC/fv3atm into fea…

afb09b5

…ture/mpas-dycore-reorg

dustinswales merged commit f571248 into NCAR:feature/mpas-dycore Oct 2, 2024

junwang-noaa reviewed Oct 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorganize code for MPAS dycore. Isolate FV3 and MPAS pieces. #129

Reorganize code for MPAS dycore. Isolate FV3 and MPAS pieces. #129

dustinswales commented Jul 19, 2024 •

edited

Loading

scrasmussen Aug 7, 2024

dustinswales Aug 7, 2024

dustinswales Aug 13, 2024

scrasmussen commented Aug 7, 2024 •

edited

Loading

dustinswales commented Aug 7, 2024

scrasmussen commented Aug 8, 2024

dustinswales commented Aug 13, 2024

mkavulich Aug 23, 2024

dustinswales commented Oct 2, 2024

junwang-noaa Oct 2, 2024

dustinswales Oct 2, 2024

Reorganize code for MPAS dycore. Isolate FV3 and MPAS pieces. #129

Reorganize code for MPAS dycore. Isolate FV3 and MPAS pieces. #129

Conversation

dustinswales commented Jul 19, 2024 • edited Loading

scrasmussen Aug 7, 2024

Choose a reason for hiding this comment

dustinswales Aug 7, 2024

Choose a reason for hiding this comment

dustinswales Aug 13, 2024

Choose a reason for hiding this comment

scrasmussen commented Aug 7, 2024 • edited Loading

dustinswales commented Aug 7, 2024

scrasmussen commented Aug 8, 2024

dustinswales commented Aug 13, 2024

mkavulich Aug 23, 2024

Choose a reason for hiding this comment

dustinswales commented Oct 2, 2024

junwang-noaa Oct 2, 2024

Choose a reason for hiding this comment

dustinswales Oct 2, 2024

Choose a reason for hiding this comment

dustinswales commented Jul 19, 2024 •

edited

Loading

scrasmussen commented Aug 7, 2024 •

edited

Loading