Implement run phase and dynamics-physics coupling for MPAS dynamical core

[kuanchihwang]

Tag name (required for release branches):

None

Originator(s):

kuanchihwang

Summary (include the keyword ['closes', 'fixes', 'resolves'] and issue number):

Implement run (i.e., time integration) phase and dynamics-physics coupling for MPAS dynamical core. CAM-SIMA with MPAS dynamical core is now able to run toward the end successfully.

Describe any changes made to build system:

None

Describe any changes made to the namelist:

None

List any changes to the defaults for the input datasets (e.g. boundary datasets):

None

List all files eliminated and why:

None

List all files added and what they do:

• A src/dynamics/mpas/dyn_coupling.F90
  • Implement dynamics-physics coupling and vice versa

List all existing files that have been modified, and describe the changes:

• M src/dynamics/mpas/driver/dyn_mpas_subdriver.F90
  • Implement MPAS subdriver
  • Support for computing edge wind tendency in MPAS subdriver
• M src/dynamics/mpas/dyn_comp.F90
  • Support for computing edge wind tendency in MPAS subdriver
  • Implement dyn_run
  • Sort use statements and add comments
  • Factor out variable finalization
  • Implement reverse
  • Refactor assignments between CAM-SIMA and MPAS in terms of reverse
  • Implement dyn_exchange_constituent_state
  • Switch to use dyn_exchange_constituent_state
• M src/dynamics/mpas/stepon.F90
  • Sort use statements and adjust indentation
  • Wire up dyn_run
  • Wire up dynamics-physics coupling

Test steps:

"<path-to>/CAM-SIMA/cime/scripts/create_newcase" --compiler "<gnu|intel>" --case "<case-name>" --compset FKESSLER --project "<project-id>" --res mpasa480_mpasa480 --run-unsupported
cd "<case-name>"
./xmlchange CAM_LINKED_LIBS="-lmpas"
./xmlchange DEBUG="TRUE"
./xmlchange DOUT_S="FALSE"
./case.setup
# Add the following to `user_nl_cam`:
# * config_block_decomp_file_prefix = '${DIN_LOC_ROOT}/atm/cam/inic/mpas/mpasa480.graph.info.part.'
# * config_coef_3rd_order = 1.0
# * config_dt = 1800.0
# * config_len_disp = 120000.0
# * config_number_cam_damping_levels = 0
# * config_number_rayleigh_damp_u_levels = 5
# * config_print_detailed_minmax_vel = .true.
# * config_rayleigh_damp_u = .true.
./case.build
./case.submit

Observe log entries similar to the following in atm.log.<job-id>.<date>-<time>. Also observe that CAM-SIMA with MPAS dynamical core is now able to run toward the end successfully.

dyn_debug_print (0): Preparing for dynamics-physics coupling
dyn_debug_print (0): Setting physics state "constituents" from MPAS state "scalars"
dyn_debug_print (0): Setting physics state variables column by column
dyn_debug_print (0): Setting physics state variables externally
dyn_debug_print (0): Converting physics state "constituents"
dyn_debug_print (0): Preparing for physics-dynamics coupling
dyn_debug_print (0): Setting MPAS state "scalars" from physics state "constituents"
dyn_debug_print (0): Converting MPAS state "scalars"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "scalars"
dyn_mpas_debug_print (0): Exchanging halo layers for "scalars"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_debug_print (0): Setting MPAS physics tendency "tend_ru_physics"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_compute_edge_wind entered
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "tend_uzonal"
dyn_mpas_debug_print (0): Exchanging halo layers for "tend_uzonal"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "tend_umerid"
dyn_mpas_debug_print (0): Exchanging halo layers for "tend_umerid"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "tend_ru_physics"
dyn_mpas_debug_print (0): Exchanging halo layers for "tend_ru_physics"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_compute_edge_wind completed
dyn_debug_print (0): Setting MPAS physics tendency "tend_rho_physics"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "tend_rho_physics"
dyn_mpas_debug_print (0): Exchanging halo layers for "tend_rho_physics"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_debug_print (0): Setting MPAS physics tendency "tend_rtheta_physics"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "tend_rtheta_physics"
dyn_mpas_debug_print (0): Exchanging halo layers for "tend_rtheta_physics"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_run entered
dyn_mpas_debug_print (0): Time integration of MPAS dynamical core begins at 0001-01-01_00:00:00
dyn_mpas_debug_print (0): Time step 1 completed
dyn_mpas_debug_print (0): Time integration of MPAS dynamical core ends at 0001-01-01_00:30:00
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_run completed
 ------------------------
 CAM-SIMA time step advanced (nstep = 1)
 ------------------------

... (SNIPPED) ...

Number of completed timesteps: 240
Time step 241 partially done to provide convectively adjusted and time filtered values for history tape.

 ******* END OF MODEL RUN *******
(atm_comp_nuopc) -------------------------------------------------------------------------
(atm_comp_nuopc) CAM: end of main integration loop
(atm_comp_nuopc) -------------------------------------------------------------------------

[nusbaume]

Thanks for the new MPAS coupling infrastructure @kuanchihwang, and apologies for the review delay! I have some questions and change requests, but hopefully they are all either easy to implement or can be pushed off to a future PR. Of course if you have any questions or concerns with anything I suggested just let me know!

[nusbaume]

Thanks for responding to my questions and review requests @kuanchihwang! Everything looks good to me now.

[nusbaume]

@kuanchihwang Just FYI that @PeterHjortLauritzen is going to look over the dyn-phys coupling just to make sure the science all seems reasonable. Thanks!

[mgduda]

The changes in this PR look good to me. I was able to run a 120-km FKESSLER case, and aside from some differences in the w field when comparing with CAM-MPAS (which I think may be attributable to reasons other than any issues in this PR), the results seemed to be in general agreement.