Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce MOAB coupler interfaces and update moab driver #5979

Merged
merged 502 commits into from
Dec 21, 2023

Conversation

rljacob
Copy link
Member

@rljacob rljacob commented Oct 10, 2023

Add interfaces to driver-moab in WaterCycle components: EAM, ELM, MOSART, MPAS-Ocean, MPAS-SeaIce. Summary of changes is below.

MOAB interfaces and support routines in components are isolated with a "HAVE_MOAB" ifdef that will be automatically defined if the MOAB coupler is chosen. This allows components to use either mct or moab couplers for now. Choice is made during create_newcase. There are also cpp defines for debugging. MOABDEBUG will output .h5m files throughout the coupling processs and additional log info (very slow). MOABCOMP will calculate diffs between MCT and MOAB coupling fields. These defines need to be set by hand if you wish to use them. These will be removed later. When using default driver-mct, there is no impact on answers and no dependency on MOAB.

Also bring in latest version of driver-moab which is feature-complete for the watercycle case. All maps (except r2o) are calculated online. Merging, fluxes and all other coupler calculations are performed with moab data structures. Currently includes MCT-scaffolding for verification (data moving between components follows both MCT- and MOAB- paths; offline maps still read by MCT; all mct-based datatypes still present).

[BFB]

Build system mods

  • If USE_MOAB is defined, call find_package(MOAB), add MOAB include dirs and HAVE_MOAB cpp.
  • Also set MOAB_ROOT on chrysalis and gce.
  • See also mods to MPAS build

EAM mods

  • cpl/atm_comp_mct.F90 - define and call routines to set the physics grid "cloud" (used when physics is on the SE points) and also define routines to export/import EAM data to/from MOAB meshes and call them when HAVE_MOAB is defined.
  • dynamics/se/semoab_mod.F90 - new module to hold routines for defining SE meshes (coarse, fine and physgrid) to MOAB
  • dynamics/se/dyn_comp.F90 - define MOAB apps for each mesh and call create_moab_meshes
  • eam/bld/config_files/definition.xml - add moab as interface option
  • eam/bld/configure - add moab as comp_intf option
  • eam/cime_config/buildnml - add moab as interface option

ELM MODS

  • cpl/lnd_comp_mct.F90: define and call the routine to define the ELM MOAB app and describe the mesh. Also define and use routines to export/import data to/from the MOAB coupler.
  • main/srfrdMod.F90: calculate vertex locations for MOAB from domain file info.
  • config_definitions.xml: add moab as comp_intf option

MOSART MODS

  • bld/build-namelist and namelist_default* files: define and set new namelist var for mesh frivinp_mesh
  • cpl/rof_comp_mct.F90: define and call the routine to define the MOSART MOAB app and describe the mesh. Also define and use routines to export/import data to/from the MOAB coupler.
  • riverroute/RtmMod.F90, RtmVar.F90: read in the new frivinp_mesh file. Pass to coupler infobuffer.

MPAS MODS

  • mpas-framework/Makefile: Add HAVE_MOAB and MOAB include paths
  • mpas-framework/src/Makefile, CMakelists.txt: Add MOAB parts if COMP_INTERFACE is moab
  • mpas-framework/driver/mpas-subdriver.F: init MOAB if using this driver
  • mpas-framework/framework/mpas-moabmesh.F: new routine to define MPAS mesh to MOAB. Works for all MPAS components
  • mpas-ocean/driver/ocn_comp_mct.F and
  • mpas-seaice/driver/ice_comp_mct.F: call routine to define mesh to MOAB. Define and call routines to export/import data to/from MOAB coupler

iulian787 and others added 30 commits January 2, 2023 20:20
really bad habit of copy and paste
lat, lon, mask are set
hgt is left default
instead of doing

  mapper_Sl2a%intx_mbid = mbintxla

  I was doing

    mapper_Sl2a%src_mbid = mbintxla

    which overwrote important stuff
Turn off domain check for the MOAB case and
add log messages about aream and areacor sections of init.
Change VECT_MAP default from cart3d to none
Need to see if that should be the default in
production settings as well.
Put back the abort calls for any nonzero error in the
MOAB seq_map_map section.  Don not set valid_context to false.
Add moab parts to mapper_SFo2i.  Only 2i mapper we need for now.
for atm add more counters in debug mode
intial state export was missing for rof, ocn, ice
Comment out abort in copy portion of moab seq_map_map.
Temporary until atm_import branch is merged.
Adjust num_moab_export values. first use in cime_init will be 00.
Second use in cime_init for atm will be 01.
Add a FlxAlb2Atm h5m file for MOABDEBUG from prep_aoflux_calc_xao_ax
need to use branch
moab : iulian07/check_existing_pargraph
Move writing of h5m files to end of seq_frac_init
since each fraction set is modified throughout the routine.

reallocated a TagValues array and make sure init value is 0._r8
Move a seq_map call to the correct place and remove
some code no longer needed now that seq_map_map works
for MOAB.

Start to introduce mbofxid
Add mapper_Fof2a and mapper_Sof2a which will map from
the flux copy of the ocean mesh to atmosphere for MOAB.
Also initialize them for MCT so seq_map_map doesn't need
any changes.
Use new mapper_Fof2a and mapper_Sof2a to map the xao fluxes.
These are needed for MOAB to operate on the correct mesh.

Also output a .h5m file of the ocean flux mesh.
it is different from ocean cplid
ocn(1)%cplcompid + 1000
xao_omct is like an AV, it needs to live longer
otherwise communication does not work
Add code to set the calculated lfrac on the MOAB atm mesh.

Also remove some commented out/blocked out code.
Copy link
Contributor

@liho745 liho745 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

River-related changes look good.

Copy link
Contributor

@jonbob jonbob left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from a visual inspection

also, replace MOABCOMP with MOABDEBUG in seq_map_map
l2racc_lm_cnt should have been read from file
@rljacob
Copy link
Member Author

rljacob commented Dec 4, 2023

@wlin7 if you want any changes, let me know before Wednesday. I'm going to merge it then.

iulian787 and others added 11 commits December 4, 2023 14:51
two major issues:
1) atm import moab had to comment a line, it was commented
in mct import after we did
moab import
2) factoring during atm second init call
that was always missing
This reverts commit a0d1b67.
Update moab driver config settings with latest from mct driver.
Remove extra MOABDEBUG blocks used for restart debugging.
Add FAN coupler pieces to moab-coupler since ELM now expects it.
it is used for normalization in mapping routines
how we did not get affected ? It was missing
Also, it made the tag Faxa_swvdf non-existent on river model
or not?
Add fix to surface pressure coupling in the moab import.  This
will be needed for the model to build after merging.
Update salt flux in moab sea-ice export which is necessary
for model to build moab-cases after merge.
Update driver moab with developments from the mct
coupler that will be needed when merged.
Commit d080e69 changed the values exported out of the ocean
for dhdx and dhdy of ssh.  Make those changes in the moab export routine.
Needed for merge to work.
@rljacob
Copy link
Member Author

rljacob commented Dec 20, 2023

merged to next

Mosart tests that use the USRDAT grid rely on testmods
to actually define the grid/domain.  Add definitions of frivinp_mesh.
Also fix some old-style xmlchange commands.
rljacob added a commit that referenced this pull request Dec 20, 2023
Second merge to next to fix some Mosart tests that failed
@rljacob rljacob merged commit 46b65c1 into master Dec 21, 2023
2 checks passed
@rljacob rljacob deleted the rljacob/driver-moab/add-mb-intf branch December 21, 2023 17:41
@rljacob
Copy link
Member Author

rljacob commented Dec 21, 2023

For the record here are the files that merged cleanly ("Changes to be committed"), files that had to have conflicts fixed ("Unmerged paths"), and files that needed further modification ("Changes not staged") to work with the new build system.

Changes to be committed:
	modified:   components/CMakeLists.txt
	modified:   components/cice/bld/config_files/definition.xml
	modified:   components/cmake/find_dep_packages.cmake
	modified:   components/eam/bld/config_files/definition.xml
	modified:   components/eam/bld/configure
	modified:   components/eam/cime_config/buildnml
	modified:   components/eam/src/cpl/atm_comp_mct.F90
	modified:   components/eam/src/dynamics/se/dyn_comp.F90
	new file:   components/eam/src/dynamics/se/semoab_mod.F90
	modified:   components/elm/bld/config_files/config_definition.xml
	modified:   components/elm/src/cpl/lnd_comp_mct.F90
	modified:   components/elm/src/main/surfrdMod.F90
	modified:   components/elm/src/utils/domainMod.F90
	modified:   components/mosart/bld/build-namelist
	modified:   components/mosart/bld/namelist_files/namelist_defaults_mosart.xml
	modified:   components/mosart/bld/namelist_files/namelist_definition_mosart.xml
	modified:   components/mosart/cime_config/testdefs/testmods_dirs/mosart/mos_usrdat/user_nl_mosart
	modified:   components/mosart/cime_config/testdefs/testmods_dirs/mosart/sediment/shell_commands
	modified:   components/mosart/cime_config/testdefs/testmods_dirs/mosart/sediment/user_nl_mosart
	modified:   components/mosart/cime_config/testdefs/testmods_dirs/mosart/unstructure/user_nl_mosart
	modified:   components/mosart/src/cpl/rof_comp_mct.F90
	modified:   components/mosart/src/riverroute/RtmMod.F90
	modified:   components/mosart/src/riverroute/RtmVar.F90
	modified:   components/mpas-framework/Makefile
	modified:   components/mpas-framework/src/CMakeLists.txt
	modified:   components/mpas-framework/src/Makefile
	modified:   components/mpas-framework/src/driver/mpas_subdriver.F
	modified:   components/mpas-framework/src/framework/Makefile
	new file:   components/mpas-framework/src/framework/mpas_moabmesh.F
	modified:   components/mpas-ocean/driver/ocn_comp_mct.F
	modified:   driver-mct/cime_config/config_component.xml
	modified:   driver-moab/cime_config/buildexe
	new file:   driver-moab/cime_config/buildlib_cmake
	modified:   driver-moab/cime_config/buildnml
	modified:   driver-moab/cime_config/config_component.xml
	modified:   driver-moab/cime_config/config_component_cesm.xml
	modified:   driver-moab/cime_config/config_component_e3sm.xml
	modified:   driver-moab/cime_config/config_pes.xml
	modified:   driver-moab/cime_config/namelist_definition_drv.xml
	modified:   driver-moab/cime_config/namelist_definition_drv_flds.xml
	modified:   driver-moab/cime_config/namelist_definition_modelio.xml
	modified:   driver-moab/main/cime_comp_mod.F90
	modified:   driver-moab/main/cime_driver.F90
	modified:   driver-moab/main/component_mod.F90
	modified:   driver-moab/main/component_type_mod.F90
	modified:   driver-moab/main/cplcomp_exchange_mod.F90
	modified:   driver-moab/main/prep_aoflux_mod.F90
	modified:   driver-moab/main/prep_atm_mod.F90
	new file:   driver-moab/main/prep_iac_mod.F90
	modified:   driver-moab/main/prep_ice_mod.F90
	modified:   driver-moab/main/prep_lnd_mod.F90
	modified:   driver-moab/main/prep_ocn_mod.F90
	modified:   driver-moab/main/prep_rof_mod.F90
	new file:   driver-moab/main/seq_diagBGC_mct.F90
	modified:   driver-moab/main/seq_diag_mct.F90
	modified:   driver-moab/main/seq_flux_mct.F90
	modified:   driver-moab/main/seq_frac_mct.F90
	modified:   driver-moab/main/seq_hist_mod.F90
	modified:   driver-moab/main/seq_io_mod.F90
	modified:   driver-moab/main/seq_map_mod.F90
	modified:   driver-moab/main/seq_map_type_mod.F90
	modified:   driver-moab/main/seq_rest_mod.F90
	modified:   driver-moab/shr/seq_comm_mct.F90
	modified:   driver-moab/shr/seq_drydep_mod.F90
	modified:   driver-moab/shr/seq_flds_mod.F90
	modified:   driver-moab/shr/seq_infodata_mod.F90
	new file:   driver-moab/shr/seq_pauseresume_mod.F90
	new file:   driver-moab/shr/shr_dust_mod.F90
	new file:   driver-moab/shr/shr_fan_mod.F90

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   cime_config/machines/cmake_macros/gnu_anlgce.cmake
	both modified:   cime_config/machines/cmake_macros/gnu_chrysalis.cmake
	both modified:   cime_config/machines/cmake_macros/intel_chrysalis.cmake
	both modified:   cime_config/machines/config_machines.xml
	both modified:   components/cmake/build_model.cmake
	both modified:   components/mpas-ocean/driver/mpaso_cpl_indices.F
	both modified:   components/mpas-seaice/driver/ice_comp_mct.F

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   components/cmake/modules/FindCsmShare.cmake
	modified:   share/CMakeLists.txt

All the files in the last two groups were "git added", after doing a few tests, and then "git commit" completed the merge. I used the git rerere cache which helped a lot in doing the merge to next a couple of times and the merge to master.

@jgfouca
Copy link
Member

jgfouca commented Dec 21, 2023

Congrats! This was a big change

@jgfouca
Copy link
Member

jgfouca commented Dec 21, 2023

Do you want me to take a second look at any of those cmake conflicts?

@rljacob
Copy link
Member Author

rljacob commented Dec 21, 2023

Thanks! You don't have to look at anything. You had already shown how to fix it in https://github.com/rljacob/E3SM/tree/jgfouca/moab_fixes.

What I didn't try was adding those changes to my branch so I wouldn't have to do it during the merge. This branch only had some of your build changes so wasn't sure it would work.

jonbob added a commit that referenced this pull request Jan 9, 2024
…#6134)

Fix MPAS standalone compile issues

These two small changes allow MPAS components to compile in stand-alone
mode. It should not affect E3SM compile or simulations. The fixes are:
* A missing endif was added to the Makefile
* A redundant corelist declaration was removed from mpas.F because it
  was added to mpas_subdriver.F in PR Introduce MOAB coupler interfaces
  and update moab driver #5979, and the subdriver is used by mpas.F.
These fixes should work for ocean, sea ice, and MALI cores.

Fixes #6133
Fixes #6138

[BFB]
jonbob added a commit that referenced this pull request Jan 10, 2024
Fix MPAS standalone compile issues

These two small changes allow MPAS components to compile in stand-alone
mode. It should not affect E3SM compile or simulations. The fixes are:
* A missing endif was added to the Makefile
* A redundant corelist declaration was removed from mpas.F because it
  was added to mpas_subdriver.F in PR Introduce MOAB coupler interfaces
  and update moab driver #5979, and the subdriver is used by mpas.F.
These fixes should work for ocean, sea ice, and MALI cores.

Fixes #6133
Fixes #6138

[BFB]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFB PR leaves answers BFB Coupled Model Coupler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants