RRFS_cloud: Adding NSSL Physics (#95)

* Add NSSL 2-Moment Microphysics

* Update data structures to handle the additional tracer species in NSSL microphysics
* Add the code files for NSSL microphysics to the CCPP prebuild config

* Add suite for NSSL microphysics

* Add a CCPP suite to test NSSL microphysics

* Initialize ice fluxes and add "tiice" array (#119)

(1) Introducing a separate array "tiice" to store internal ice temperature. "tiice" will be added to phyf*nc output only when frac_grid=T.
(2) When cplflx=T, initializing ice fluxes by PBL calculated values when fluxes from CICE are unavailable.
(3) Adding frac_grid to namelist and no longer modifying it inside FV3GFS_io.F90. frac_grid=F by default.

* Remove cheyenne.pgi from CCPP config (#117)

remove cheyenne.pgi from CCPP config (ccpp/set_compilers.sh, ccpp/build_ccpp.sh)

* develop: merge HWRF version of saSAS with GFS version (#93)

* Merge branch 'man_hafs_sas_without_updates_of_dtc_develop_49b73a8f4149ca88f071dbaae81f1769b120f1e0' into HEAD
* gfsphysics/physics/GFS_debug.F90: add capability to debug 1-d logical arrays
* Clean up error messages in gfsphysics/GFS_layer/GFS_typedefs.F90

Co-authored-by: Dusan Jovic <[email protected]>

* Merge GFDL FV3 dynamic core new (201912) version (#65)

* Change 'mytile' to 'mygrid' corresponding to the change in atmos_cubed_sphere

* GFS.v16.0.4 physics update (#128)

* physics update from GFS.v16.0.4
* GFS.v16.0.4 physics udpate
* remove whitespaces in sfc_diff.f
* keep z0 unchanged in coupled mode in sfc_diff.f
* fix IPD run
* point to NCAR ccpp-physics branch

Co-authored-by: Jun Wang <[email protected]>

* Update for GFDL_atmos_cubed_sphere (dycore GNU bugfix) (#131)

* Bugfix for regional runs when dycore is compiled in double precision (#133)

* Update .gitmodules and submodule pointer for fv3atm for code review and testing

* Revert change to .gitmodules and update submodule pointer for GFDL_atmos_cubed_sphere

* Update develop from gsd/develop 2020/06/30 (#134)

- output of auxiliary arrays (2d, 3d) from CCPP to assist developers debugging their physics
- correctly output 3-dim tendencies for t, u, v, qv, ozone from CCPP physics
- mandatory changes for the CCPP framework updates to handle blocked data structures correctly: fix inconsistent use of `horizontal_dimension` (ix) versus `horizontal_loop_extent` (im) and other metadata corrections
- remove old pset logic from `ccpp_prebuild_config.py` (was required for CCPP dynamic build)
- updates to all suite definition files: change the order of interstitial schemes get_phi_fv3+GFS_suite_interstitial_3 and GFS_DCNV_generic_pre to match the logical flow (does not change the answer)
- move CCPP physics initialization calls in `atmos_model.F90` so that the physics can see the data from the restart files at initialization time
- add capability to read large-scale and small-scale orographic data when using GSL drag suite
- add additional fields to restart files as required for GSL physics updates
- new suite definition files for RAP and HRRR configurations
- capability to call radiation for the first `nhfrad` physics time steps for cold starts, afterwards on the cycle defined by the radiation time step

* Add regression tests with MULTI_GASES (#108)

* Update suite_FV3_GFS_2017_fv3wam.xml

* Remove unnecessary file from the CCPP prebuild configuration

* Add TACC frontera configuration

* Update to RAS in CCPP, update submodule pointer for ccpp-framework (CI tests) (NOAA-EMC#252)

* Reactivate mass flux arrays
* fix the argument list in inline_post_getattr and remove prints and post_gfs_stub.F90
* ccpp/data/GFS_typedefs.F90: correct initialization of Model%fscav
* Add FV3_GFS_v16_ras

* Cld perts (NOAA-EMC#233)

Addition of stochastic physics restarts in netCDF format that will be written at the restart interval, and at end of run.  
Also, addition of stochastic cloud perturbation and microphyiscs perturbations for SPPT.

* upgrade OPAC aerosol with more advanced MERRA2 aerosol (NOAA-EMC#254)

* changed .gitmodules to point to merra2 ccpp/physics
* revert .gitmodule

* update to Larissa version

* add atmos_cube_sphere and ccpp/physics as submodules

* add submodules of emc-2021-cloud

* update ccpp/physics: remove CAPS version mp_nssl_2mom

* ccpp/physics update: fix nssl mp reproducibility issue

* Updating pointer to ccpp/physics.

* Pointing to Christin's branch

* Adding new suite for NSSL physics.

* Updating suites to match HWT.

* Updating gitmodules

* Updating pointer to atmos_cubed_sphere.

Co-authored-by: Tim Supinie <[email protected]>
Co-authored-by: shansun6 <[email protected]>
Co-authored-by: Dom Heinzeller <[email protected]>
Co-authored-by: Dusan Jovic <[email protected]>
Co-authored-by: XiaqiongZhou-NOAA <[email protected]>
Co-authored-by: Jun Wang <[email protected]>
Co-authored-by: Jun Wang <[email protected]>
Co-authored-by: Phil Pegion <[email protected]>
Co-authored-by: AnningCheng-NOAA <[email protected]>
Co-authored-by: Jili Dong <[email protected]>

.gitignore

@@ -2,3 +2,4 @@
 *.mod
 *.a
 *.pyc
+*.i90

.gitmodules

@@ -1,12 +1,12 @@
 [submodule "atmos_cubed_sphere"]
 	path = atmos_cubed_sphere
-	url = https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere
-	branch = dev/emc
+	url = https://github.com/NOAA-GSL/GFDL_atmos_cubed_sphere
+	branch = RRFS_cloud
 [submodule "ccpp/framework"]
 	path = ccpp/framework
 	url = https://github.com/NOAA-GSL/ccpp-framework
 	branch = gsl/develop
 [submodule "ccpp/physics"]
 	path = ccpp/physics
 	url = https://github.com/NOAA-GSL/ccpp-physics
-	branch = gsl/develop
+	branch = RRFS_cloud

CMakeLists.txt

@@ -165,6 +165,7 @@ add_dependencies(fv3atm ccppdriver ccppdata ccppphys ccpp)
 target_link_libraries(fv3atm PUBLIC ccppdriver ccppdata ccppphys ccpp)
 
 target_include_directories(fv3atm PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/stochastic_physics)
+target_include_directories(fv3atm PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/../stochastic_physics)
 target_compile_definitions(fv3atm PRIVATE "${_fv3atm_defs_private}")
 
 target_link_libraries(fv3atm PUBLIC fv3dycore
@@ -192,7 +193,7 @@ endif()
 ### Install
 ###############################################################################
 install(
-  TARGETS fv3atm fv3dycore io ${CCPP_LIBRARIES} cpl stochastic_physics stochastic_physics_wrapper
+  TARGETS fv3atm fv3dycore io ${CCPP_LIBRARIES} cpl stochastic_physics_wrapper
   EXPORT fv3atm-config
   LIBRARY DESTINATION lib
   ARCHIVE DESTINATION lib)

atmos_model.F90

@@ -869,6 +869,7 @@ end subroutine update_atmos_model_state
 ! </INOUT>
 
 subroutine atmos_model_end (Atmos)
+  use get_stochy_pattern_mod, only: write_stoch_restart_atm
   type (atmos_data_type), intent(inout) :: Atmos
 !---local variables
   integer :: idx, seconds, ierr
@@ -878,12 +879,12 @@ subroutine atmos_model_end (Atmos)
 
     call atmosphere_end (Atmos % Time, Atmos%grid, restart_endfcst)
 
-    call stochastic_physics_wrapper_end(GFS_control)
-
     if(restart_endfcst) then
       call FV3GFS_restart_write (GFS_data, GFS_restart_var, Atm_block, &
                                  GFS_control, Atmos%domain)
+      call write_stoch_restart_atm('RESTART/atm_stoch.res.nc')
     endif
+    call stochastic_physics_wrapper_end(GFS_control)
 
 !   Fast physics (from dynamics) are finalized in atmosphere_end above;
 !   standard/slow physics (from IPD) are finalized in CCPP_step 'finalize'.

ccpp/CMakeLists.txt

@@ -220,18 +220,6 @@ else (SIONLIB)
   message (STATUS "Disable SIONlib support")
 endif (SIONLIB)
 
-#------------------------------------------------------------------------------
-# Set Intel MKL flags for preprocessor, compiler and linker (if defined)
-if(MKL_DIR)
-  set (MKL_INC "-m64 -I${MKL_DIR}/include")
-  set (MKL_LIB "-L${MKL_DIR}/lib -Wl,-rpath,${MKL_DIR}/lib -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl")
-  set (CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} ${MKL_INC} ${MKL_LIB}")
-  ADD_DEFINITIONS(-DMKL)
-  message (STATUS "Enable Intel MKL support")
-else(MKL_DIR)
-  message (STATUS "Disable Intel MKL support")
-endif(MKL_DIR)
-
 #------------------------------------------------------------------------------
 # Set netCDF flags for preprocessor, compiler and linker (if defined)
 # Legacy settings for old make build

ccpp/config/ccpp_prebuild_config.py

@@ -159,6 +159,7 @@
     'ccpp/physics/physics/mp_thompson_pre.F90',
     'ccpp/physics/physics/mp_thompson.F90',
     'ccpp/physics/physics/mp_thompson_post.F90',
+    'ccpp/physics/physics/mp_nsslg.F90',
     'ccpp/physics/physics/ozphys.f',
     'ccpp/physics/physics/ozphys_2015.f',
     'ccpp/physics/physics/precpd.f',
@@ -301,6 +302,13 @@
             'rime_factor',
             ],
         },
+    'mp_nsslg' : {
+         'mp_nsslg_run' : [
+           'effective_radius_of_stratiform_cloud_liquid_water_particle_in_um',
+           'effective_radius_of_stratiform_cloud_ice_particle_in_um',
+           'effective_radius_of_stratiform_cloud_snow_particle_in_um',
+             ],
+         },
     'rrtmgp_sw_rte' : {
          'rrtmgp_sw_rte_run' : [
              'components_of_surface_downward_shortwave_fluxes',