Skip to content

Commit

Permalink
Merge branch 'jgfouca/scream_downstream_04_11_24' into master (PR #6343)
Browse files Browse the repository at this point in the history
Bring latest EAMxx and other changes from scream repo.

Non-eamxx changes:
1) Change GitHub CI to skip E3SM CI for eamxx (for now)
2) Add an eamxx-specific workflow in the CI to ensure default input files for SCREAM are available and downloadable
3) ELM config changes: new namelist defaults added for ne256pg2 and ne1024pg2 land configurations to support F20TR compsets with these resolutions
4) EAM changes: scream/micro_p3.F90 updates/fixes. Should only impact SCREAM stuff. Also, there's a fix for DP SCREAM output coordinate issue where x-coordinates should be assocated with spherep lon values and y-coordinates should be associated with lat values
5) Change kokkos build to ensure cxx17 standard

On mappy, this PR is non-BFB for these cases, which I think is expected:

DIFF ERS_D.ne4pg2_oQU480.F2010.mappy_gnu.eam-hommexx (phase BASELINE)
DIFF SMS_Ln9_P24x1.ne4_ne4.FDPSCREAM-ARM97.mappy_gnu (phase BASELINE)
DIFF SMS_R_Ld5.ne4_ne4.FSCM-ARM97.mappy_gnu.eam-scm (phase BASELINE)

[non-BFB] for HOMMExx cases.
  • Loading branch information
jgfouca committed Apr 18, 2024
2 parents de85394 + 064ac82 commit baef174
Show file tree
Hide file tree
Showing 412 changed files with 14,163 additions and 10,904 deletions.
2 changes: 2 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ updates:
reviewers:
- "mahf708"
- "bartgol"
labels:
- "AT: Integrate Without Testing"
1 change: 1 addition & 0 deletions .github/workflows/e3sm-gh-ci-cime-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ on:
jobs:

ci:
if: ${{ github.event.repository.name == 'e3sm' }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/e3sm-gh-pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ concurrency:

jobs:
Build-and-Deploy-docs:
if: ${{ github.event.repository.name != 'scream' }}
if: ${{ github.event.repository.name == 'e3sm' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down
62 changes: 62 additions & 0 deletions .github/workflows/eamxx_default_files.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: inputdata

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
schedule:
- cron: '00 00 * * *'
workflow_dispatch:

jobs:
scream-defaults:
runs-on: ubuntu-latest
outputs:
event_name: ${{ github.event_name }}
steps:
- name: Check out the repository
uses: actions/checkout@v4
with:
show-progress: false
submodules: false
- name: Set up Python 3.11
uses: actions/[email protected]
with:
python-version: "3.11"
- name: Run unit tests
working-directory: components/eamxx/cime_config/
run: |
python -m unittest tests/eamxx_default_files.py -v
notify-scream-defaults:
needs: scream-defaults
if: ${{ failure() && needs.scream-defaults.outputs.event_name != 'pull_request' }}
runs-on: ubuntu-latest
steps:
- name: Create issue
run: |
previous_issue_number=$(gh issue list \
--label "$LABELS" \
--json number \
--jq '.[0].number')
if [[ -n $previous_issue_number ]]; then
gh issue comment "$previous_issue_number" \
--body "$BODY"
else
gh issue create \
--title "$TITLE" \
--assignee "$ASSIGNEES" \
--label "$LABELS" \
--body "$BODY"
fi
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
TITLE: Inputdata server file missing
ASSIGNEES: mahf708,bartgol
LABELS: bug,input file,notify-file-gh-action
BODY: |
Workflow failed! There's likely a missing file specified in the configs! For more information, please see:
- Workflow URL: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }} (number ${{ github.run_number }}, attempt ${{ github.run_attempt }})
- Workflow SHA: ${{ github.sha }}
5 changes: 5 additions & 0 deletions cime_config/config_grids.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4522,6 +4522,11 @@
<map name="ROF2LND_FMAPNAME">lnd/clm2/mappingdata/maps/ne240np4/map_0.1x0.1_nomask_to_ne240np4_nomask_aave_da_c120706.nc</map>
</gridmap>

<gridmap lnd_grid="ne256np4.pg2" rof_grid="r0125">
<map name="LND2ROF_FMAPNAME">cpl/gridmaps/ne256pg2/map_ne256pg2_to_r0125_mono.200212.nc</map>
<map name="ROF2LND_FMAPNAME">cpl/gridmaps/ne256pg2/map_r0125_to_ne256pg2_mono.200212.nc</map>
</gridmap>

<gridmap lnd_grid="ne1024np4.pg2" rof_grid="r0125">
<map name="LND2ROF_FMAPNAME">cpl/gridmaps/ne1024pg2/map_ne1024pg2_to_r0125_mono.200212.nc</map>
<map name="ROF2LND_FMAPNAME">cpl/gridmaps/ne1024pg2/map_r0125_to_ne1024pg2_mono.200212.nc</map>
Expand Down
32 changes: 16 additions & 16 deletions cime_config/machines/config_machines.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2967,10 +2967,10 @@
<MPILIBS>spectrum-mpi</MPILIBS>
<PROJECT>cbronze</PROJECT>
<CIME_OUTPUT_ROOT>/usr/workspace/$USER/e3sm_scratch</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/usr/gdata/climdat/ccsm3data/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/gdata/climdat/ccsm3data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DIN_LOC_ROOT>/usr/gdata/e3sm/ccsm3data/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/gdata/e3sm/ccsm3data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/usr/workspace/$USER/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>/usr/gdata/climdat/baselines/$COMPILER</BASELINE_ROOT>
<BASELINE_ROOT>/usr/gdata/e3sm/baselines/$COMPILER</BASELINE_ROOT>
<GMAKE_J>16</GMAKE_J>
<BATCH_SYSTEM>lsf</BATCH_SYSTEM>
<SUPPORTED_BY>donahue5 -at- llnl.gov</SUPPORTED_BY>
Expand Down Expand Up @@ -3024,9 +3024,9 @@
</environment_variables>
<environment_variables compiler="gnugpu">
<env name="LLNL_USE_OMPI_VARS">y</env>
<env name="PATH">/usr/gdata/climdat/netcdf/bin:$ENV{PATH}</env>
<env name="LD_LIBRARY_PATH">/usr/gdata/climdat/netcdf/lib:$ENV{LD_LIBRARY_PATH}</env>
<env name="NETCDF_PATH">/usr/gdata/climdat/netcdf</env>
<env name="PATH">/usr/gdata/e3sm/netcdf/bin:$ENV{PATH}</env>
<env name="LD_LIBRARY_PATH">/usr/gdata/e3sm/netcdf/lib:$ENV{LD_LIBRARY_PATH}</env>
<env name="NETCDF_PATH">/usr/gdata/e3sm/netcdf</env>
<env name="RS_PER_NODE">2</env>
<env name="CPU_PER_RS">20</env>
<env name="GPU_PER_RS">2</env>
Expand All @@ -3043,11 +3043,11 @@
<MPILIBS>mpich</MPILIBS>
<PROJECT>cbronze</PROJECT>
<CIME_OUTPUT_ROOT>/p/lustre2/$USER/e3sm_scratch/ruby</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/usr/gdata/climdat/ccsm3data/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/gdata/climdat/ccsm3data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DIN_LOC_ROOT>/usr/gdata/e3sm/ccsm3data/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/gdata/e3sm/ccsm3data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/p/lustre2/$USER/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>/p/lustre2/$USER/ccsm_baselines/$COMPILER</BASELINE_ROOT>
<CCSM_CPRNC>/usr/gdata/climdat/tools/cprnc</CCSM_CPRNC>
<CCSM_CPRNC>/usr/gdata/e3sm/tools/cprnc</CCSM_CPRNC>
<GMAKE_J>8</GMAKE_J>
<BATCH_SYSTEM>lc_slurm</BATCH_SYSTEM>
<SUPPORTED_BY>donahue5 -at- llnl.gov</SUPPORTED_BY>
Expand Down Expand Up @@ -3075,7 +3075,7 @@
<command name="load">intel-classic/2021.6.0-magic</command>
<command name="load">mvapich2/2.3.7</command>
<command name="load">cmake/3.19.2</command>
<command name="use --append">/usr/gdata/climdat/install/quartz/modulefiles</command>
<command name="use --append">/usr/gdata/e3sm/install/quartz/modulefiles</command>
<command name="load">hdf5/1.12.2</command>
<command name="load">netcdf-c/4.9.0</command>
<command name="load">netcdf-fortran/4.6.0</command>
Expand All @@ -3086,7 +3086,7 @@
<RUNDIR>$CIME_OUTPUT_ROOT/$CASE/run</RUNDIR>
<EXEROOT>$CIME_OUTPUT_ROOT/$CASE/bld</EXEROOT>
<environment_variables compiler="intel">
<env name="NETCDF_PATH">/usr/gdata/climdat/install/quartz/netcdf-fortran/</env>
<env name="NETCDF_PATH">/usr/gdata/e3sm/install/quartz/netcdf-fortran/</env>
<env name="PNETCDF_PATH">/usr/tce/packages/parallel-netcdf/parallel-netcdf-1.12.3-mvapich2-2.3.7-intel-classic-2021.6.0</env>
</environment_variables>
</machine>
Expand All @@ -3098,11 +3098,11 @@
<MPILIBS>mpich</MPILIBS>
<PROJECT>cbronze</PROJECT>
<CIME_OUTPUT_ROOT>/p/lustre2/$USER/e3sm_scratch/quartz</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/usr/gdata/climdat/ccsm3data/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/gdata/climdat/ccsm3data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DIN_LOC_ROOT>/usr/gdata/e3sm/ccsm3data/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/gdata/e3sm/ccsm3data/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/p/lustre2/$USER/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>/p/lustre2/$USER/ccsm_baselines/$COMPILER</BASELINE_ROOT>
<CCSM_CPRNC>/usr/gdata/climdat/tools/cprnc</CCSM_CPRNC>
<CCSM_CPRNC>/usr/gdata/e3sm/tools/cprnc</CCSM_CPRNC>
<GMAKE_J>8</GMAKE_J>
<BATCH_SYSTEM>lc_slurm</BATCH_SYSTEM>
<SUPPORTED_BY>donahue5 -at- llnl.gov</SUPPORTED_BY>
Expand Down Expand Up @@ -3130,7 +3130,7 @@
<command name="load">intel-classic/2021.6.0-magic</command>
<command name="load">mvapich2/2.3.7</command>
<command name="load">cmake/3.19.2</command>
<command name="use --append">/usr/gdata/climdat/install/quartz/modulefiles</command>
<command name="use --append">/usr/gdata/e3sm/install/quartz/modulefiles</command>
<command name="load">hdf5/1.12.2</command>
<command name="load">netcdf-c/4.9.0</command>
<command name="load">netcdf-fortran/4.6.0</command>
Expand All @@ -3141,7 +3141,7 @@
<RUNDIR>$CIME_OUTPUT_ROOT/$CASE/run</RUNDIR>
<EXEROOT>$CIME_OUTPUT_ROOT/$CASE/bld</EXEROOT>
<environment_variables compiler="intel">
<env name="NETCDF_PATH">/usr/gdata/climdat/install/quartz/netcdf-fortran/</env>
<env name="NETCDF_PATH">/usr/gdata/e3sm/install/quartz/netcdf-fortran/</env>
<env name="PNETCDF_PATH">/usr/tce/packages/parallel-netcdf/parallel-netcdf-1.12.3-mvapich2-2.3.7-intel-classic-2021.6.0</env>
</environment_variables>
</machine>
Expand Down
48 changes: 31 additions & 17 deletions cime_config/tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -610,40 +610,47 @@

"e3sm_scream_v1_lowres" : {
"time" : "01:00:00",
"inherit" : ("e3sm_scream_mam4xx_v1_lowres"),
"tests" : (
"ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1",
"ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1",
"SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero",
"ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1",
"ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.scream-rad_frequency_2",
"ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.scream-output-preset-1",
"ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.scream-output-preset-2",
"SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.scream-output-preset-3",
"ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.scream-output-preset-4",
"ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.scream-rad_frequency_2--scream-output-preset-5",
)
},

"e3sm_scream_v1_dp-eamxx" : {
"time" : "01:00:00",
"tests" : (
"ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01", # 225 phys cols, roughly size of ne2
)
},

# Tests run on exclusively on mappy for scream AT testing. These tests
# should be fast, so we limit it to low res and add some thread tests
# specifically for mappy.
"e3sm_scream_v1_at" : {
"inherit" : ("e3sm_scream_v1_lowres"),
"tests" : ("PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1")
"inherit" : ("e3sm_scream_v1_lowres", "e3sm_scream_v1_dp-eamxx"),
"tests" : ("PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.scream-output-preset-1")
},

"e3sm_scream_v1_medres" : {
"time" : "02:00:00",
"tests" : (
# "SMS_D_Ln2.ne30_ne30.F2000-SCREAMv1-AQP1", # Uncomment once IC file for ne30 is ready
"ERS_Ln22.ne30_ne30.F2010-SCREAMv1.scream-internal_diagnostics_level",
"PEM_Ln90.ne30pg2_ne30pg2.F2010-SCREAMv1",
"ERS_Ln90.ne30pg2_ne30pg2.F2010-SCREAMv1.scream-small_kernels",
"ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.scream-bfbhash",
"ERS_Ln22.ne30_ne30.F2010-SCREAMv1.scream-internal_diagnostics_level--scream-output-preset-3",
"PEM_Ln90.ne30pg2_ne30pg2.F2010-SCREAMv1.scream-spa_remap--scream-output-preset-4",
"ERS_Ln90.ne30pg2_ne30pg2.F2010-SCREAMv1.scream-small_kernels--scream-output-preset-5",
"ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.scream-bfbhash--scream-output-preset-6",
)
},

# Used to track performance
"e3sm_scream_v1_hires" : {
"time" : "03:00:00",
"time" : "01:00:00",
"tests" : (
"SMS_D_Ln12.ne120_r0125_oRRS18to6v3.F2010-SCREAMv1",
"SMS_Ln12.ne120_ne120.F2010-SCREAMv1",
# "SMS_Ln12.ne120_r0125_oRRS18to6v3.F2000-SCREAMv1-AQP1", add when aquap 120 inputs available
"SMS_Ln300.ne30pg2_ne30pg2.F2010-SCREAMv1.scream-perf_test--scream-output-preset-1"
)
},

Expand All @@ -655,7 +662,7 @@
# Disable the two 111422-commented tests b/c they fail on pm-gpu and
# we're not using MPASSI right now.
#111422 "ERP_Ln22.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",
"ERS_D_Ln22.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",
"ERS_D_Ln22.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off--scream-output-preset-1",
# "ERS_Ln22.ne30_oECv3.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",
#111422 "PEM_Ln90.ne30pg2_EC30to60E2r2.F2010-SCREAMv1-MPASSI",
# "ERS_Ln22.ne30pg2_EC30to60E2r2.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",
Expand All @@ -679,6 +686,14 @@
)
},

"e3sm_scream_mam4xx_v1_lowres" : {
"time" : "01:00:00",
"tests" : (
"SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.scream-mam4xx-optics",
)
},


"e3sm_gpuacc" : {
"tests" : (
"SMS_Ld1.T62_oEC60to30v3.CMPASO-NYF",
Expand Down Expand Up @@ -1000,4 +1015,3 @@
"e3sm_superbfb_atm", "e3sm_superbfb_wcycl"),
},
}

4 changes: 2 additions & 2 deletions components/eam/src/dynamics/se/stepon.F90
Original file line number Diff line number Diff line change
Expand Up @@ -585,8 +585,8 @@ subroutine stepon_run3(dtime, cam_out, phys_state, dyn_in, dyn_out)
if (dp_crm) then

do ie=1,nelemd
out_gridx(:,:) = dyn_in%elem(ie)%spherep(:,:)%lat
out_gridy(:,:) = dyn_in%elem(ie)%spherep(:,:)%lon
out_gridx(:,:) = dyn_in%elem(ie)%spherep(:,:)%lon
out_gridy(:,:) = dyn_in%elem(ie)%spherep(:,:)%lat
call outfld('crm_grid_x', out_gridx, npsq, ie)
call outfld('crm_grid_y', out_gridy, npsq, ie)
enddo
Expand Down
13 changes: 7 additions & 6 deletions components/eam/src/physics/p3/scream/micro_p3.F90
Original file line number Diff line number Diff line change
Expand Up @@ -1846,7 +1846,8 @@ subroutine get_rain_dsd2(qr,nr,mu_r,lamr,cdistr,logn0r)
real(rtype), intent(out) :: lamr,mu_r,cdistr,logn0r

!local variables:
real(rtype) :: inv_dum,lammax,lammin
real(rtype) :: lammax,lammin
real(rtype) :: mass_to_d3_factor

!--------------------------------------------------------------------------

Expand All @@ -1858,25 +1859,25 @@ subroutine get_rain_dsd2(qr,nr,mu_r,lamr,cdistr,logn0r)
! find spot in lookup table
! (scaled N/q for lookup table parameter space_
nr = max(nr,nsmall)
inv_dum = bfb_cbrt(qr/(cons1*nr*6._rtype))

! Apply constant mu_r: Recall the switch to v4 tables means constant mu_r
mu_r = mu_r_constant
lamr = bfb_cbrt(cons1*nr*(mu_r+3._rtype)*(mu_r+2._rtype)*(mu_r+1._rtype)/(qr)) ! recalculate slope based on mu_r
mass_to_d3_factor = cons1*(mu_r+3._rtype)*(mu_r+2._rtype)*(mu_r+1._rtype)
lamr = bfb_cbrt(mass_to_d3_factor*nr/qr) ! recalculate slope based on mu_r
lammax = (mu_r+1._rtype)*1.e+5_rtype ! check for slope
lammin = (mu_r+1._rtype)*500._rtype !500=1/(2mm) is inverse of max allowed number-weighted mean raindrop diameter

! apply lambda limiters for rain
if (lamr.lt.lammin) then
lamr = lammin
nr = bfb_exp(3._rtype*bfb_log(lamr)+bfb_log(qr)+bfb_log(bfb_gamma(mu_r+1._rtype))-bfb_log(bfb_gamma(mu_r+4._rtype)))/(cons1)
nr = lamr * lamr * lamr * qr / mass_to_d3_factor
elseif (lamr.gt.lammax) then
lamr = lammax
nr = bfb_exp(3._rtype*bfb_log(lamr)+bfb_log(qr)+bfb_log(bfb_gamma(mu_r+1._rtype))-bfb_log(bfb_gamma(mu_r+4._rtype)))/(cons1)
nr = lamr * lamr * lamr * qr / mass_to_d3_factor
endif

cdistr = nr/bfb_gamma(mu_r+1._rtype)
logn0r = bfb_log10(nr)+(mu_r+1._rtype)*bfb_log10(lamr)-bfb_log10(bfb_gamma(mu_r+1._rtype)) !note: logn0r is calculated as log10(n0r)
logn0r = bfb_log10(cdistr)+(mu_r+1._rtype)*bfb_log10(lamr) !note: logn0r is calculated as log10(n0r)

else

Expand Down
Loading

0 comments on commit baef174

Please sign in to comment.