Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UFS-dev PR#69 #107

Merged
merged 13 commits into from
Jan 11, 2024
Merged

UFS-dev PR#69 #107

merged 13 commits into from
Jan 11, 2024

Conversation

grantfirl
Copy link
Collaborator

@grantfirl grantfirl commented Nov 2, 2023

Identical to ufs-community#1734

Includes changes from:
ufs-community#1811 (no BL changes)
ufs-community#1794 (BL changes expected): The UPP output grib2 files for global and regional FV3, such as datasets GFSPRS, NATLEV, PRSLEV due to upp upgrade.
ufs-community#1825 (no BL changes)
ufs-community#1831 (BL changes expected): control_CubedSphereGrid_parallel
ufs-community#1842 (no BL changes)
ufs-community#1745 (BL changes expected): All tests with GOCART will have gocart output files changed.
All tests will have attributes data type change in the atm history files.
ufs-community#1807 (BL changes expected): Cpld and hafs test change baselines because sign for the sensible, evaporative, latent and momentum within the mediator files are different.
ufs-community#1864 (BL changes expected): all coupled tests include DATM

MatthewMasarik-NOAA and others added 11 commits July 26, 2023 08:59
* Add new BL_DATE

Update inline post with:

*Clean up legacy code in inline post interface

*Relocate computation of diagnostic variables

*Sync changes from offline post interface

*Update upp revision
* Update empty_output_bundle

* Update pull_request_template.md from 1813

Co-authored-by: Brian Curtis <[email protected]>
Co-authored-by: Dusan Jovic <[email protected]>
* Update control_CubedSphereGrid_parallel test

* The control_CubedSphereGrid_parallel test has been updated to run post
and create native grid history files
* reverting the bug fix for ktherm=2 allows all cpld tests to pass
and the single datm test using ktherm=2 (datm_cdeps_gfs) to also pass
all other datm tests which use ktherm=1 fail

* update CICE

* change freq in for ice_diag.d global values

* remove unused history* settings

* update DISKNM with epic and rt log

* Update bl.py for new Hera blstore
* add gridtype to gocart CAP.rc

* move to GOCART 20230227 version with threading capability

* udpate to netcdf/4.9.2 and add threading capability for gocart

* update esmf library

* update GOCART to allow no-Nitrates run
…r CMEPS (ufs-community#1807)

* commit changes for sensible heat flux

* add sensible heat changes

* update submodules and fd for evap fields

* backup to pre-evap changes to test
…s-community#1864)

* modify mom6_files.cmake to reflect files being removed/added
* update FV3 submodule pointer after GT combined his physics PRs
@grantfirl
Copy link
Collaborator Author

grantfirl commented Nov 2, 2023

Expected RT failures for #107:
001 cpld_control_p8_mixedmode_intel failed in check_result cpld_control_p8_mixedmode_intel 001 failed in run_test 002 cpld_control_gfsv17_intel failed in check_result cpld_control_gfsv17_intel 002 failed in run_test 003 cpld_control_p8_intel failed in check_result cpld_control_p8_intel 003 failed in run_test 005 cpld_control_qr_p8_intel failed in check_result cpld_control_qr_p8_intel 005 failed in run_test 007 cpld_2threads_p8_intel failed in check_result cpld_2threads_p8_intel 007 failed in run_test 008 cpld_decomp_p8_intel failed in check_result cpld_decomp_p8_intel 008 failed in run_test 009 cpld_mpi_p8_intel failed in check_result cpld_mpi_p8_intel 009 failed in run_test 010 cpld_control_ciceC_p8_intel failed in check_result cpld_control_ciceC_p8_intel 010 failed in run_test 011 cpld_control_c192_p8_intel failed in check_result cpld_control_c192_p8_intel 011 failed in run_test 013 cpld_bmark_p8_intel failed in check_result cpld_bmark_p8_intel 013 failed in run_test 015 cpld_control_noaero_p8_intel failed in check_result cpld_control_noaero_p8_intel 015 failed in run_test 016 cpld_control_nowave_noaero_p8_intel failed in check_result cpld_control_nowave_noaero_p8_intel 016 failed in run_test 017 cpld_debug_p8_intel failed in check_result cpld_debug_p8_intel 017 failed in run_test 018 cpld_debug_noaero_p8_intel failed in check_result cpld_debug_noaero_p8_intel 018 failed in run_test 019 cpld_control_noaero_p8_agrid_intel failed in check_result cpld_control_noaero_p8_agrid_intel 019 failed in run_test 020 cpld_control_c48_intel failed in check_result cpld_control_c48_intel 020 failed in run_test 021 cpld_control_p8_faster_intel failed in check_result cpld_control_p8_faster_intel 021 failed in run_test 022 cpld_control_pdlib_p8_intel failed in check_result cpld_control_pdlib_p8_intel 022 failed in run_test 025 cpld_debug_pdlib_p8_intel failed in check_result cpld_debug_pdlib_p8_intel 025 failed in run_test 040 control_p8_intel failed in check_result control_p8_intel 040 failed in run_test 042 control_qr_p8_intel failed in check_result control_qr_p8_intel 042 failed in run_test 044 control_decomp_p8_intel failed in check_result control_decomp_p8_intel 044 failed in run_test 045 control_2threads_p8_intel failed in check_result control_2threads_p8_intel 045 failed in run_test 046 control_p8_lndp_intel failed in check_result control_p8_lndp_intel 046 failed in run_test 047 control_p8_rrtmgp_intel failed in check_result control_p8_rrtmgp_intel 047 failed in run_test 048 control_p8_mynn_intel failed in check_result control_p8_mynn_intel 048 failed in run_test 049 merra2_thompson_intel failed in check_result merra2_thompson_intel 049 failed in run_test 050 regional_control_intel failed in check_result regional_control_intel 050 failed in run_test 052 regional_control_qr_intel failed in check_result regional_control_qr_intel 052 failed in run_test 054 regional_decomp_intel failed in check_result regional_decomp_intel 054 failed in run_test 055 regional_2threads_intel failed in check_result regional_2threads_intel 055 failed in run_test 056 regional_noquilt_intel failed in check_result regional_noquilt_intel 056 failed in run_test 057 regional_netcdf_parallel_intel failed in check_result regional_netcdf_parallel_intel 057 failed in run_test 058 regional_2dwrtdecomp_intel failed in check_result regional_2dwrtdecomp_intel 058 failed in run_test 060 rap_control_intel failed in check_result rap_control_intel 060 failed in run_test 061 regional_spp_sppt_shum_skeb_intel failed in check_result regional_spp_sppt_shum_skeb_intel 061 failed in run_test 062 rap_decomp_intel failed in check_result rap_decomp_intel 062 failed in run_test 063 rap_2threads_intel failed in check_result rap_2threads_intel 063 failed in run_test 065 rap_sfcdiff_intel failed in check_result rap_sfcdiff_intel 065 failed in run_test 066 rap_sfcdiff_decomp_intel failed in check_result rap_sfcdiff_decomp_intel 066 failed in run_test 068 hrrr_control_intel failed in check_result hrrr_control_intel 068 failed in run_test 069 hrrr_control_qr_intel failed in check_result hrrr_control_qr_intel 069 failed in run_test 070 hrrr_control_decomp_intel failed in check_result hrrr_control_decomp_intel 070 failed in run_test 071 hrrr_control_2threads_intel failed in check_result hrrr_control_2threads_intel 071 failed in run_test 074 rrfs_v1beta_intel failed in check_result rrfs_v1beta_intel 074 failed in run_test 077 rrfs_smoke_conus13km_hrrr_warm_intel failed in check_result rrfs_smoke_conus13km_hrrr_warm_intel 077 failed in run_test 078 rrfs_smoke_conus13km_hrrr_warm_qr_intel failed in check_result rrfs_smoke_conus13km_hrrr_warm_qr_intel 078 failed in run_test 079 rrfs_smoke_conus13km_hrrr_warm_2threads_intel failed in check_result rrfs_smoke_conus13km_hrrr_warm_2threads_intel 079 failed in run_test 080 rrfs_conus13km_hrrr_warm_intel failed in check_result rrfs_conus13km_hrrr_warm_intel 080 failed in run_test 081 rrfs_smoke_conus13km_radar_tten_warm_intel failed in check_result rrfs_smoke_conus13km_radar_tten_warm_intel 081 failed in run_test 088 control_p8_faster_intel failed in check_result control_p8_faster_intel 088 failed in run_test 089 regional_control_faster_intel failed in check_result regional_control_faster_intel 089 failed in run_test 090 rrfs_smoke_conus13km_hrrr_warm_debug_intel failed in check_result rrfs_smoke_conus13km_hrrr_warm_debug_intel 090 failed in run_test 091 rrfs_smoke_conus13km_hrrr_warm_debug_2threads_intel failed in check_result rrfs_smoke_conus13km_hrrr_warm_debug_2threads_intel 091 failed in run_test 092 rrfs_conus13km_hrrr_warm_debug_intel failed in check_result rrfs_conus13km_hrrr_warm_debug_intel 092 failed in run_test 101 control_debug_p8_intel failed in check_result control_debug_p8_intel 101 failed in run_test 102 regional_debug_intel failed in check_result regional_debug_intel 102 failed in run_test 103 rap_control_debug_intel failed in check_result rap_control_debug_intel 103 failed in run_test 104 hrrr_control_debug_intel failed in check_result hrrr_control_debug_intel 104 failed in run_test 105 rap_unified_drag_suite_debug_intel failed in check_result rap_unified_drag_suite_debug_intel 105 failed in run_test 106 rap_diag_debug_intel failed in check_result rap_diag_debug_intel 106 failed in run_test 107 rap_cires_ugwp_debug_intel failed in check_result rap_cires_ugwp_debug_intel 107 failed in run_test 108 rap_unified_ugwp_debug_intel failed in check_result rap_unified_ugwp_debug_intel 108 failed in run_test 109 rap_lndp_debug_intel failed in check_result rap_lndp_debug_intel 109 failed in run_test 110 rap_progcld_thompson_debug_intel failed in check_result rap_progcld_thompson_debug_intel 110 failed in run_test 111 rap_noah_debug_intel failed in check_result rap_noah_debug_intel 111 failed in run_test 112 rap_sfcdiff_debug_intel failed in check_result rap_sfcdiff_debug_intel 112 failed in run_test 113 rap_noah_sfcdiff_cires_ugwp_debug_intel failed in check_result rap_noah_sfcdiff_cires_ugwp_debug_intel 113 failed in run_test 114 rrfs_v1beta_debug_intel failed in check_result rrfs_v1beta_debug_intel 114 failed in run_test 115 rap_clm_lake_debug_intel failed in check_result rap_clm_lake_debug_intel 115 failed in run_test 116 rap_flake_debug_intel failed in check_result rap_flake_debug_intel 116 failed in run_test 118 regional_spp_sppt_shum_skeb_dyn32_phy32_intel failed in check_result regional_spp_sppt_shum_skeb_dyn32_phy32_intel 118 failed in run_test 119 rap_control_dyn32_phy32_intel failed in check_result rap_control_dyn32_phy32_intel 119 failed in run_test 120 hrrr_control_dyn32_phy32_intel failed in check_result hrrr_control_dyn32_phy32_intel 120 failed in run_test 121 hrrr_control_qr_dyn32_phy32_intel failed in check_result hrrr_control_qr_dyn32_phy32_intel 121 failed in run_test 122 rap_2threads_dyn32_phy32_intel failed in check_result rap_2threads_dyn32_phy32_intel 122 failed in run_test 123 hrrr_control_2threads_dyn32_phy32_intel failed in check_result hrrr_control_2threads_dyn32_phy32_intel 123 failed in run_test 124 hrrr_control_decomp_dyn32_phy32_intel failed in check_result hrrr_control_decomp_dyn32_phy32_intel 124 failed in run_test 128 rrfs_smoke_conus13km_fast_phy32_intel failed in check_result rrfs_smoke_conus13km_fast_phy32_intel 128 failed in run_test 129 rrfs_smoke_conus13km_fast_phy32_qr_intel failed in check_result rrfs_smoke_conus13km_fast_phy32_qr_intel 129 failed in run_test 132 rap_control_dyn64_phy32_intel failed in check_result rap_control_dyn64_phy32_intel 132 failed in run_test 133 rap_control_debug_dyn32_phy32_intel failed in check_result rap_control_debug_dyn32_phy32_intel 133 failed in run_test 134 hrrr_control_debug_dyn32_phy32_intel failed in check_result hrrr_control_debug_dyn32_phy32_intel 134 failed in run_test 135 rap_control_dyn64_phy32_debug_intel failed in check_result rap_control_dyn64_phy32_debug_intel 135 failed in run_test 137 hafs_regional_atm_thompson_gfdlsf_intel failed in check_result hafs_regional_atm_thompson_gfdlsf_intel 137 failed in run_test 175 control_p8_atmlnd_sbs_intel failed in check_result control_p8_atmlnd_sbs_intel 175 failed in run_test 176 atmwav_control_noaero_p8_intel failed in check_result atmwav_control_noaero_p8_intel 176 failed in run_test 178 atmaero_control_p8_intel failed in check_result atmaero_control_p8_intel 178 failed in run_test 179 atmaero_control_p8_rad_intel failed in check_result atmaero_control_p8_rad_intel 179 failed in run_test 180 atmaero_control_p8_rad_micro_intel failed in check_result atmaero_control_p8_rad_micro_intel 180 failed in run_test 187 control_p8_gnu failed in check_result control_p8_gnu 187 failed in run_test 189 rap_control_gnu failed in check_result rap_control_gnu 189 failed in run_test 190 rap_decomp_gnu failed in check_result rap_decomp_gnu 190 failed in run_test 191 rap_2threads_gnu failed in check_result rap_2threads_gnu 191 failed in run_test 193 rap_sfcdiff_gnu failed in check_result rap_sfcdiff_gnu 193 failed in run_test 194 rap_sfcdiff_decomp_gnu failed in check_result rap_sfcdiff_decomp_gnu 194 failed in run_test 196 hrrr_control_gnu failed in check_result hrrr_control_gnu 196 failed in run_test 197 hrrr_control_qr_gnu failed in check_result hrrr_control_qr_gnu 197 failed in run_test 198 hrrr_control_2threads_gnu failed in check_result hrrr_control_2threads_gnu 198 failed in run_test 199 hrrr_control_decomp_gnu failed in check_result hrrr_control_decomp_gnu 199 failed in run_test 202 rrfs_v1beta_gnu failed in check_result rrfs_v1beta_gnu 202 failed in run_test 203 rrfs_smoke_conus13km_hrrr_warm_gnu failed in check_result rrfs_smoke_conus13km_hrrr_warm_gnu 203 failed in run_test 204 rrfs_smoke_conus13km_hrrr_warm_qr_gnu failed in check_result rrfs_smoke_conus13km_hrrr_warm_qr_gnu 204 failed in run_test 205 rrfs_smoke_conus13km_hrrr_warm_2threads_gnu failed in check_result rrfs_smoke_conus13km_hrrr_warm_2threads_gnu 205 failed in run_test 206 rrfs_smoke_conus13km_radar_tten_warm_gnu failed in check_result rrfs_smoke_conus13km_radar_tten_warm_gnu 206 failed in run_test 207 rrfs_conus13km_hrrr_warm_gnu failed in check_result rrfs_conus13km_hrrr_warm_gnu 207 failed in run_test 211 regional_debug_gnu failed in check_result regional_debug_gnu 211 failed in run_test 212 rap_control_debug_gnu failed in check_result rap_control_debug_gnu 212 failed in run_test 213 hrrr_control_debug_gnu failed in check_result hrrr_control_debug_gnu 213 failed in run_test 214 rap_diag_debug_gnu failed in check_result rap_diag_debug_gnu 214 failed in run_test 215 rap_noah_sfcdiff_cires_ugwp_debug_gnu failed in check_result rap_noah_sfcdiff_cires_ugwp_debug_gnu 215 failed in run_test 216 rap_progcld_thompson_debug_gnu failed in check_result rap_progcld_thompson_debug_gnu 216 failed in run_test 217 rrfs_v1beta_debug_gnu failed in check_result rrfs_v1beta_debug_gnu 217 failed in run_test 220 control_debug_p8_gnu failed in check_result control_debug_p8_gnu 220 failed in run_test 221 rrfs_smoke_conus13km_hrrr_warm_debug_gnu failed in check_result rrfs_smoke_conus13km_hrrr_warm_debug_gnu 221 failed in run_test 222 rrfs_smoke_conus13km_hrrr_warm_debug_2threads_gnu failed in check_result rrfs_smoke_conus13km_hrrr_warm_debug_2threads_gnu 222 failed in run_test 223 rrfs_conus13km_hrrr_warm_debug_gnu failed in check_result rrfs_conus13km_hrrr_warm_debug_gnu 223 failed in run_test 224 rap_flake_debug_gnu failed in check_result rap_flake_debug_gnu 224 failed in run_test 225 rap_clm_lake_debug_gnu failed in check_result rap_clm_lake_debug_gnu 225 failed in run_test 227 rap_control_dyn32_phy32_gnu failed in check_result rap_control_dyn32_phy32_gnu 227 failed in run_test 228 hrrr_control_dyn32_phy32_gnu failed in check_result hrrr_control_dyn32_phy32_gnu 228 failed in run_test 229 hrrr_control_qr_dyn32_phy32_gnu failed in check_result hrrr_control_qr_dyn32_phy32_gnu 229 failed in run_test 230 rap_2threads_dyn32_phy32_gnu failed in check_result rap_2threads_dyn32_phy32_gnu 230 failed in run_test 231 hrrr_control_2threads_dyn32_phy32_gnu failed in check_result hrrr_control_2threads_dyn32_phy32_gnu 231 failed in run_test 232 hrrr_control_decomp_dyn32_phy32_gnu failed in check_result hrrr_control_decomp_dyn32_phy32_gnu 232 failed in run_test 236 rrfs_smoke_conus13km_phy32_gnu failed in check_result rrfs_smoke_conus13km_phy32_gnu 236 failed in run_test 237 rrfs_smoke_conus13km_phy32_qr_gnu failed in check_result rrfs_smoke_conus13km_phy32_qr_gnu 237 failed in run_test 240 rap_control_dyn64_phy32_gnu failed in check_result rap_control_dyn64_phy32_gnu 240 failed in run_test 241 rap_control_debug_dyn32_phy32_gnu failed in check_result rap_control_debug_dyn32_phy32_gnu 241 failed in run_test 242 hrrr_control_debug_dyn32_phy32_gnu failed in check_result hrrr_control_debug_dyn32_phy32_gnu 242 failed in run_test 243 rap_control_dyn64_phy32_debug_gnu failed in check_result rap_control_dyn64_phy32_debug_gnu 243 failed in run_test 244 cpld_control_p8_gnu failed in check_result cpld_control_p8_gnu 244 failed in run_test 245 cpld_control_nowave_noaero_p8_gnu failed in check_result cpld_control_nowave_noaero_p8_gnu 245 failed in run_test 246 cpld_debug_p8_gnu failed in check_result cpld_debug_p8_gnu 246 failed in run_test 247 cpld_control_pdlib_p8_gnu failed in check_result cpld_control_pdlib_p8_gnu 247 failed in run_test 248 cpld_debug_pdlib_p8_gnu failed in check_result cpld_debug_pdlib_p8_gnu 248 failed in run_test

@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: intel
Job: RT
[RT] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model//run//1584888755/20231102192008/ufs-weather-model
[RT] Error: Test 001 cpld_control_p8_mixedmode_intel FAIL Tries: 2
[RT] Error: Test 002 cpld_control_gfsv17_intel FAIL Tries: 2
[RT] Error: Test 003 cpld_control_p8_intel FAIL Tries: 2
[RT] Error: Test 005 cpld_control_qr_p8_intel FAIL Tries: 2
[RT] Error: Test 007 cpld_2threads_p8_intel FAIL Tries: 2
[RT] Error: Test 008 cpld_decomp_p8_intel FAIL Tries: 2
[RT] Error: Test 009 cpld_mpi_p8_intel FAIL Tries: 2
[RT] Error: Test 010 cpld_control_ciceC_p8_intel FAIL Tries: 2
[RT] Error: Test 011 cpld_control_c192_p8_intel FAIL Tries: 2
[RT] Error: Test 013 cpld_bmark_p8_intel FAIL Tries: 2
[RT] Error: Test 015 cpld_control_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 016 cpld_control_nowave_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 017 cpld_debug_p8_intel FAIL Tries: 2
[RT] Error: Test 018 cpld_debug_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 019 cpld_control_noaero_p8_agrid_intel FAIL Tries: 2
[RT] Error: Test 020 cpld_control_c48_intel FAIL Tries: 2
[RT] Error: Test 021 cpld_control_p8_faster_intel FAIL Tries: 2
[RT] Error: Test 022 cpld_control_pdlib_p8_intel FAIL Tries: 2
[RT] Error: Test 025 cpld_debug_pdlib_p8_intel FAIL Tries: 2
[RT] Error: Test 026 control_flake_intel FAIL Tries: 2
[RT] Error: Test 027 control_CubedSphereGrid_intel FAIL Tries: 2
[RT] Error: Test 028 control_CubedSphereGrid_parallel_intel FAIL Tries: 2
[RT] Error: Test 029 control_latlon_intel FAIL Tries: 2
[RT] Error: Test 030 control_wrtGauss_netcdf_parallel_intel FAIL Tries: 2
[RT] Error: Test 032 control_c192_intel FAIL Tries: 2
[RT] Error: Test 033 control_c384_intel FAIL Tries: 2
[RT] Error: Test 034 control_c384gdas_intel FAIL Tries: 2
[RT] Error: Test 035 control_stochy_intel FAIL Tries: 2
[RT] Error: Test 037 control_lndp_intel FAIL Tries: 2
[RT] Error: Test 038 control_iovr4_intel FAIL Tries: 2
[RT] Error: Test 039 control_iovr5_intel FAIL Tries: 2
[RT] Error: Test 040 control_p8_intel FAIL Tries: 2
[RT] Error: Test 042 control_qr_p8_intel FAIL Tries: 2
[RT] Error: Test 044 control_decomp_p8_intel FAIL Tries: 2
[RT] Error: Test 045 control_2threads_p8_intel FAIL Tries: 2
[RT] Error: Test 046 control_p8_lndp_intel FAIL Tries: 2
[RT] Error: Test 047 control_p8_rrtmgp_intel FAIL Tries: 2
[RT] Error: Test 048 control_p8_mynn_intel FAIL Tries: 2
[RT] Error: Test 049 merra2_thompson_intel FAIL Tries: 2
[RT] Error: Test 050 regional_control_intel FAIL Tries: 2
[RT] Error: Test 052 regional_control_qr_intel FAIL Tries: 2
[RT] Error: Test 054 regional_decomp_intel FAIL Tries: 2
[RT] Error: Test 055 regional_2threads_intel FAIL Tries: 2
[RT] Error: Test 056 regional_noquilt_intel FAIL Tries: 2
[RT] Error: Test 057 regional_netcdf_parallel_intel FAIL Tries: 2
[RT] Error: Test 058 regional_2dwrtdecomp_intel FAIL Tries: 2
[RT] Error: Test 059 regional_wofs_intel FAIL Tries: 2
[RT] Error: Test 060 rap_control_intel FAIL Tries: 2
[RT] Error: Test 061 regional_spp_sppt_shum_skeb_intel FAIL Tries: 2
[RT] Error: Test 062 rap_decomp_intel FAIL Tries: 2
[RT] Error: Test 063 rap_2threads_intel FAIL Tries: 2
[RT] Error: Test 065 rap_sfcdiff_intel FAIL Tries: 2
[RT] Error: Test 066 rap_sfcdiff_decomp_intel FAIL Tries: 2
[RT] Error: Test 068 hrrr_control_intel FAIL Tries: 2
[RT] Error: Test 069 hrrr_control_qr_intel FAIL Tries: 2
[RT] Error: Test 070 hrrr_control_decomp_intel FAIL Tries: 2
[RT] Error: Test 071 hrrr_control_2threads_intel FAIL Tries: 2
[RT] Error: Test 074 rrfs_v1beta_intel FAIL Tries: 2
[RT] Error: Test 075 rrfs_v1nssl_intel FAIL Tries: 2
[RT] Error: Test 076 rrfs_v1nssl_nohailnoccn_intel FAIL Tries: 2
[RT] Error: Test 077 rrfs_smoke_conus13km_hrrr_warm_intel FAIL Tries: 2
[RT] Error: Test 078 rrfs_smoke_conus13km_hrrr_warm_qr_intel FAIL Tries: 2
[RT] Error: Test 079 rrfs_smoke_conus13km_hrrr_warm_2threads_intel FAIL Tries: 2
[RT] Error: Test 080 rrfs_conus13km_hrrr_warm_intel FAIL Tries: 2
[RT] Error: Test 081 rrfs_smoke_conus13km_radar_tten_warm_intel FAIL Tries: 2
[RT] Error: Test 084 control_csawmg_intel FAIL Tries: 2
[RT] Error: Test 085 control_csawmgt_intel FAIL Tries: 2
[RT] Error: Test 086 control_ras_intel FAIL Tries: 2
[RT] Error: Test 088 control_p8_faster_intel FAIL Tries: 2
[RT] Error: Test 089 regional_control_faster_intel FAIL Tries: 2
[RT] Error: Test 090 rrfs_smoke_conus13km_hrrr_warm_debug_intel FAIL Tries: 2
[RT] Error: Test 091 rrfs_smoke_conus13km_hrrr_warm_debug_2threads_intel FAIL Tries: 2
[RT] Error: Test 092 rrfs_conus13km_hrrr_warm_debug_intel FAIL Tries: 2
[RT] Error: Test 093 control_CubedSphereGrid_debug_intel FAIL Tries: 2
[RT] Error: Test 101 control_debug_p8_intel FAIL Tries: 2
[RT] Error: Test 102 regional_debug_intel FAIL Tries: 2
[RT] Error: Test 103 rap_control_debug_intel FAIL Tries: 2
[RT] Error: Test 104 hrrr_control_debug_intel FAIL Tries: 2
[RT] Error: Test 105 rap_unified_drag_suite_debug_intel FAIL Tries: 2
[RT] Error: Test 106 rap_diag_debug_intel FAIL Tries: 2
[RT] Error: Test 107 rap_cires_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 108 rap_unified_ugwp_debug_intel FAIL Tries: 2
[RT] Error: Test 109 rap_lndp_debug_intel FAIL Tries: 2
[RT] Error: Test 110 rap_progcld_thompson_debug_intel FAIL Tries: 2
[RT] Error: Test 111 rap_noah_debug_intel FAIL Tries: 2
[RT] Error: Test 112 rap_sfcdiff_debug_intel FAIL Tries: 2
[RT] Error: Test 114 rrfs_v1beta_debug_intel FAIL Tries: 2
[RT] Error: Test 115 rap_clm_lake_debug_intel FAIL Tries: 2
[RT] Error: Test 116 rap_flake_debug_intel FAIL Tries: 2
[RT] Error: Test 118 regional_spp_sppt_shum_skeb_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 119 rap_control_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 120 hrrr_control_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 121 hrrr_control_qr_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 122 rap_2threads_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 123 hrrr_control_2threads_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 124 hrrr_control_decomp_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 128 rrfs_smoke_conus13km_fast_phy32_intel FAIL Tries: 2
[RT] Error: Test 129 rrfs_smoke_conus13km_fast_phy32_qr_intel FAIL Tries: 2
[RT] Error: Test 132 rap_control_dyn64_phy32_intel FAIL Tries: 2
[RT] Error: Test 133 rap_control_debug_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 134 hrrr_control_debug_dyn32_phy32_intel FAIL Tries: 2
[RT] Error: Test 135 rap_control_dyn64_phy32_debug_intel FAIL Tries: 2
[RT] Error: Test 136 hafs_regional_atm_intel FAIL Tries: 2
[RT] Error: Test 137 hafs_regional_atm_thompson_gfdlsf_intel FAIL Tries: 2
[RT] Error: Test 138 hafs_regional_atm_ocn_intel FAIL Tries: 2
[RT] Error: Test 140 hafs_regional_atm_ocn_wav_intel FAIL Tries: 2
[RT] Error: Test 146 hafs_global_multiple_4nests_atm_intel FAIL Tries: 2
[RT] Error: Test 147 hafs_global_multiple_4nests_atm_qr_intel FAIL Tries: 2
[RT] Error: Test 148 hafs_regional_specified_moving_1nest_atm_intel FAIL Tries: 2
[RT] Error: Test 155 hafs_regional_docn_intel FAIL Tries: 2
[RT] Error: Test 156 hafs_regional_docn_oisst_intel FAIL Tries: 2
[RT] Error: Test 157 hafs_regional_datm_cdeps_intel FAIL Tries: 2
[RT] Error: Test 158 datm_cdeps_control_cfsr_intel FAIL Tries: 2
[RT] Error: Test 160 datm_cdeps_control_gefs_intel FAIL Tries: 2
[RT] Error: Test 161 datm_cdeps_iau_gefs_intel FAIL Tries: 2
[RT] Error: Test 162 datm_cdeps_stochy_gefs_intel FAIL Tries: 2
[RT] Error: Test 163 datm_cdeps_ciceC_cfsr_intel FAIL Tries: 2
[RT] Error: Test 164 datm_cdeps_bulk_cfsr_intel FAIL Tries: 2
[RT] Error: Test 165 datm_cdeps_bulk_gefs_intel FAIL Tries: 2
[RT] Error: Test 166 datm_cdeps_mx025_cfsr_intel FAIL Tries: 2
[RT] Error: Test 167 datm_cdeps_mx025_gefs_intel FAIL Tries: 2
[RT] Error: Test 169 datm_cdeps_3072x1536_cfsr_intel FAIL Tries: 2
[RT] Error: Test 170 datm_cdeps_gfs_intel FAIL Tries: 2
[RT] Error: Test 171 datm_cdeps_debug_cfsr_intel FAIL Tries: 2
[RT] Error: Test 172 datm_cdeps_control_cfsr_faster_intel FAIL Tries: 2
[RT] Error: Test 175 control_p8_atmlnd_sbs_intel FAIL Tries: 2
[RT] Error: Test 176 atmwav_control_noaero_p8_intel FAIL Tries: 2
[RT] Error: Test 177 control_atmwav_intel FAIL Tries: 2
[RT] Error: Test 178 atmaero_control_p8_intel FAIL Tries: 2
[RT] Error: Test 179 atmaero_control_p8_rad_intel FAIL Tries: 2
[RT] Error: Test 180 atmaero_control_p8_rad_micro_intel FAIL Tries: 2
[RT] Error: Test 185 control_stochy_gnu FAIL Tries: 2
[RT] Error: Test 186 control_ras_gnu FAIL Tries: 2
[RT] Error: Test 187 control_p8_gnu FAIL Tries: 2
[RT] Error: Test 188 control_flake_gnu FAIL Tries: 2
[RT] Error: Test 189 rap_control_gnu FAIL Tries: 2
[RT] Error: Test 190 rap_decomp_gnu FAIL Tries: 2
[RT] Error: Test 191 rap_2threads_gnu FAIL Tries: 2
[RT] Error: Test 193 rap_sfcdiff_gnu FAIL Tries: 2
[RT] Error: Test 194 rap_sfcdiff_decomp_gnu FAIL Tries: 2
[RT] Error: Test 196 hrrr_control_gnu FAIL Tries: 2
[RT] Error: Test 197 hrrr_control_qr_gnu FAIL Tries: 2
[RT] Error: Test 198 hrrr_control_2threads_gnu FAIL Tries: 2
[RT] Error: Test 199 hrrr_control_decomp_gnu FAIL Tries: 2
[RT] Error: Test 202 rrfs_v1beta_gnu FAIL Tries: 2
[RT] Error: Test 203 rrfs_smoke_conus13km_hrrr_warm_gnu FAIL Tries: 2
[RT] Error: Test 204 rrfs_smoke_conus13km_hrrr_warm_qr_gnu FAIL Tries: 2
[RT] Error: Test 205 rrfs_smoke_conus13km_hrrr_warm_2threads_gnu FAIL Tries: 2
[RT] Error: Test 206 rrfs_smoke_conus13km_radar_tten_warm_gnu FAIL Tries: 2
[RT] Error: Test 207 rrfs_conus13km_hrrr_warm_gnu FAIL Tries: 2
[RT] Error: Test 211 regional_debug_gnu FAIL Tries: 2
[RT] Error: Test 212 rap_control_debug_gnu FAIL Tries: 2
[RT] Error: Test 213 hrrr_control_debug_gnu FAIL Tries: 2
[RT] Error: Test 214 rap_diag_debug_gnu FAIL Tries: 2
[RT] Error: Test 215 rap_noah_sfcdiff_cires_ugwp_debug_gnu FAIL Tries: 2
[RT] Error: Test 216 rap_progcld_thompson_debug_gnu FAIL Tries: 2
[RT] Error: Test 217 rrfs_v1beta_debug_gnu FAIL Tries: 2
[RT] Error: Test 220 control_debug_p8_gnu FAIL Tries: 2
[RT] Error: Test 221 rrfs_smoke_conus13km_hrrr_warm_debug_gnu FAIL Tries: 2
[RT] Error: Test 222 rrfs_smoke_conus13km_hrrr_warm_debug_2threads_gnu FAIL Tries: 2
[RT] Error: Test 223 rrfs_conus13km_hrrr_warm_debug_gnu FAIL Tries: 2
[RT] Error: Test 224 rap_flake_debug_gnu FAIL Tries: 2
[RT] Error: Test 225 rap_clm_lake_debug_gnu FAIL Tries: 2
[RT] Error: Test 227 rap_control_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 228 hrrr_control_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 229 hrrr_control_qr_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 230 rap_2threads_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 231 hrrr_control_2threads_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 232 hrrr_control_decomp_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 236 rrfs_smoke_conus13km_phy32_gnu FAIL Tries: 2
[RT] Error: Test 237 rrfs_smoke_conus13km_phy32_qr_gnu FAIL Tries: 2
[RT] Error: Test 240 rap_control_dyn64_phy32_gnu FAIL Tries: 2
[RT] Error: Test 241 rap_control_debug_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 242 hrrr_control_debug_dyn32_phy32_gnu FAIL Tries: 2
[RT] Error: Test 243 rap_control_dyn64_phy32_debug_gnu FAIL Tries: 2
[RT] Error: Test 244 cpld_control_p8_gnu FAIL
[RT] Error: Test 244 cpld_control_p8_gnu FAIL
[RT] Error: Test 245 cpld_control_nowave_noaero_p8_gnu FAIL Tries: 2
[RT] Error: Test 246 cpld_debug_p8_gnu FAIL Tries: 2
[RT] Error: Test 247 cpld_control_pdlib_p8_gnu FAIL Tries: 2
[RT] Error: Test 248 cpld_debug_pdlib_p8_gnu FAIL Tries: 2
[RT] Error: Test 249 datm_cdeps_control_cfsr_gnu FAIL Tries: 2
[RT] Log file shows failures.
[RT] Please obtain logs from /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model//run//1584888755/20231102192008/ufs-weather-model

@grantfirl
Copy link
Collaborator Author

@mkavulich I've gone through the list of reported failures and have the following list of failures that were not expected for the parent ufs/dev PR (ufs-community#1734):
control_flake_intel
control_CubedSphereGrid_intel - 1831
control_CubedSphereGrid_parallel_intel
control_latlon_intel
control_wrtGauss_netcdf_parallel_intel
control_c192_intel
control_c384_intel
control_c384gdas_intel
control_stochy_intel
control_lndp_intel
control_iovr4_intel
control_iovr5_intel
regional_wofs_intel
rrfs_v1nssl_intel
rrfs_v1nssl_nohailnoccn_intel
control_csawmg_intel
control_csawmgt_intel
control_ras_intel
control_CubedSphereGrid_debug_intel
hafs_regional_atm_intel
hafs_regional_atm_ocn_intel
hafs_regional_atm_ocn_wav_intel
hafs_global_multiple_4nests_atm_intel
hafs_global_multiple_4nests_atm_qr_intel
hafs_regional_specified_moving_1nest_atm_intel
hafs_regional_docn_intel
hafs_regional_docn_oisst_intel
hafs_regional_datm_cdeps_intel
datm*
control_atmwav_intel
control_stochy_gnu
control_ras_gnu
control_flake_gnu

I'm going to go through this list and see which other included ufs-wm PRs could cause these failures. Assuming that the failures can be explained, I'll go ahead and add the BL creation tags.

@grantfirl
Copy link
Collaborator Author

I've whittled the unexplained failures down. This is now the list with the UFS-WM PR that causes the failure listed afterward:
control_flake_intel - 1794
control_CubedSphereGrid_intel - 1831
control_CubedSphereGrid_parallel_intel - 1831
control_latlon_intel - 1794
control_wrtGauss_netcdf_parallel_intel - 1794
control_c192_intel - 1794
control_c384_intel - 1794
control_c384gdas_intel - 1794
control_stochy_intel - 1794
control_lndp_intel - 1794
control_iovr4_intel - 1794
control_iovr5_intel - 1794
regional_wofs_intel - 1794
rrfs_v1nssl_intel - 1794
rrfs_v1nssl_nohailnoccn_intel - 1794
control_csawmg_intel - 1794
control_csawmgt_intel - 1794
control_ras_intel - 1794
control_CubedSphereGrid_debug_intel - 1831
hafs_regional_atm_intel - 1794
hafs_regional_atm_ocn_intel - 1794
hafs_regional_atm_ocn_wav_intel
hafs_global_multiple_4nests_atm_intel - 1794
hafs_global_multiple_4nests_atm_qr_intel - 1794
hafs_regional_specified_moving_1nest_atm_intel - 1794
hafs_regional_docn_intel - 1807
hafs_regional_docn_oisst_intel - 1807
hafs_regional_datm_cdeps_intel - 1807
datm* - 1864
control_atmwav_intel
control_stochy_gnu
control_ras_gnu
control_flake_gnu

I'm hypothesizing that the remaining unexplained test failures are also the result of 1794.

@grantfirl
Copy link
Collaborator Author

@mkavulich OK, I think that I've solved the mystery of the final failing tests. The RT log for 1794 was only attached for WCOSS. All remaining failed tests are not conducted on WCOSS for the code as of this date. Since the tests that fail in 1794 on WCOSS are all similar to the remaining unexplained failed tests in this PR, I feel confident that the tests are supposed to fail for 1794. I'll start new BLs.

@grantfirl
Copy link
Collaborator Author

@mkavulich Do you know if the scripts are active on Hera to listen for the added labels?

@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: gnu
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model//run//1584888755/20240110180016/ufs-weather-model
Please make changes and add the following label back: hera-gnu-BL

@mkavulich
Copy link
Collaborator

@grantfirl It looks like they are running, the cron job only runs every 5 minutes so it can take up to that long to start a script, and only one compiler is processed at a time so it can take a while before the second label is processed.

It looks like the regtests haven't been updated with the new way of specifying compiler, which is why the GNU test is failing. I'll look into getting that working.

@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: intel
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model//run//1584888755/20240110180514/ufs-weather-model
Please make changes and add the following label back: hera-intel-BL

@mkavulich
Copy link
Collaborator

Okay... I've figured out the issue. Since there is no longer any way to specify one compiler or another, we will have to adjust the labels to just be platform-specific rather than platform and compiler specific. Re-adding the gnu label just now caused the original tests to fail due to the run directory being deleted, which is also not good. I need to think about the best way to prevent that going forward.

@grantfirl
Copy link
Collaborator Author

So, should we manually do RTs for this or are you going to try to fix the RT scripts within the context of this PR?

@mkavulich
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: gnu
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model//run//1584888755/20240110195011/ufs-weather-model
Please make changes and add the following label back: hera-gnu-BL

@grantfirl
Copy link
Collaborator Author

@mkavulich OK, so the tests ran and the log is showing the BL creation as successful (See /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/run/1584888755/20240110195011/ufs-weather-model/tests/logs/RegressionTests_hera.log). It looks like the new baselines are stored in /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/new. So, we're supposed to manually move this to /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/baselines/main-20240110 and update bl_date.ncar.conf before merging, correct?

@grantfirl
Copy link
Collaborator Author

@mkavulich I don't have permissions to move the new baselines into the right place. Could you change permissions on the directories so that I can do this, please? Otherwise, I'm ready to start the merge process.

Copy link
Collaborator

@mkavulich mkavulich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay the baseline has been staged (also just realized I hadn't approved until now, sorry about that). I also added a umask change that will hopefully give group write permissions for files written by the regtest; I thought I had done this already but I guess not.

@grantfirl
Copy link
Collaborator Author

What about permissions in /scratch1/BMC/gmtb/CCPP_regression_testing/NCAR_ufs-weather-model/baselines? For example, I still can't remove old baselines. We're supposed to keep the last 2 to save space.

@mkavulich
Copy link
Collaborator

The umask command was added as a line in the crontab entry so it should allow group read/write permissions for all files created by the regression tests going forward. I have recursively updated permissions in baselines/main-20240110 to have group read/write just now.

FYI it looks like you created baselines/main-20231031, so you'll have to update the permissions or delete that one when the time comes.

@grantfirl grantfirl merged commit 4f76537 into NCAR:main Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants