Bfb merge tag dev166

[slevis-lmwg]

Description of changes

Simple BFB merge tag:
#2315 @TeaganKing Refactoring run_neon for PLUMBER2 part1
#2326 @samsrabin run_sys_tests: Check Python environment for FatesColdTwoStream tests
#2213 @samsrabin Automatically assign high priority items to project 25
#2330 @samsrabin Add Izumi version of the aux_clm unit testing
#2124 REMOVED due to failing tests

Specific notes

Contributors other than yourself, if any:
@TeaganKing @samsrabin @ekluzek

CTSM Issues Fixed (include github issue #):
Fixes #2315
Fixes #2326
Fixes #2213
Fixes #2330

Are answers expected to change (and if so in what way)?
No

Any User Interface Changes (namelist or namelist defaults changes)?

Testing performed, if any:
#2315 "make all" testing passes for slevis-lmwg other than long list of pylint warnings (EXPECTED)
#2124 REMOVED due to failing tests (derecho and izumi test-suites)

[slevis-lmwg]

Submitting test-suites first, then will prep the ChangeLog.

Python tests: make all OK, pylint gives long list of warnings, which is expected and there's an open issue about that.
Derecho FAIL
Izumi FAIL
Problem seems to point to #2124, so I'm in communication with @adrifoster about next steps.

[slevis-lmwg]

For the record, I received an email titled
[slevis-lmwg/ctsm] Run failed: .github/workflows/assign-to-project.yml - bfb_merge_tag_2315_et_al (5fa6ecb)
stating that no jobs were run. (In case it matters.)

[ekluzek]

For the record, I received an email titled [slevis-lmwg/ctsm] Run failed: .github/workflows/assign-to-project.yml - bfb_merge_tag_2315_et_al (5fa6ecb) stating that no jobs were run. (In case it matters.)

That's #2213. That probably will just be a message we'll start to see. It also might mean we need to adjust it so that it's triggered less often. So it is good to know...

[slevis-lmwg]

@adrifoster this is the error I see in the cesm.log of test
SMS_Lm1_D.f10_f10_mg37.I1850Clm50BgcCrop.derecho_intel.clm-output_crop_highfreq

dec0959.hsn.de.hpc.ucar.edu 21: forrtl: severe (174): SIGSEGV, segmentation fault occurred
dec0959.hsn.de.hpc.ucar.edu 21: Image              PC                Routine            Line        Source
dec0959.hsn.de.hpc.ucar.edu 21: libpthread-2.31.s  00001500296E78C0  Unknown               Unknown  Unknown
dec0959.hsn.de.hpc.ucar.edu 21: cesm.exe           00000000025B9321  satellitephenolog         499  SatellitePhenologyMod.F90
dec0959.hsn.de.hpc.ucar.edu 21: cesm.exe           00000000025B1FDB  satellitephenolog         253  SatellitePhenologyMod.F90

@adrifoster and I agreed that I will revert #2124 out of this PR and test again.

[slevis-lmwg]

@samsrabin on izumi
./run_sys_tests -c ctsm5.1.dev165 -g ctsm5.1.dev166 -s aux_clm
gives
FAIL ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream CREATE_NEWCASE
with the error found in this file
ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream.GC.0124-120243iz_nag/TestStatus.log

Submitting the test manually
./create_test ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream -c /fs/cgd/csm/ccsm_baselines/ctsm5.1.dev165/ -g /fs/cgd/csm/ccsm_baselines/ctsm5.1.dev166/
passed the CREATE_NEWCASE phase successfully.

This seems related to #2326 so I need your feedback on whether there a simple fix or I should back 2326 out and try again.

[ekluzek]

@slevis-lmwg depending on how you revert #2124 here will make a difference in how we need to handle #2124. If you use git revert it will autoclose #2124 even though it's not really in. In that case we'll need @adrifoster to create a new branch and PR to keep as an outstanding draft. If you don't use git revert we should meet to talk about the trickier options. This is also something that we should decide how to handle and document it so that anyone can follow the recipe.

[samsrabin]

@slevis-lmwg I don't think this is being caused by #2326. Could you try ./run_sys_tests --skip-compare --skip-generate -t ERS_D_Ld15.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream with ctsm5.1.dev164 and see if it still fails?

[samsrabin]

I'm getting the same error you saw with ctsm5.1.dev164 and 165. So it's not caused by #2326, but it's also not fixed by it either…

[samsrabin]

If I add the following to my .bash_profile on Izumi, it works:

module load lang/python/3.7.0

I suggest doing that for now; I'll file an issue about how I think this should be permanently fixed.

[slevis-lmwg]

Thank you @samsrabin

[slevis-lmwg]

If you use git revert it will autoclose #2124 even though it's not really in. In that case we'll need @adrifoster to create a new branch and PR to keep as an outstanding draft. If you don't use git revert we should meet to talk about the trickier options. This is also something that we should decide how to handle and document it so that anyone can follow the recipe.

Thanks @ekluzek. I had already git reverted when you posted this, so I will proceed with #2334 as is and @adrifoster will likely need to replace #2124 with a new PR. Yes, we should agree on the group's accepted approach going fwd.

[slevis-lmwg]

Test-suites OK