cam6_3_079: Fix problems with irreproducible results using variable resolution grids

[cacraigucar]

Closes #631

[nusbaume]

Should this PR be taken out of draft mode? Also should this PR close issue #665 as well?

[fvitt]

Should use tag ccs_config_cesm0.0.45. The ccs_config_cesm0.0.44 tag includes the -mismatch nag compiler flag for mpi-serial builds which we don't want. This was reverted in tag ccs_config_cesm0.0.45.

[cacraigucar]

done

[cacraigucar]

Actually, I am not bringing in the updated grids - that will come in in a later tag, so I removed the reference to it

[cacraigucar]

I have been having repeated attempts to get the test:  ERP_Ln9_Vnuopc.C96_C96_mg17.F2000climo.cheyenne_intel.cam-outfrq9s_mg3to run successfully using the testlist_cam.xml setup.

Here is a summary of what I've tried:

• When I run using test_driver.sh (and the corresponding create_test), it has always failed.  This is 2 or 3 times using aux_cam and 3 or 4 times using a "test_cac" which I setup for just this one test.  Sometimes it breaks in run and sometimes it makes it to case2run and breaks there.  An example run is: /glade/scratch/cacraig/test_cac_20221013095258/ERP_Ln9_Vnuopc.C96_C96_mg17.F2000climo.cheyenne_intel.cam-outfrq9s_mg3.GC.test_cac_20221013095258/run
• When I run using ./create_test ERP_Ln9_Vnuopc.C96_C96_mg17.F2000climo.cheyenne_intel.cam-outfrq9s_mg3 it works fine.  I've done this at least twice
• In order to get a baseline I can store I ran the following command which ran successfully (basically the command that test_driver.sh generates and uses the testname instead of getting it via aux_cam):qcmd -- ./create_test --xml-machine cheyenne --retry 2 --xml-compiler intel --queue regular --test-root /glade/scratch/cacraig/test_cac_try01 --output-root /glade/scratch/cacraig/test_cac_try01 --project P93300606 --test-id test_cac_try01 --compare /glade/p/cesm/amwg/cesm_baselines/cam6_3_078 --generate /glade/scratch/cacraig/test_cac_try01/baselines ERP_Ln9_Vnuopc.C96_C96_mg17.F2000climo.cheyenne_intel.cam-outfrq9s_mg3
• The full testsuite is: /glade/scratch/cacraig/aux_cam_20221011212558. This has the failing test, but everything else passing

[fischer-ncar]

@cacraigucar, the test being run in /glade/scratch/cacraig/aux_cam_20221011212558 is exceeding the wallclock of 10 minutes.
The test your running in /glade/scratch/cacraig/test_cac_try01 is using a wallclock of 12 hours. You can look at the test.ERP* files
in your run directory to see the "PBS: job killed: walltime 689 exceeded limit 600" message.

[cacraigucar]

@fischer-ncar - Thanks for the information. I always look at the TestStatus.log file and that information is not there. I'll make sure to add looking at the test.ERP* files as well. I've never had a job run out of time on me (at least that I am aware of).