Turns on debugging for multiple land tests.

[bishtgautam]

Creates a new test suite, e3sm_land_debug, in which multiple existing land tests
are moved with debugging turned on.

DIFFs for some existing land tests and replaces some land tests with DEBUG versions
[non-BFB]

[peterdschwartz]

Is this change related to running a debug test or an open issue?

[peterdschwartz]

I don't think it should be removed if it's tangential to the debug tests -- just thinking the PR description should mention it.

[bishtgautam]

This fix is needed to allow the few BGC tests to pass when debugging is turned on.

[peterdschwartz]

Testing this PR after a merge to next, I encountered a FAIL with SMS_Ld30_D.f45_f45.IELMFATES.chrysalis_intel.elm-fates_satphen due to an FPE in NitrogenDynamicsMod.F90

Seems likely due to PR #5604 and the variable that is possibly NaN is a component of the alm_fates variable.

124: forrtl: error (65): floating invalid
124: Image              PC                Routine            Line        Source
124: libpnetcdf.so.3.0  000015555101868C  for__signal_handl     Unknown  Unknown
124: libpthread-2.28.s  000015554EA19B20  Unknown               Unknown  Unknown
124: e3sm.exe           000000000315995F  nitrogendynamicsm         209  NitrogenDynamicsMod.F90
124: e3sm.exe           0000000002FB7AF0  ecosystemdynmod_m         372  EcosystemDynMod.F90
124: e3sm.exe           00000000009B01BD  elm_driver_mp_elm        1025  elm_driver.F90
124: e3sm.exe           00000000009317D3  lnd_comp_mct_mp_l         514  lnd_comp_mct.F90
124: e3sm.exe           000000000049F2A3  component_mod_mp_         751  component_mod.F90
124: e3sm.exe           000000000045CA14  cime_comp_mod_mp_        2904  cime_comp_mod.F90
124: e3sm.exe           0000000000486C38  MAIN__                    153  cime_driver.F90
124: e3sm.exe           0000000000424F22  Unknown               Unknown  Unknown
124: libc-2.28.so       000015554E665493  __libc_start_main     Unknown  Unknown
124: e3sm.exe           0000000000424E2E  Unknown               Unknown  Unknown

[bishtgautam]

For now, I have reverted the ELMFATES.chrysalis_intel.elm-fates_satphen test to be non-debugging.

[peterdschwartz]

@bishtgautam I also found DIFFs in the unchanged land tests that only occurs when I merge this pr to the current next. Perhaps, a consequence of your change to PhenologyMod? Here are the tests:

  ERS.f19_g16.I1850GSWCNPECACNTBC.chrysalis_intel.elm-eca_f19_g16_I1850GSWCNPECACNTBC (Overall: DIFF) details:
  ERS.f19_g16.I20TRGSWCNPECACNTBC.chrysalis_intel.elm-eca_f19_g16_I20TRGSWCNPECACNTBC (Overall: DIFF) details:
  ERS.f19_g16.I20TRGSWCNPRDCTCBC.chrysalis_intel.elm-ctc_f19_g16_I20TRGSWCNPRDCTCBC (Overall: DIFF) details:

[bishtgautam]

@peterdschwartz You are correct. I will change the description and label of this PR to non-BFB.

[peterdschwartz]

Merged to next -- only 3 existing land tests should DIFF and 7 new debug tests will need new baselines generated.

[rgknox]

I'll setup an issue on this FATES fail. Thanks for reporting.

I believe that the issue is this variable is not being set to something meaningful:
https://github.com/NGEET/fates/blob/sci.1.65.7_api.25.4.0/main/FatesInterfaceMod.F90#L623