Overflow with Carbon isotopes...

[ekluzek]

Erik Kluzek < erik > - 2017-09-18 00:28:32 -0600
Bugzilla Id: 2511
Bugzilla CC: cdkoven, dlawren, dll, oleson, rfisher, sacks,

This occurs about 20 days in with clm4_5_16_r255 with SMS_D_Lm9.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-ciso_monthly

or with a Hist case as well.

44: Negative conc. in ch4tran. c,j,deficit (mol): 3113 1
44: 2.390122000974219E-003
44: Negative conc. in ch4tran. c,j,deficit (mol): 3126 1
44: 2.186088406244086E-003
7:forrtl: error (72): floating overflow
7:Image PC Routine Line Source
7:cesm.exe 00000000049EF671 Unknown Unknown Unknown
7:cesm.exe 00000000049ED7AB Unknown Unknown Unknown
7:cesm.exe 00000000049A0114 Unknown Unknown Unknown
7:cesm.exe 000000000499FF26 Unknown Unknown Unknown
7:cesm.exe 0000000004925079 Unknown Unknown Unknown
7:cesm.exe 000000000493166C Unknown Unknown Unknown
7:libpthread-2.19.s 00002AAAAFAC1870 Unknown Unknown Unknown
7:cesm.exe 0000000003119840 cncisofluxmod_mp_ 1594 CNCIsoFluxMod.F90
7:cesm.exe 00000000030A7468 cncisofluxmod_mp_ 149 CNCIsoFluxMod.F90
7:cesm.exe 0000000002B9C9FD cndrivermod_mp_cn 531 CNDriverMod.F90
7:cesm.exe 00000000011604E0 cnvegetationfacad 837 CNVegetationFacade.F90
7:cesm.exe 000000000089677F clm_driver_mp_clm 803 clm_driver.F90
7:cesm.exe 0000000000849921 lnd_comp_mct_mp_l 444 lnd_comp_mct.F90
7:cesm.exe 0000000000469CBE component_mod_mp_ 681 component_mod.F90
7:cesm.exe 000000000043BD99 cesm_comp_mod_mp_ 2639 cesm_comp_mod.F90
7:cesm.exe 000000000045285B MAIN__ 68 cesm_driver.F90

[ekluzek]

Erik Kluzek < erik > - 2017-09-18 11:37:12 -0600

The problem is from this line:

    if (ctot_state(i) /= 0._r8 .and. ciso_state(i) /= 0._r8) then
       ciso_flux(i) = ctot_flux(i) * (ciso_state(i)/ctot_state(i)) * frax
    else

ciso_state is set to something like -5e+305 for some reason, and hence dividing by the small value of ctot_state makes it overflow.

And a similar test fails on yellowstone.

SMS_D_Lm13.f10_f10_musgs.I1850Clm50BgcCrop.yellowstone_intel.clm-ciso_monthly

I think this worked in simulations that were done because they weren't run in DEBUG mode.

I'm going to leave this problem as an expected fail.

[ekluzek]

Bill Sacks < sacks > - 2017-09-18 14:10:17 -0600

From a quick look through the code: It looks like it could be worth determining which of the many fluxes added to / subtracted from cpool in CStateUpdate1 has this huge value.

[ekluzek]

Keith Oleson < oleson > - 2017-09-18 17:11:24 -0600

This bug seems quite similar to a bug Charlie and Danica fixed with regard to crop seed C as follows from Danica:

"I'm happy to dive into the details of the bugs when you return. A brief description (you shouldn't think about it much while you're traveling): The bugs mostly dealt with the crop seed deficit & the fact that crop seed C wasn't initialized with isotope values. In the desert regions, the deficit accumulated when crops didn't grow and the seed C was still getting removed and was subtracted from total veg. C, causing that pool to go negative in crop regions when the crops didn't grow. The fix is to treat the deficit similar to the excess maintenance respiration pool, and subtract it from the total soil column C rather than veg C. It's not a perfect solution, but it works and is consistent with some of the other fixes that were previously implemented. The third bug was the one that we found during the CESM meeting (live stem C) that didn't get incorporated into the tests that we were doing."

However, I believe we were getting large negative values of ciso_state which I think was associated with totvegc, and the line number in the error here seems to indicate that it is a cpool problem, not a totvegc problem, so this may be a red herring.

I'm not sure exactly what the bug fix was (I'm adding Danica on this bug report), but it seems like we could check to make sure it didn't get lost or something, just in case it is related to the current problem.

I guess this particular test must be new since clm4_5_16_r252, because this test wasn't present in my cheyenne testing with smanchanges_n06_clm4_5_16_r252.

[ekluzek]

Erik Kluzek < erik > - 2017-09-19 11:21:22 -0600

OK, I did more testing, and found this is a problem in more situations than I thought. Hence, I'm upping it's priority and importance.

It does show up in runs with DEBUG=FALSE, just with different behavior with the flux_dribbler complaining. My tests were with f10, so I tried f09 and get the same problem. The tests that Keith did were with f09, and I also replicated his test when I first took this over again. That may mean I can go back through and see what update on the branch caused the test to break. It could be an update to a newer version of CLM, or another update that happened.

And yes this is a new test in order to exercise running crop and c-isotopes on at the same time.

[ekluzek]

Erik Kluzek < erik > - 2017-10-05 11:49:56 -0600

OK, when I use a properly spunup initial condition file (with isotopes on) these tests work fine.

ERS_D_Lm9.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_monthly
SMS_D_Lm9.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-ciso_monthly

So it's a problem of starting up from a non-isotope case to an isotope case.

[billsacks]

@ekluzek these tests are now passing (if we give them enough wallclock time): SMS_D_Lm13.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_monthly and ERP_D_Lm9.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_monthly. I haven't followed this closely enough to know if that means we should close this issue, or if this issue still exists but these tests just happen to pass now.

[billsacks]

Based on the last few comments, it looks like the remaining issue is a duplicate of #67