pio2 failures

[ekluzek]

Brief summary of bug

The following tests fail with pio2 but pass with pio1.
SMS_D_Vmct.f09_g17.I2000Clm50SpGs.cheyenne_intel `SMS_D_Vnuopc.f09_g17.I2000Clm50SpGs.cheyenne_intel

General bug information

CTSM version you are using: ctsm1.0.dev084

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: CTSM with mct or nuopc

SMS_D_Vnuopc.f09_g17.I2000Clm50SpGs.cheyenne_intel fails...
SMS_D_Vmct.f09_g17.I2000Clm50SpGs.cheyenne_intel fails...

Details of bug

Here is a brief traceback

  /glade/work/mvertens/cesm.nuopc_dev/cime/src/externals/pio2/src/clib/pioc.c:731
  /glade/work/mvertens/cesm.nuopc_dev/cime/src/externals/pio2/src/flib/piolib_mod.F90:869
  /glade/work/mvertens/cesm.nuopc_dev/cime/src/externals/pio2/src/flib/piolib_mod.F90:899
  /glade/work/mvertens/cesm.nuopc_dev/components/clm/src/main/ncdio_pio.F90.in:2535
  /glade/work/mvertens/cesm.nuopc_dev/components/clm/src/main/ncdio_pio.F90.in:1690
  /glade/work/mvertens/cesm.nuopc_dev/components/clm/src/main/surfrdMod.F90:202

[ekluzek]

Here's the CESM log, which shows it's failing in ESMF/NUOPC..

cesm.log

[mvertens]

You need to have updated CIME and CMEPS externals before running this test. I will try running it from my sandbox.

…

On Wed, Mar 11, 2020 at 6:14 PM Erik Kluzek ***@***.***> wrote: Here's the CESM log, which shows it's failing in ESMF/NUOPC.. cesm.log <https://github.com/ESCOMP/CTSM/files/4321673/cesm.log> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#940 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4XCE7DQNS3YLQKC6MJ3F3RHASNPANCNFSM4LGBHH2Q> .

[ekluzek]

OK, let us know what versions to use.

Also note that the mosart test failed in ctsm1.0.dev083 (where the stub-ROF case passed previously).

[mvertens]

@JEdwards @ekluzek - I have changed the original bug summary to reflect the pio1/pio2 problem.

[jedwards4b]

This turns out to be an error in a debug print statement in PIO. I have submitted PR NCAR/ParallelIO#1638 and will update PIO in cime master when that is accepted. A workaround is to use PIO_REARRANGER=2.

[mvertens]

@jedwards4b - how can this happen out of the box? If PIO2 is the default io mechanism for NUOPC then we also need to set PIO_REARRANGER=2 for all nuopc cases. Or we can simply set PIO1 as the default until the PR is accepted. What should we do?

…

On Thu, Mar 12, 2020 at 11:32 AM jedwards4b ***@***.***> wrote: This turns out to be an error in a debug print statement in PIO. I have submitted PR NCAR/ParallelIO#1638 <NCAR/ParallelIO#1638> and will update PIO in cime master when that is accepted. A workaround is to use PIO_REARRANGER=2. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#940 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4XCEY2A3IPTSZ2DOEADCDRHEMEBANCNFSM4LGBHH2Q> .

[jedwards4b]

I think that this closed automatically because I mentioned it in the cime PR.

[billsacks]

Thanks @jedwards4b . But I'm actually going to close it, because I believe this problem doesn't affect any existing versions of CTSM master. (I think it is only a problem with a set of cime commits that are not currently on CTSM master.)