[BUG/ISSUE] Negative tracer issue in wet dep 8 months into simulation

[theloniuspunk]

Describe the bug:

Hi folks, I've been running a standard (non-TOMAS) global 4x5 simulation (dev/13.0.0-final pulled on 1/22/2021), and 8 months in to the simulation I get the the error pasted below in RESUSPENSION. If I put in a fix to make small negative numbers = 0, a similar error shows up in rainout instead.

I've seen discussion here on negative values in wet dep, but they all seem to be at the start of the simulation, not so far in.

The error is with tracer 194, which seems to be "Coarse mode sea salt alkalinity". I've never ran a simulation with this as a tracer until this version (not sure when it got added), but this is a strange tracer since it's not representing the abundance of a species.

Expected behavior:

Not getting negative values and the model stopping with an error.

Actual behavior:

See pasted info from log file.

Steps to reproduce: the bug:

Full Chem
Trop only
Standard
MERRA-2
4x5
47 layers
Switched biomass burning inventory to GFAS (but also happened in a simulation with all biomass burning off)
Used existing restart. Ran for 8 months.

Compilation commands

cd build
cmake ../CodeDir -DRRTMG=y -DRUNDIR=..
make -j8 install

**Run commands**
export OMP_NUM_THREADS=20
ulimit -s unlimited
export OMP_STACKSIZE=500m
./gcclassic > log

Error messages

===============================================================================
WETDEP: ERROR at   42  40  45 for species  194 in area RESUSPENSION in middle levels
 LS          :  T
 PDOWN       :    0.0000000000000000     
 QQ          :    0.0000000000000000     
 ALPHA       :    0.0000000000000000     
 ALPHA2      :    0.0000000000000000     
 RAINFRAC    :    0.0000000000000000     
 WASHFRAC    :    0.0000000000000000     
 MASS_WASH   :    0.0000000000000000     
 MASS_NOWASH :    0.0000000000000000     
 WETLOSS     :   -0.0000000000000000     
 GAINED      :    0.0000000000000000     
 LOST        :    0.0000000000000000     
 DSpc(NW,:)  :    0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.000000000
0000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.000000000000
0000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.000000000000000
0        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000  
      0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
   0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        
0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0
000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000
000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000
000000000        0.0000000000000000     
 Spc(I,J,:N) :    0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.000000000
0000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.000000000000
0000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.000000000000000
0        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000  
      0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000     
   0.0000000000000000        0.0000000000000000       -2.7192475214214300E-019  -1.1232100282227625E-017  -5.1485941224395791E-017  -
7.9229757610939311E-017  -9.4242926039643332E-017  -1.1084442243204630E-016  -1.1638646238616589E-016  -1.1797990245260574E-016  -1.1
888970379134583E-016  -2.0055604646319200E-016  -1.3258575230623451E-016  -4.0508276663297763E-016  -7.7982964267720829E-016  -3.8407
424712852040E-015  -2.3234387363284682E-015  -1.1315299673279659E-015  -5.4682417845637622E-016  -1.8326817611879806E-016   0.0000000
000000000       -2.6783488847695411E-017
===============================================================================''

===============================================================================
GEOS-Chem ERROR: Error encountered in wet deposition!
 -> at SAFETY (in module GeosCore/wetscav_mod.F90)
===============================================================================

===============================================================================
GEOS-Chem ERROR: Error encountered in "Safety"!
 -> at Do_Complete_Reevap (in module GeosCore/wetscav_mod.F90)
===============================================================================

===============================================================================
GEOS-Chem ERROR:
 -> at WetDep (in module GeosCore/wetscav_mod.F90)
===============================================================================

===============================================================================
GEOS-Chem ERROR: Error encountered in "Wetdep"!
 -> at Do_WetDep (in module GeosCore/wetscav_mod.F90)
===============================================================================

===============================================================================
GEOS-CHEM ERROR: Error encountered in "Do_WetDep"!
STOP at  -> at GEOS-Chem (in GeosCore/main.F90)
===============================================================================
     - CLEANUP: deallocating arrays now...

Required information:

Your GEOS-Chem version and runtime environment:

• GEOS-Chem version: dev/13.0.0-final pulled on 1/22/2021
• Compiler version: GNU 7.5.0
• netCDF version: netcdf-c-4.7.4
• netCDF-Fortran version (if applicable): netcdf-f-4.5.3
• Did you run on a computational cluster, on the AWS cloud: My own cluster
  • If you ran on the AWS cloud, please specify the Amazon Machine Image (AMI) ID: __
• Are you using GEOS-Chem "out of the box" (i.e. unmodified): Basically, I have the recent dry dep updates that I sent to the support team that will be made standard in a few months. I made some changes to TOMAS too, but this is not a TOMAS sim.
  • If you have modified GEOS-Chem, please list what was changed: __

Input and log files to attach

HEMCO_Config.rc.txt
HEMCO.log.txt
log.txt
input.geos.txt

[yantosca]

Thanks @theloniuspunk for writing. There have been similar issues: #259 and #501. Species 194 is SALACL (I think Cl on SALA aerosol). This came in with the 12.9.0 halogen updates (I believe).

I wonder if this is caused by an initial condition issue. If you are using a restart file that does not have SALACL, then the model will assume a value of 1.0e-30. This is close to zero.

In the 13.0.0 final release, we will be issuing a new set of restart files taken from the 10 year benchmarks. Try getting this one:

http://ftp.as.harvard.edu/gcgrid/data/ExtData/GEOSCHEM_RESTARTS/GC_13.0.0/GEOSChem.Restart.fullchem.20190101_0000z.nc4

there's also one for 20190701. You can look at the value of SALACL in that restart and compare to the restart that you are using to see how the values compare.

[theloniuspunk]

Thanks, Bob. I saw these issues. Also, this error started happening 8 months into the simulation, so it would be surprising if it's an initial condition issue, but maybe the 10-year spinup would help. It is happening in level 45 of the 47-level grid, so it's well up in the stratosphere where there will not be much coarse sea-spray regardless... but why is there wet dep happening at level 45?

I just wrote a quick fix to check for species 194 being negative and then setting it to 0. Hopefully, I'm not masking a bigger issue, but it's working for now and I don't suspect that SALACL will have much effect on what I'm looking at.

[yantosca]

Thanks @theloniuspunk. Maybe we can add a fix into 13.1.0 to cycle where precipitation is zero. I know we start at the top of the atmopshere and go down level by level (perhaps to get some of scavenging on ice) but this might be a simple fix.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[stale]

Closing due to inactivity