Perform gridcell-level water balance checks bracketing the entire run loop

[billsacks]

From @billsacks on March 17, 2017 20:38

Currently, the CLM driver loop looks like this, in terms of pieces relevant to this issue:

• Compute pre-dynlu gridcell-level water mass

• Adjust column areas via dynamic landunits

• Compute post-dynlu gridcell-level water mass

• Compute adjustment fluxes based on difference between pre- and post-dynlu gridcell-level water mass

• Compute pre-physics column-level water mass

• Do physics

• Compute post-physics column-level water mass

• Do column-level balance checks using the pre- and post-physics column-level water mass and the relevant fluxes

As long as you trust:

1. that the calculation of the dynlu adjustment fluxes is done correctly, and

2. that the column-level fluxes used in the column-level balance checks correctly mirror the relevant gridcell-level fluxes

then you can convince yourself that this sequence ensures gridcell-level conservation.

However, it would be more robust and more reassuring if we performed gridcell-level water conservation checks, bracketing essentially the entire driver run loop (including dynamic landunits adjustments as well as the physics). Ideally, the fluxes considered in these gridcell-level checks would be the actual fluxes received by and sent from CLM (gridcell-level precipitation, runoff, etc.). This last point is particularly important for runoff, which is now split into a few different terms, and it seems easy for some term to accidentally not be included in any of the runoff terms sent to the coupler - so we should do these gridcell-level balance checks using the runoff terms that are actually sent to the coupler.

The sequence would then look like:

• Compute beginning gridcell-level water mass

• Adjust column areas via dynamic landunits

• Compute post-dynlu gridcell-level water mass

• Compute adjustment fluxes based on difference between pre- and post-dynlu gridcell-level water mass

• Compute pre-physics column-level water mass

• Do physics

• Compute post-physics column-level water mass; also average this to the gridcell level

• Do column-level balance checks using the pre- and post-physics column-level water mass and the relevant fluxes

• Do gridcell-level balance checks using the beginning gridcell-level water mass and ending gridcell-level water mass. For this check, ideally use the gridcell-level fluxes that are passed to and from CLM

Note that this plan still maintains the column-level balance checks: These are useful for seeing which column(s) have balance errors. But it's possible that we could avoid doing these in general, and just drill down to the column-level if a gridcell-level balance check fails.

Copied from original issue: NCAR/CLM#3

[billsacks]

From @billsacks on March 17, 2017

One thing needed in counting up the gridcell-level water mass is the pseudo-state from the annual dribblers (i.e., the result of get_amount_left_to_dribble): treat this like any other state for purposes of the conservation check (i.e., add it to the initial state and to the final state)

[billsacks]

From @billsacks on April 28, 2017

Note that, with the change in clm4_5_15_r235 (addition of for_testing_zero_dynbal_fluxes namelist flag), it's possible that we sometimes won't conserve gridcell-level water and energy (if this flag is set). We'll need to handle this case, probably by introducing a term to track the lack of conservation, and including that term in the balance checks.

[billsacks]

Another thing that I think the current balance checks are missing is a check of the downscaling of atm2lnd terms, such as rain-snow repartitioning. The balance checks should do checks using the pre-downscaled fluxes to ensure that we're not gaining or losing mass in the downscaling code. (This may be relatively independent of the rest of this issue.)

[billsacks]

Tagging this high priority along with #314 to avoid further problems like #675 .

[billsacks]

It looks like this has been done in e3sm (maybe by @bishtgautam ), so that implementation could be used as a guide.

[bishtgautam]

A grid level water balance check have been added in ELM, but I believe that balance check only works for the case when dyn subgrid driver isn't active.

• BeginGridWaterBalance() in BalanceCheckMod.F90#L703
  
• EndGridWaterBalance() in BalanceCheckMod.F90#L844
  

[billsacks]

Thanks, @bishtgautam !

Also, a suggestion from @dlawrenncar - If the performance cost of the column-level checks is significant, we could do something like: Initialize both the gridcell and column-level balance checks. Then, at the end of the time step, just do the gridcell-level check; only do the column-level check if this gridcell-level check fails. The rationale is: the gridcell-level check should probably be sufficient to catch balance errors; the main point of the column-level check would then be to give additional information about the column with the problem.

Whether or not we do that, I realized that: We should either do the column-level checks first or, if we do the gridcell-level checks first (e.g., as per @dlawrenncar 's suggestion), then we should wait to abort until after we've done the column-level checks, so that we print information from the column-level checks.

[billsacks]

Referring to the last comment (#201 (comment)): I think it's possible that column-level balance checks would fail even if the gridcell-level checks pass. However, this would often (always?) be due to a problem in the balance check itself, such as forgetting to add a flux in the check. This makes me think: If we go with Dave Lawrence's suggestion of only doing the column-level checks if the gridcell-level checks have failed, then we should probably additionally do the column-level checks always in DEBUG mode (using #ifndef NDEBUG), to catch code errors in the column-level balance checks.