Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce logfile output in HEMCO, and combine Verbose and Warnings into a single switch #182

Merged
merged 15 commits into from
Feb 7, 2023

Conversation

yantosca
Copy link
Contributor

@yantosca yantosca commented Dec 6, 2022

This is the PR corresponding to issue #180 and to geoschem/geos-chem#1549.

In this PR we seek to:

  1. Convert most HEMCO logfile output to debug-only printout
  2. Consolidate Verbose and Warnings into a single logical switch in HEMCO_Config.rc (e.g. Verbose: true) so that warnings will also be toggled by Verbose. There isn't much need for separate verbose & warnings levels any more.

Tagging @lizziel @msulprizio

src/Core/hcoio_read_std_mod.F90
- Change warnlev from 1 to 2 in the call to HCO_Warning.  This
  will only print the "data is treated as unitless" warning if
  "Warnings: 2" is defined in HEMCO_Config.rc.
- TODO: Combine Verbose and Warnings into a single logical toggle.

Signed-off-by: Bob Yantosca <[email protected]>
src/Core/hco_error_mod.F90
- Define HCO_IsVerb as an interface of 2 overloaded subroutines (one that
  takes the VerbNr argument and one that does not).  This will facilitate
  combining the Verbose and Warnings toggles into a single logical switch.
- The HCO_IsVerb* routines have been rewritten to avoid ELSE statements,
  which is more computationally efficient.
- Routine HCO_MsgErr now will only print the passed string if
  Verbose >= 3.
- Also use WRITE( 6, '(a)' ) statements instead of PRINT*, so as to
  line up the printout with the first column.
- Cosmetic changes

Signed-off-by: Bob Yantosca <[email protected]>
src/Extensions/hcox_custom_mod.F90
src/Extensions/hcox_dustdead_mod.F90
src/Extensions/hcox_dustginoux_mod.F90
src/Extensions/hcox_finn_mod.F90
src/Extensions/hcox_gc_POPs_mod.F90
src/Extensions/hcox_gc_RnPbBe_mod.F90
src/Extensions/hcox_iodine_mod.F90
src/Extensions/hcox_lightnox_mod.F90
src/Extensions/hcox_megan_mod.F90
src/Extensions/hcox_paranox_mod.F90
src/Extensions/hcox_seaflux_mod.F90
src/Extensions/hcox_seasalt_mod.F90
src/Extensions/hcox_template_mod.F90x
src/Extensions/hcox_tomas_dustdead_mod.F
src/Extensions/hcox_tomas_jeagle_mod.F90
src/Extensions/hcox_volcano_mod.f90
- If VERBOSE is not set to the highest setting (currently 3),
  then each extension will print a string to stdout of the format
  "Using HEMCO extension Name (Description)"
- If VERBOSE is set to the highest setting (currently 3), then
  the extension will print the standardized string plus other
  informational messages.
- This will allow users to see which HEMCO extensions are activated,
  but not necessarily get a lot of unnecessary printout unless they
  set VERBOSE properly.

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca added category: Feature Request New feature or request topic: Configuration Files Related to HEMCO configuration files topic: Structural Modifications Related to HEMCO structural modifications (as opposed to scientific updates) no-diff-to-benchmark This update will have no impact on benchmark simulations labels Dec 6, 2022
@yantosca
Copy link
Contributor Author

yantosca commented Dec 6, 2022

Also tagging @christophkeller

We have changed calls to

   HCO_IsVerb( HcoState%Config%Err, 3 )
   HCO_IsVerb( HcoState%Config%Err, 2 )
   HCO_IsVerb( HcoState%Config%Err, 1 )

to

   HCO_IsVerb( HcoState%Config%Err )

because we now use a logical on/off (stored in Err%doVerbose) to toggle
verbose output instead of the numerical levels.

Other related modifications:

src/Core/hco_calc_mod.F90
- Rewrite IF block where EOL is computed to avoid ELSE statement

src/Core/hco_config_mod.F90
- Now read "Verbose: true" or "Verbose: false" from HEMCO_Config.rc
- Skip reading "Warnings"
- Remove Warnings argument from call to HCO_Error_Set

src/Core/hco_emislist_mod.F90
- Remove 3rd argument (verbose level) from call to HCO_PrintDataCont

src/Core/hco_error_mod.F90
- Restore HCO_IsVerb to a single subroutine and not an overloaded
  module interface
- Add doVerbose logical variable to the TYPE(HcoErr)
- Remove verb argument from HCO_MSGErr routine
- Refactor logic in HCO_MSGErr to avoid ELSE blocks
- Change type of verb argument in HCO_MSGNoErr from integer to logical
- Remove Verbose and Warnings integer arguments in HCO_ErrorSet routine,
  and add doVerbose logical argument

src/Core/hco_logfile_mod.F90
- Remove integer argument "Verbose" from HCO_PrintList
- Remove Verbose from call to HCO_PrintDataCont
- Remove integer Verbose argument in subroutine HCO_PrintDataCont

src/Core/hco_readlist_mod.F90
- Remove integer argument "verb" from ReadList_Print routine
- Remove 3rd argument from calls to HCO_PrintList

src/Extensions/hcox_custom_mod.F90
src/Extensions/hcox_dustdead_mod.F90
src/Extensions/hcox_dustginoux_mod.F90
src/Extensions/hcox_finn_mod.F90
src/Extensions/hcox_gc_RnPbBe_mod.F90
src/Extensions/hcox_gfed_mod.F90
src/Extensions/hcox_iodine_mod.F90
src/Extensions/lightnox_mod.F90
src/Extensions/megan_mod.F90
src/Extensions/hcox_paranox_mod.F90
src/Extensions/hcox_seaflux_mod.F90
src/Extensions/hcox_seasalt_mod.F90
src/Extensions/hcox_soilnox_mod.F90
src/Extensions/hcox_template_mod.F90
src/Extensions/hcox_tomas_dustdead_mod.F90
src/Extensions/hcox_tomas/jeagle_mod.F90
src/Extensions/hcox_volcano_mod.F90
- Change verb=3 to verb=.TRUE. in calls to HCO_Msg

Signed-off-by: Bob Yantosca <[email protected]>
src/Core/hco_calc_mod.F90
src/Core/hco_clock_mod.F90
src/Core/hco_diagn_mod.F90
src/Core/hco_geotools_mod.F90
src/Core/hcoio_read_std_mod.F90
src/Core/hcoio_util_mod.F90
src/Core/hcoio_write_std_F90
src/Extensions/hcox_volcano_mod.F90
- Removed the WARNLEV argument from calls to HCO_Warning, as warnings
  are now printed when "Verbose: true" is in HEMCO_Config.rc

src/Core/hco_error_mod.f90
- Removed WARNLEV argument from HCO_WarningErr and HCO_WarningNoErr
  module routines
- Add verb argument to HCO_WarningNoErr module routine
- Removed the WARNLEV argument from calls to HCO_Warning, as warnings
  are now printed when "Verbose: true" is in HEMCO_Config.rc

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca requested review from msulprizio and lizziel and removed request for msulprizio December 8, 2022 20:34
@yantosca yantosca self-assigned this Dec 8, 2022
CHANGELOG.md
- Updated accordingly

run/HEMCO_sa_Config.rc.template
run/HEMCO_Config.rc.sample
- Removed warnings
- Changed Verbose: 0 to Verbose: false

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca changed the title [WIP] Reduce logfile output in HEMCO, and combine Verbose and Warnings into a single switch Reduce logfile output in HEMCO, and combine Verbose and Warnings into a single switch Dec 8, 2022
@yantosca
Copy link
Contributor Author

yantosca commented Dec 8, 2022

This PR is now ready for review.

@yantosca yantosca modified the milestones: 3.6.0, 3.7.0 Dec 8, 2022
CHANGELOG.md
- Keep all bullet points on one line so to avoid line breaks
  when rendering in GitHub

Signed-off-by: Bob Yantosca <[email protected]>
run/HEMCO_sa_Config.template
- Now set "Logfile: *", which will direct all output to the
  stdout (aka screen).  This can be piped to a log file with the
  tee command
- Remove reference to separate Verbose and Warnings flags; we now
  only need set Verbose to "true" to obtain maximum debug printout

Signed-off-by: Bob Yantosca <[email protected]>
run/HEMCO_sa_Config.template
- We forgot to refresh add this modification in the prior commit;
  "Logfile: HEMCO.log" is now changed to "Logfile: *", which will
  send output to the stdout.  This can be piped to a log with either
  the Unix > or tee commands.  This is now consistent with the
  settings of HEMCO_Config.rc in GEOS-Chem.

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca changed the base branch from dev/3.6.0 to dev/3.7.0 December 12, 2022 16:09
@yantosca
Copy link
Contributor Author

Integration tests pending (with GEOS-Chem)

HEMCO standalone simulation log output (24hr run)

@yantosca
Copy link
Contributor Author

Also tagging @jimmielin. Long story short, we changed "Verbose" and "Warnings" to a single "Verbose: true" flag to toggle debug output. I didn't touch the CESM config files but maybe I should. When you have time, could let us know if you see anything that would break CESM-GC functionality?

src/Core/hco_config_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_config_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_config_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_config_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_error_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_error_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_error_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_error_mod.F90 Show resolved Hide resolved
src/Core/hco_error_mod.F90 Outdated Show resolved Hide resolved
src/Core/hcoio_read_std_mod.F90 Outdated Show resolved Hide resolved
src/Core/hco_config_mod.F90
- Updated the program logic so that HEMCO verbose output will be
  computed properly if Verbose is specified as a logical value, or if
  Verbose and Warnings are specified as integer values (as was done
  previously).
- This is necessary in order to avoid conflicts when using GEOS-Chem
  in external models such as CESM or WRF.

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca
Copy link
Contributor Author

@jimmielin: I have updated the program logic (in 5041a20) so that HEMCO verbose will be toggled either with Verbose true/false, or with Verbose and Warnings having integer values. See the section here:

! !LOCAL VARIABLES:
!
    ! Scalars
    LOGICAL               :: doVerbose,   found
    LOGICAL               :: foundVerb,   foundWarn
    LOGICAL               :: verboseBool
    INTEGER               :: I,           N,         POS
    INTEGER               :: verb,        warn

    ! Strings
    CHARACTER(LEN=255)    :: line
    CHARACTER(LEN=255)    :: loc
    CHARACTER(LEN=255)    :: LogFile
    CHARACTER(LEN=255)    :: DiagnPrefix
    CHARACTER(LEN=255)    :: MetField
    CHARACTER(LEN=255)    :: GridRes
    CHARACTER(LEN=512)    :: msg

    !======================================================================
    ! ReadSettings begins here
    !======================================================================

    ! Enter
    Loc = 'ReadSettings (hco_config_mod.F90)'

    ... etc ...
    !-----------------------------------------------------------------------
    ! Read settings and add them as options to core extensions
    !-----------------------------------------------------------------------

    ! Do until exit
    DO

       ! Read line
       CALL HCO_ReadLine ( IU_HCO, LINE, EOF, RC )
       IF ( RC /= HCO_SUCCESS ) THEN
          msg = 'Error in HEMCO_Config.rc @ line: ' // TRIM( Line )
          CALL HCO_Error( msg, RC, thisLoc=loc )
          RETURN
       ENDIF

       ! Return if EOF
       IF ( EOF ) EXIT

       ! Test if the Verbose flag is logical.  We need to know this ahead
       ! of time in order to avoid input errors (i.e. reading characters
       ! when integers are expected) in the code that follows below.
       !  -- Bob Yantosca (14 Dec 2022)
       IF ( INDEX( line, 'Verbose' ) > 0 ) THEN
          verboseBool = (                                                    &
             ( INDEX( line, 't' ) > 0 ) .or. ( INDEX( line, 'f' ) > 0 ) .or. &
             ( INDEX( line, 'T' ) > 0 ) .or. ( INDEX( line, 'F' ) > 0 )     )
       ENDIF

       ... etc...

       IF ( TRIM(LINE) == '' ) CYCLE

       ! Add this option to HEMCO core
       CALL AddExtOpt ( HcoConfig, TRIM(LINE), &
                        CoreNr, RC, IgnoreIfExist=.TRUE. )
       IF ( RC /= HCO_SUCCESS ) THEN
          msg = 'Error in HEMCO_Config.rc @ line: ' // TRIM( Line )
          CALL HCO_Error( msg, RC, thisLoc=loc )
          RETURN
       ENDIF

    ENDDO

    ... etc ...

    !-----------------------------------------------------------------------
    ! Initialize error object if needed.
    ! Extract values to initialize error module and set some further
    ! HEMCO variables. Only the first time the settings are read (settings
    ! can be read multiple times if nested HEMCO configuration files are
    ! used)
    !
    ! NOTE: In HEMCO 3.7.0, the Verbose and Warnings integers in the
    ! HEMCO_Config.rc file have been replaced with "Verbose: true".
    ! Update the logic to make the test for Verbose backwards compatible
    ! with HEMCO_Config.files prior to HEMCO 3.7.0.
    !   -- Bob Yantosca (14 Dec 2022)
    !-----------------------------------------------------------------------
    IF ( .NOT. ASSOCIATED(HcoConfig%Err) ) THEN

       ! Initialize
       doVerbose = .FALSE.
       verb      = 0
       warn      = 0

       ! Check if Verbose is a logical entry in HEMCO_Config.rc
       IF ( verboseBool ) THEN

          !-----------------------------------------------------------------
          ! "Verbose: true" or "Verbose: false" was found
          !-----------------------------------------------------------------

          ! First look for Verbose (logical).  This is now the default
          ! inthe HEMCO_Config.rc file for HEMCO 3.7.0 and later.
          CALL GetExtOpt( HcoConfig,            CoreNr,       'Verbose',     &
                          OptValBool=doVerbose, found=found,  RC=RC         )
          IF ( RC /= HCO_SUCCESS ) THEN
             msg = 'Error looking for "Verbose" (logical) in HEMCO_Config.rc!'
             CALL HCO_Error( msg, RC, thisLoc=loc )
             RETURN
          ENDIF

          ! Print status message
          IF ( doVerbose ) THEN
             msg = NEW_LINE( 'A' ) // 'HEMCO verbose output is ON'
          ELSE
             msg = NEW_LINE( 'A' ) // 'HEMCO verbose output is OFF'
          ENDIF
          CALL HCO_Msg( msg, verb=.TRUE. )

       ELSE

          !-----------------------------------------------------------------
          ! "Verbose: true" or "Verbose: false" was not found
          !-----------------------------------------------------------------

          ! Check for Verbose (integer)
          CALL GetExtOpt( HcoConfig,      CoreNr,          'Verbose',        &
                          OptValInt=verb, found=foundVerb,  RC=RC           )
          IF ( RC /= HCO_SUCCESS ) THEN
             msg = 'Error looking for "Verbose" (integer) in HEMCO_Config.rc!'
             CALL HCO_Error( msg, RC, thisLoc=loc )
             RETURN
          ENDIF

          IF ( foundVerb .and. verb > 0 ) THEN

             ! Toggle HEMCO verbose output on for nonzero integer values
             doVerbose = .TRUE.
             msg = NEW_LINE( 'A' )                                        // &
                  'HEMCO verbose output is ON.'                           // &
                   NEW_LINE( 'A' )                                        // &
                  'Numbered Verbose and Warning options are deprecated.'  // &
                   NEW_LINE( 'A' )                                        // &
                  'Please use "Verbose: true:" or "Verbose: false" for '  // &
                  'controlling verbose output.'
             CALL HCO_Msg( msg, verb=.TRUE. )

          ELSE

             ! Verbose (logical) and Verbose (integer) were not found,
             ! now look for Warnings (integer)
             CALL GetExtOpt( HcoConfig,      CoreNr,         'Warnings',     &
                             OptValInt=warn, found=foundWarn, RC=RC         )
             IF ( RC /= HCO_SUCCESS ) THEN
                msg = &
                 'Error looking for "Warnings" (integer) in HEMCO_Config.rc!'
                CALL HCO_Error( msg, RC, thisLoc=loc )
                RETURN
             ENDIF
             IF ( foundWarn .and. warn > 0 ) THEN

                ! Toggle HEMCO verbose output on
                ! (Verbose = 0; Warnings > 0)
                doVerbose = .TRUE.
                msg =                                                        &
                   NEW_LINE( 'A' )                                        // &
                  'HEMCO verbose output is ON.'                           // &
                   NEW_LINE( 'A' )                                        // &
                  'Numbered Verbose and Warning options are deprecated.'  // &
                   NEW_LINE( 'A' )                                        // &
                  'Please use "Verbose: true:" or "Verbose: false" for '  // &
                  'controlling verbose output.'
                CALL HCO_Msg( msg, verb=.TRUE. )

             ELSE

                ! Toggle HEMCO verbose off
                ! (Verbose: false, Verbose = 0 and Warnings = 0)
                doVerbose = .FALSE.
                msg =                                                        &
                   NEW_LINE( 'A' )                                        // &
                  'HEMCO verbose output is OFF.'                          // &
                   NEW_LINE( 'A' )                                        // &
                  'Numbered Verbose and Warning options are deprecated.'  // &
                   NEW_LINE( 'A' )                                        // &
                  'Please use "Verbose: true:" or "Verbose: false" for '  // &
                  'controlling verbose output.'
                CALL HCO_Msg( msg, verb=.TRUE. )
             ENDIF
          ENDIF
       ENDIF

   ... etc ...

TL;DR:

  1. When reading each line from HEMCO_Config.rc, it checks to see if Verbose is logical (t, T, f, F in the same line with Verbose) or not.
  2. If Verbose is logical, then we get the value of Verbose into a logical variable doVerbose. We then print a message about whether or not verbose output is on.
  3. If Verbose is not logical, then try to read the integer values of Verbose and, if necessary Warnings. If either of these are nonzero, verbose output will be turned on. We print a message about the status of verbose output (on/off) and then also a note about the integer values being deprecated.

I ran a bunch of tests with the various settings of Verbose and Warnings, and show the message that is printed out.

-----------------------------------------------------------------------------------
Verbose:                     0
Warnings:                    0

HEMCO verbose output is OFF.
Numbered Verbose and Warning options are deprecated.
Please use "Verbose: true:" or "Verbose: false" for controlling verbose output.

-----------------------------------------------------------------------------------
Verbose:                     1
Warnings:                    0

HEMCO verbose output is ON.
Numbered Verbose and Warning options are deprecated.
Please use "Verbose: true:" or "Verbose: false" for controlling verbose output.

-----------------------------------------------------------------------------------
Verbose:                     2
Warnings:                    0

HEMCO verbose output is ON.
Numbered Verbose and Warning options are deprecated.
Please use "Verbose: true:" or "Verbose: false" for controlling verbose output.

-----------------------------------------------------------------------------------
Verbose:                     3
Warnings:                    0

HEMCO verbose output is ON.
Numbered Verbose and Warning options are deprecated.
Please use "Verbose: true:" or "Verbose: false" for controlling verbose output.

-----------------------------------------------------------------------------------
Verbose:                     0
Warnings:                    1

HEMCO verbose output is ON.
Numbered Verbose and Warning options are deprecated.
Please use "Verbose: true:" or "Verbose: false" for controlling verbose output.

-----------------------------------------------------------------------------------
Verbose:                     0
Warnings:                    2

HEMCO verbose output is ON.
Numbered Verbose and Warning options are deprecated.
Please use "Verbose: true:" or "Verbose: false" for controlling verbose output.


-----------------------------------------------------------------------------------
Verbose:                     true

HEMCO verbose output is ON

-----------------------------------------------------------------------------------
Verbose:                     false

HEMCO verbose output is OFF

-----------------------------------------------------------------------------------
no verbose in HEMCO_Config.rc

HEMCO verbose output is OFF

-----------------------------------------------------------------------------------

So I think this should be OK.

NOTE: I had to pre-determine whether Verbose is logical or not in order to avoid runtime read errors (i.e. reading integer values and getting characters, etc.)

src/Core/hco_config_mod.F90
- Removed commented-out call to HCO_ERROR_SET (with verb & warn args)

src/Core/hco_error_mod.F90
- Removed HCO_VERBOSE_INQ (this now does the same thing as HCO_IsVerb,
  and is superfluous)
- Remove Warnings and Verbose from TYPE(HcoErr)
- Remove Verbose and WarningLevel variables
- Remove Err%Warnings from logfile output

src/Core/hcoio_read_std_mod.F90
- Removed "TODO" comment

Signed-off-by: Bob Yantosca <[email protected]>
src/Interfaces/MAPL_ESMF/hcoi_esmf_mod.F90
- Remove the 2nd numerical argument from calls to HCO_IsVerb.
  These are no longer necessary as we have removed verbose levels.

Signed-off-by: Bob Yantosca <[email protected]>
src/Core/hco_config_mod.F90
- Simplify the prior code:
  - Read the Verbose setting from HEMCO_Config.rc
  - Read the VerboseOnCores setting from HEMCO_Config.rc (string)
    and define the doVerboseOnRoot logical flag accordingly
  - Now pass doVerbose and doVerboseOnRoot to HCO_Error_Set routine
  - Print an informational message

src/Core/hco_error_mod.F90
- The HCO_Error_Set routine now accepts doVerboseOnRoot as an argument
- Use both doVerbose and doVerboseOnRoot to set Err%doVerbose (which
  will determine if verbose is printed on the root core or on all cores)

Signed-off-by: Bob Yantosca <[email protected]>
src/Core/hco_error_mod.F90
- Remove extraneous characters at line 732

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca
Copy link
Contributor Author

I have removed much of the prior logic in commits 824b07f and 78238df.

Also, in commit geoschem/geos-chem@0df54f6 I have added the VerboseOnCores setting to allow users to specify if they would like to send verbose output to the root core or to all cores (analogous to GEOS-Chem).

Integration tests running

This brings the feature/clean-log-file up to date with HEMCO 3.6.0.

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca
Copy link
Contributor Author

I have merged the logfile cleanup updates to GEOS-Chem 14.1.0 and HEMCO 3.6.0. New integration tests are running.

@msulprizio msulprizio merged commit 40e906c into dev/3.7.0 Feb 7, 2023
@msulprizio msulprizio deleted the feature/clean-log-file branch February 9, 2023 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Feature Request New feature or request no-diff-to-benchmark This update will have no impact on benchmark simulations topic: Configuration Files Related to HEMCO configuration files topic: Structural Modifications Related to HEMCO structural modifications (as opposed to scientific updates)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants