Skip to content

Latest commit

 

History

History
409 lines (297 loc) · 25.9 KB

README.md

File metadata and controls

409 lines (297 loc) · 25.9 KB
Raw

ESMX

ESMX is the Earth System Model eXecutable layer.

http://earthsystemmodeling.org/esmx/

The ESMX layer is built on top of the ESMF and NUOPC APIs.

The idea behind ESMX is to make it as simple as possible for a user to build, run, and test NUOPC based systems. The approach implemented is the same whether applied to a single component, or a fully coupled system of NUOPC-compliant components. ESMX user interfaces are implemented through YAML based configuration files.

Major objectives of ESMX are:

  • Simplification of standing up new NUOPC-based systems.
  • Promotion of hierarchical model component testing.
  • Reduction of maintenance cost for established NUOPC-based systems.
  • Improved alignment and interoperability between different NUOPC-based systems. (Configuration files, build procedures, etc.)
  • Faster and more coordinated roll-out and adoption of new NUOPC/ESMF features.

ESMX unifies, provides, and maintains those parts of NUOPC-based modeling systems that are common across most such implementations. This includes the top level application and driver code, parts of the build infrastructure, and tools to manage run configurations.

The Unified Driver Executable

One of the main pieces provided by ESMX is the unified driver executable. A good starting point to explore this feature is the ESMX_AtmOcnProto example under the NUOPC prototype repository.

The unified driver executable created by ESMX is defined by exe_name under Application Options (default: esmx_app). ESMX_EXE_NAME will be used for the remainder of this section to refer to the unfied driver executable.

ESMX_Builder

The ESMX_Builder is a shell script included with ESMF installations that facilitates building ESMX. The script is installed into the ESMF binary directory within an ESMF installation. For more information on installing ESMF see the ESMF User's Guide.

If the ESMF binary directory is included in the PATH environment variable then ESMX_Builder can be called from any directory as follows:

ESMX_Builder [OPTIONS ...] [ESMX_BUILD_FILE]

options:
  [--esmx-dir=ESMF_ESMXDIR]
  [--esmfmkfile=ESMFMKFILE]
  [--build-dir=BUILD_DIR]
  [--prefix=INSTALL_PREFIX]
  [--disable-comps=DISABLE_COMPS]
  [--build-type=BUILD_TYPE] or [-g]
  [--build-args=BUILD_ARGS]
  [--cmake-args=CMAKE_ARGS]
  [--build-jobs=JOBS]
  [--load-modulefile=MODULEFILE]
  [--load-bashenv=BASHENV]
  [--test[=TEST_ARGS]] or [-t[=TEST_ARGS]]
  [--verbose] or [-v]

where:
  ESMX_BUILD_FILE          ESMX build configuration file
                           (default: esmxBuild.yaml)

  --esmx-dir=ESMF_ESMXDIR  ESMX source directory
                           (default: use ESMF_ESMXDIR in ESMFMKFILE)

  --esmfmkfile=ESMFMKFILE  ESMF makefile fragment
                           (default: use environment)

  --build-dir=BUILD_DIR    build directory
                           (default: build)

  --prefix=INSTALL_PREFIX  installation prefix
                           (default: install)

  --disable-comps=DISABLE_COMPS  disable components (e.g. ESMX_Data)

  --build-type=BUILD_TYPE  build type; valid options are 'debug', 'release', 'relWithDebInfo'
  -g                       (default: release) (-g sets BUILD_TYPE to debug)

  --build-args=BUILD_ARGS  build arguments passed to every component (e.g. -DVAR=val)

  --cmake-args=CMAKE_ARGS  cmake arguments are used to configure cmake while building ESMX
                           (e.g. -DVAR=val)

  --build-jobs=BUILD_JOBS  number of jobs used for building esmx and components

  --load-module=MODULEFILE load modulefile before building

  --load-bashenv=BASHENV   load bash environment file before building

  --test[=TEST_ARGS] or    (beta) run ctest suite for ESMX Driver, pass TEST_ARGS to ctest
  -t[=TEST_ARGS]

  --verbose or -v          build with verbose output

This script installs ESMX_EXE_NAME into INSTALL_PREFIX/bin.

ESMX Build Configuration

The ESMX build system depends on a build file defined by the ESMX_BUILD_FILE variable. When unspecified the ESMX_BUILD_FILE defaults esmxBuild.yaml. This is a YAML file with a very simple format. An example is given here:

application:

  disable_comps: ESMX_Data
  cmake_build_args: -DOPTION=VALUE

components:

  TaWaS:
    cmake_config: TaWaS/tawas.cmake
    fort_module:  tawas.mod

  Lumo:
    cmake_config: Lumo/lumo.cmake

tests:

  TestTawasLumo:
    dir: path/to/test/data

In this example two components are built into ESMX_EXE_NAME explicitly. (Read about dynamically loading of components from shared objects at run-time later.) Each component is given a name, here TaWaS and Lumo, respectively. Components will be referenced by this component-name in the run-time configuration (esmxRun.yaml) discussed below.

CAUTION: Component names are case-sensitive when used e.g. by default to construct library names, etc. However, they are treated case-insensitive when referenced from within the esmxRun.yaml file due to the case-insensitive nature of Fortran when referencing modules via the USE statement.

ESMX comes with a default data component called ESMX_Data. It is built into ESMX_EXE_NAME by default. This example disables this behavior by setting disable_comps: ESMX_Data.

There are three top level sections recognized in the ESMX build file. Each is introduced by a specific key using the YAML map syntax: application:, components:, and tests:. The value associated with each key is a map of key/value pairs. The available option keys under each top level section are discussed below.

Application Options (application key)

These options affect the ESMX application layer. If no key/value pair is provided then the default will be used.

Option key Description / Value options Default
exe_name executable name for application esmx_app
disable_comps scalar or list of components to disable None
link_module_paths scalar or list of search paths for CMake modules None
link_libraries scalar or list of external libraries, linked to esmx None
link_options scalar or list of options used during linking of esmx None
link_packages scalar or list of cmake packages, use link_libraries to link to esmx None
link_paths scalar or list of search path for external libraries None
build_args scalar or list of arguments passed to all build_types None
build_jobs job number used for all build_types None
build_verbose verbosity setting used for all build_types None
cmake_build_args scalar or list of argumens passed to all cmake builds None
cmake_build_jobs job number used for all cmake builds None
cmake_build_verbose verbosity setting used for all cmake builds None
make_build_args scalar or list of argumens passed to all make builds None
make_build_jobs job number used for all make builds None
script_build_args scalar or list of argumens passed to all script builds None
test (beta) add test cases: on or off off
test_exe (beta) executable used to launch test cases ESMF_INTERNAL_MPIRUN
test_dir (beta) output directory for test cases CMAKE_BINARY_DIR-tests
test_tasks (beta) number of tasks used to run test cases 4

Component Options (components key)

This section contains a key for for each component-name, specifying component specific options. If no key/value pair is provided then the default will be used.

Option key Description / Value options Default
build_type build type: auto, cmake, make, script, none auto
build_script build script compile
build_args scalar or list of arguments for building component None
source_dir source directory for build component-name
cmake_config CMake configuration file component-name.cmake
libraries component libraries, linked to esmx component-name
fort_module fortran module filename for NUOPC SetServices component-name.mod
install_prefix root directory for installation install
config_dir subdirectory for cmake configuration file cmake
library_dir subdirectory for library file lib
include_dir subdirectory for fortran module file include
link_paths search path for external libraries None
link_into_app whether to link component into the app True
link_libraries external libraries, linked to esmx None
git_repository URL for downloading git repository None
git_tag tag for downloading git repository None
git_dir download directory for git repository None
test_dir (beta) directory used for test case data None
test_exe (beta) executable used to run test case ESMX_TEST_EXE
test_tasks (beta) number of tasks used to run test case ESMX_TEST_TASKS
Build Types:

auto - The ESMX build system searches for CMakeLists.txt, Makefile, and a build_script in order in the source_dir and uses the first build option it finds. If no build files are found then the ESMX build system searches for the CMake configuration file, fortran module, and libraries in the install_prefix directory but does not build the model. If no build option, CMake configuration, or libraries are found then the build fails.

cmake - The ESMX build system searches for CMakeLists.txt in the source_dir and builds using CMake. Once built, the ESMX build system searches for the CMake configuration file and libraries in the install_prefix directory.

make - The ESMX build system searches for Makefile in the source_dir and builds using Make without a target. If a specific target is desired then it can be configured using build_args. Once built, the ESMX build system searches for libraries and fortran modules in the install_prefix directory.

script - The ESMX build system searches for a build_script in the source_dir and builds using this script. Once built, the ESMX build system searches for libraries and fortran modules in the install_prefix directory.

none - The ESMX build system will not build the component. The ESMX build system searches for the CMake configuration file, fortran module, and libraries in the install_prefix directory.

Test Options (tests key)

This section contains a key for for each test-name, specifying test specific options.

Option key Description / Value options Default
dir (beta) directory used for test case data None
exe (beta) executable used to run test case ESMX_TEST_EXE
tasks (beta) number of tasks used to run test case ESMX_TEST_TASKS

ESMX Run Configuration

At startup, the ESMX_EXE_NAME executable checks the first command line argument for a filename. If no argument is provided then the filename defaults to esmxRun.yaml. This is the ESMX run-time configuration file in YAML format, providing settings such as the list of components active at run-time, component attributes, the run sequence, as well as application level options.

An example ESMX Run Configuration file is given here:

ESMX:

  App:
    globalResourceControl:  true
    logKindFlag:            ESMF_LOGKIND_Multi
    logAppendFlag:          false
    logFlush:               true
    startTime:              2012-10-24T18:00:00
    stopTime:               2012-10-24T19:00:00

  Driver:
    componentList:          [ATM, OCN]
    attributes:
      Verbosity: low

    runSequence: |
      @900
        ATM -> OCN
        OCN -> ATM
        ATM
        OCN
      @

ATM:
  model:            Tawas     # model value is case insensitive to match Fortran
  ompNumThreads:    4
  attributes:
    Verbosity:  low
  petList:          [3, [2-0]]  # petList is list of scalars and lists.
                                # each list again can be of scalars and lists
                                # recursively.

OCN:
  model:            lumo
  petList:          [0-1, 3]
  attributes:
    Verbosity:  low

On the highest level, ESMX Run Configuration is expected to define the ESMX key, as well as a key for every component that is listed in the componentList found under the Driver level. The ESMX key is associated with a map containing the App and Driver keys. The App key must be present if ESMX Run Configuration is read by the ESMX_EXE_NAME executable, but is optional (and will be ignored) in case esmfRun.yaml is read by the esmx_driver.

ESMX/App Options

This section affects the application level.

CAUTION: Specifying any of the ESMF_RUNTIME_* keys overrides the corresponding environment variables set in the user's environment.

Option key Description / Value options Default
startTime string setting the application start time non-optional
stopTime string setting the application stop time non-optional
globalResourceControl enable/disable global resource control: true or false false
logKindFlag ESMF logging kind, see ESMF RefDoc for options ESMF_LOGKIND_Multi_On_Error
logAppendFlag enable/disable log append: true or false true
defaultLogFilename name of the default ESMF log file (suffix if multi PET) ESMF_LogFile
defaultCalKind ESMF calendar kind used by default, see ESMF RefDoc for options ESMF_CALKIND_GREGORIAN
logFlush enable/disable log flush for each write: true or false false
fieldDictionary name of the NUOPC field dictionary file to be loaded None
ESMF_RUNTIME_PROFILE enable/disable all profiling functions: ON or OFF OFF
ESMF_RUNTIME_PROFILE_OUTPUT output format; multiple can be selected: TEXT SUMMARY BINARY TEXT
ESMF_RUNTIME_PROFILE_PETLIST limit profiling to an explicit list of PETs all PETs
ESMF_RUNTIME_TRACE enable/disable all tracing functions: ON or OFF OFF
ESMF_RUNTIME_TRACE_CLOCK type of clock for events: REALTIME or MONOTONIC or MONOTONIC_SYNC REALTIME
ESMF_RUNTIME_TRACE_PETLIST limit tracing to an explicit list of PETs all PETs
ESMF_RUNTIME_TRACE_COMPONENT enable/disable tracing of component events: ON or OFF ON
ESMF_RUNTIME_TRACE_FLUSH frequency of event stream flushing to file: DEFAULT or EAGER DEFAULT
ESMF_RUNTIME_COMPLIANCECHECK enable/disable NUOPC compliance checking: ON or OFF with DEPTH OFF

ESMX/Driver Options

This section affects the driver level.

Option key Description / Value options Default
componentList list of component labels, each matching a top level key in this file Empty
runSequence block literal string defining the run sequence NUOPC default
attributes map of key value pairs, each defining a driver attribute None

Component Label Options

This section affects the specific component instance.

Option key Description / Value options Default
model string associating the instance with a component-name defined in esmxBuild.yaml non-optional
petList list of PETs on which the component executes None
devList list of DEVs (accelerator devices) to be associated with the component None
ompNumThreads setting of /NUOPC/Hint/PePerPet/MaxCount (see NUOPC ref doc) None
attributes map of key value pairs, each defining a component attribute None
model specific yaml each model can define its own YAML section, e.g. with key value pairs, etc. None

Dynamically loading components from shared objects at run-time

There are two options recognized when specifying the value of the model field for a component in the esmxRun.yaml file:

  • First, if the value specified is recognized as a component-name provided by any of the components built into the esmx_app during build-time, as specified by esmxBuild.yaml, the respective component is accessed via its Fortran module.
  • Second, if the value does not match a build-time dependency, it is assumed to correspond to a shared object file instead. In that case the attempt is made to load the specified shared object file at run-time, and, if successful, is associated with the generic component label. The search order details of the OS dependent dynamic linker apply when looking for the specified shared object file on the system. A convenient way to target a shared object file at a specific location is to use absolute or relative paths, i.e. the value specified in the model field contains at least one slash ("/") character. The asterisk character ("*") is supported as a wildcard for the file name suffix of the specified shared object. This allows portability across systems that differ in shared object suffix. The implemented search order is "so", followed by "dylib", and finally "dll", where the first successfully loaded shared object file is used.

The Unfied ESMX_Driver

The ESMX layer provides access to the ESMX_Driver via the public NUOPC Driver API. This means that ESMX_Driver can be plugged into a larger NUOPC application as a standard NUOPC component, or alternatively be accessed through the External NUOPC Interface. A good starting point to explore this feature is the ESMX_ExternalDriverAPIProto under the NUOPC prototype repository.

Project integration

The typical situation where ESMX_Driver comes into play is where a user application needs to access a NUOPC based system that uses the unified ESMX driver. Assuming the user application uses CMake, integration of ESMX is straight forward. The critical piece required is to add add_subdirectory() in the application's CMakeLists.txt file to bring in the ${ESMF_ESMXDIR}/Driver directory, and make the application dependent on target esmx_driver. An example for a very simple application is shown:

cmake_minimum_required(VERSION 3.22)

# Where to look for the local Find<Package>.cmake files
list(APPEND CMAKE_MODULE_PATH "${ESMF_ESMXDIR}/Driver/cmake")

# Find ESMF
find_package(ESMF 8.5.0 MODULE REQUIRED)

# Set compilers consistent with ESMF
set(CMAKE_Fortran_COMPILER        "${ESMF_F90COMPILER}")
set(CMAKE_CXX_COMPILER            "${ESMF_CXXCOMPILER}")
set(CMAKE_C_COMPILER              "${ESMF_CCOMPILER}")

# Project
project(ExternalDriverAPIProto
        VERSION 1.0.0
        LANGUAGES Fortran CXX C
        )

# Add ESMX driver
add_subdirectory(${ESMF_ESMXDIR}/Driver ./ESMX_Driver)

# Create executable
add_executable(externalApp externalApp.F90)
target_include_directories(externalApp PUBLIC ${PROJECT_BINARY_DIR})
target_link_libraries(externalApp PUBLIC esmx_driver)

# Install executable
install(
  TARGETS externalApp
  EXPORT externalApp
  RUNTIME DESTINATION bin
  LIBRARY DESTINATION lib
  ARCHIVE DESTINATION lib)

The applcation can then be built as typically via cmake commands, only requiring that the ESMF_ESMXDIR variable is passed in. It can be convenient to wrap the cmake commands into a GNU Makefile, accessing the ESMF_ESMXDIR variable through the ESMFMKFILE mechanism.

When executing cmake, the ESMX_BUILD_FILE can be specified using -DESMX_BUILD_FILE=<value>. If ESMX_BUILD_FILE is not defined in the command line arguments then it will default to esmxBuild.yaml.

include $(ESMFMKFILE)

build/externalApp: externalApp.F90 esmxBuild.yaml
        cmake -S. -Bbuild -DESMF_ESMXDIR=$(ESMF_ESMXDIR) -DESMX_BUILD_FILE="<value>"
        cmake --build ./build

esmxBuild.yaml and esmxRun.yaml

The esmx_driver target defined by the add_subdirectory(${ESMF_ESMXDIR}/Driver ./ESMX_Driver) has a build-time dependency on the ESMX_BUILD_FILE already discussed under the ESMX Build Configuration section. The identical file can be used when working on the ESMX_Driver level.

The run-time configuration needed by ESMX_Driver can either be supplied by the user application, or alternatively default to esmxRun.yaml. The following rules apply:

  • ESMX_Driver, at the beginning of its SetModelServices() method checks whether the parent level has provided an ESMF_Config object by setting the config member on the ESMX_Driver component. If so, the provided config object is used. Otherwise ESMX_Driver itself creates config from file esmxRun.yaml.
  • For the case where the config object was provided by the parent layer, ESMX_Driver does not ingest attributes from config. Instead the assumption is made that the parent layer sets the desired attributes on ESMX_Driver.
  • For the case where the config object was loaded from esmxRun.yaml by ESMX_Driver, the driver ingests attributes from config, potentially overriding parent level settings.
  • The ESMX_component_list, child component, and run sequence information is ingested from config as described under the ESMX Run Configuration section.
  • If the parent level passes an ESMF_Clock object to ESMX_Driver during initialize, the driver uses it instead of looking for startTime and stopTime in config.
  • For the case where a clock is provided by the parent layer, its timeStep is used as the default time step of the outer run sequence loop when using the @* syntax. If a specific time step is set in the run sequence with @DT, then DT must be a divisor of the timeStep provided by the parent clock.

ESMX Test System (beta)

The ESMX layer includes a test system based on CTest. This system is still in beta and as it evolves features may be added or removed. When enabled the test system adds a suite of component and system level tests. All tests in the test suite can be executed via ESMX_Builder -t. If the ESMX_Builder is not used then tests can be added by setting test: on under Application Options. Tests can be manually executed from each directory in test_dir or through executing ctest in the <CMAKE_BINARY_DIR>/Driver directory.

ESMX Components

ESMX includes a data component, which can be used for testing NUOPC caps. This component is known as ESMX_Data.

ESMX Software Dependencies

The ESMX layer has the following dependencies:

  • ESMF Library: The ESMX layer is part of the ESMF repository. In order to use ESMX as described above, the ESMF library first needs to be built following the instructions for Building ESMF.
  • CMake: v3.22 or greater.
  • Python: v3.5 or greater.
    • python3 must be in $PATH.
    • PyYaml must be installed in the Python environment.

There are many ways to provide a suitable Python environment. One portable way based on venv is shown below.

python3 -m venv ESMXenv
source ESMXenv/bin/activate (for Bash)  or   source ESMXenv/bin/activate.csh (for Csh)
pip install pyyaml
... Environment ready for ESMX ...
deactivate