ICON4PyTools

Description

Tools and utilities for integrating icon4py code into the ICON model.

Installation instructions

To install icon4pytools in a virtual environment, one can use pip with either the requirements-dev.txt or requirements.txt file. While the requirements.txt file will install the package along with its runtime dependencies, the requirements-dev.txt file additionally includes development dependencies required for running tests, generating documentation, and building the package from source. Furthermore by using the requirements-dev.txt file, the package will be installed in editable mode, allowing the user to make changes to the package's source code and immediately see the effects without having to reinstall the package every time. This is particularly useful during development and testing phases.

# create a virtual environment
python3 -m venv .venv

# activate the virtual environment
source .venv/bin/activate

# install all dependencies
pip install -r requirements-dev.txt

Command-line tools

A variety of command-line tools are available in the shell after installation of icon4pytools.

`icon4pygen`

A bindings generator that generates C++ and Fortran bindings from Gt4Py programs. This tools generates the following code:

GridTools C++ gridtools::fn header file (.hpp).
A Fortran interface file containing all wrapper functions which can be called from within ICON (.f90).
A corresponding C (.cpp, .h) interface which in turn calls the Gridtools C++ code, as well as the C++ verification utils.

Usage

Usage: icon4pygen [OPTIONS] FENCIL [BLOCK_SIZE] [LEVELS_PER_THREAD] [OUTPATH]

  Generate Gridtools C++ code for an icon4py fencil as well as all the associated C++ and Fortran bindings.

  Arguments:
    FENCIL: may be specified as <module>:<member>, where <module>
            is the dotted name of the containing module and <member> is the name of the fencil.

    BLOCK_SIZE: refers to the number of threads per block to use in a cuda kernel.

    LEVELS_PER_THREAD: how many k-levels to process per thread.

    OUTPATH: represents a path to the folder in which to write all generated code.

Options:
  --is_global   Whether this is a global run.
  --imperative  Whether to use the imperative code generation backend.
  --help        Show this message and exit.

Autocomplete

In order to turn on autocomplete for the available fencils in your shell for icon4pygen you need to execute the following in your shell:

eval "$(_ICON4PYGEN_COMPLETE=bash_source icon4pygen)"

To permanently enable autocomplete on your system add the above statement to your ~/.bashrc file.

`icon_liskov`

A preprocessor that facilitates integration of GT4Py code into the ICON model. icon_liskov is a CLI tool which takes a fortran file as input and processes it with the ICON-Liskov DSL Preprocessor, generating code and inserting that into the target output file.

icon_liskov can either operate in integration or serialisation mode. In integration mode liskov generates calls to Fortran wrapper functions which enable calling icon4py DSL code inside of ICON from Fortran. In serialisation mode ppser serialbox statements are generated allowing the serialisation of all variables in all stencils decorated with liskov directives.

Usage

Integration mode

icon_liskov integrate [--profile] [--metadatagen] <input_filepath> <output_filepath>

Options:

profile: adds nvtx profile statements to the stencils (optional).
metadatagen: generates a metadata header at the top of the file which includes information on icon_liskov such as the version used.

Serialisation mode

icon_liskov serialise [--multinode] <input_filepath> <output_filepath>

Options:

multinode: ppser init contains the rank of the MPI process to facilitate writing files in a multinode context.

Note: By default the data will be saved at the default folder location of the currently run experiment and will have a prefix of liskov-serialisation.

Preprocessor directives

The ICON-Liskov DSL Preprocessor supports the following directives:

`!$DSL IMPORTS()`

This directive generates the necessary USE statements to import the Fortran to C interfaces.

`!$DSL START CREATE()`

This directive generates an OpenACC DATA CREATE statement. The directive requires an optional keyword argument to specify extra fields to include in the DATA CREATE statement called extra_fields. Here you can specify a comma-separated list of strings which should be added to the DATA CREATE statement as follows extra_fields=foo,bar.

`!$DSL END CREATE()`

This directive generates an OpenACC END DATA statement which is neccessary to close the OpenACC data region.

`!$DSL DECLARE()`

This directive is used to declare all DSL input/output fields. The required arguments are the field name and its associated dimensions. For example:

!$DSL DECLARE(vn=(nproma, p_patch%nlev, p_patch%nblks_e))

will generate the following code:

! DSL INPUT / OUTPUT FIELDS
REAL(wp), DIMENSION((nproma, p_patch%nlev, p_patch%nblks_e)) :: vn_before

Furthermore, this directive also takes two optional keyword arguments. type takes a string which will be used to fill in the type of the declared field, for example type=LOGICAL. suffix takes a string which will be used as the suffix of the field e.g. suffix=dsl, by default the suffix is before.

`!$DSL START STENCIL()`

This directive denotes the start of a stencil. Required arguments are name, vertical_lower, vertical_upper, horizontal_lower, horizontal_upper. The value for name must correspond to a stencil found in one of the stencil modules inside icon4py, and all fields defined in the directive must correspond to the fields defined in the respective icon4py stencil. Optionally, absolute and relative tolerances for the output fields can also be set using the _tol or _abs suffixes respectively. For each stencil, an ACC DATA region will be created. This ACC DATA region contains the before fileds of the according stencil. An example call looks like this:

!$DSL START STENCIL(name=mo_nh_diffusion_stencil_06; &
!$DSL       z_nabla2_e=z_nabla2_e(:,:,1); area_edge=p_patch%edges%area_edge(:,1); &
!$DSL       fac_bdydiff_v=fac_bdydiff_v; vn=p_nh_prog%vn(:,:,1); vn_abs_tol=1e-21_wp; &
!$DSL       vertical_lower=1; vertical_upper=nlev; &
!$DSL       horizontal_lower=i_startidx; horizontal_upper=i_endidx)

In addition, other optional keyword arguments are the following:

accpresent: Takes a boolean string input, and controls the default data-sharing behavior for variables used in the OpenACC parallel region. Setting the flag to true will cause all variables to be assumed present on the device by default (DEFAULT(PRESENT)), and no explicit data-sharing attributes need to be specified. Setting it to false will require explicit data-sharing attributes for every variable used in the parallel region (DEFAULT(NONE)). By default it is set to false.
mergecopy: Takes a boolean string input. When set to True consecutive before field copy regions of stencils that have the mergecopy flag set to True are combined into a single before field copy region with a new name created by concatenating the names of the merged stencil regions. This is useful when there are consecutive stencils. By default it is set to false.
copies: Takes a boolean string input, and controls whether before field copies should be made or not. If set to False only the #ifdef __DSL_VERIFY directive is generated. Defaults to true.
optional_module: Takes a boolean string input, and controls whether stencils is part of an optional module. Defaults to "None".

`!$DSL END STENCIL()`

This directive denotes the end of a stencil. The required argument is name, which must match the name of the preceding START STENCIL directive.

Together, the START STENCIL and END STENCIL directives result in the following generated code at the start and end of a stencil respectively.

!$ACC DATA CREATE( &
!$ACC   vn_before)

#ifdef __DSL_VERIFY
!$ACC KERNELS DEFAULT(NONE) ASYNC(1)
vn_before(:, :, :) = vn(:, :, :)
!$ACC END KERNELS

call nvtxEndRange()
#endif
call wrap_run_mo_nh_diffusion_stencil_06( &
z_nabla2_e=z_nabla2_e(:, :, 1), &
area_edge=p_patch%edges%area_edge(:, 1), &
fac_bdydiff_v=fac_bdydiff_v, &
vn=p_nh_prog%vn(:, :, 1), &
vn_before=vn_before(:, :, 1), &
vn_abs_tol=1e-21_wp, &
vertical_lower=1, &
vertical_upper=nlev, &
horizontal_lower=i_startidx, &
horizontal_upper=i_endidx
)

!$ACC END DATA

Additionally, there are the following keyword arguments:

noendif: Takes a boolean string input and controls whether an #endif is generated or not. Defaults to false.
noprofile: Takes a boolean string input and controls whether a nvtx end profile directive is generated or not. Defaults to false.
noaccenddata: Takes a boolean string input and controls whether a !$ACC END DATA directive is generated or not. Defaults to false.

`!$DSL FUSED START STENCIL()`

This directive denotes the start of a fused stencil. Required arguments are name, vertical_lower, vertical_upper, horizontal_lower, horizontal_upper. The value for name must correspond to a stencil found in one of the stencil modules inside icon4py, and all fields defined in the directive must correspond to the fields defined in the respective icon4py stencil. Optionally, absolute and relative tolerances for the output fields can also be set using the _tol or _abs suffixes respectively. For each stencil, an ACC ENTER/EXIT DATA statements will be created. This ACC ENTER/EXIT DATA region contains the before fileds of the according stencil. An example call looks like this:

        !$DSL START FUSED STENCIL(name=calculate_diagnostic_quantities_for_turbulence; &
        !$DSL  kh_smag_ec=kh_smag_ec(:,:,1); vn=p_nh_prog%vn(:,:,1); e_bln_c_s=p_int%e_bln_c_s(:,:,1); &
        !$DSL  geofac_div=p_int%geofac_div(:,:,1); diff_multfac_smag=diff_multfac_smag(:); &
        !$DSL  wgtfac_c=p_nh_metrics%wgtfac_c(:,:,1); div_ic=p_nh_diag%div_ic(:,:,1); &
        !$DSL  hdef_ic=p_nh_diag%hdef_ic(:,:,1); &
        !$DSL  div_ic_abs_tol=1e-18_wp; vertical_lower=2; &
        !$DSL  vertical_upper=nlev; horizontal_lower=i_startidx; horizontal_upper=i_endidx)

`!$DSL END FUSED STENCIL()`

This directive denotes the end of a fused stencil. The required argument is name, which must match the name of the preceding START STENCIL directive.

Note that each START STENCIL and END STENCIL will be transformed into a DELETE section, when using the --fused mode. Together, the START FUSED STENCIL and END FUSED STENCIL directives result in the following generated code at the start and end of a stencil respectively.

        !$ACC DATA CREATE( &
        !$ACC   kh_smag_e_before, &
        !$ACC   kh_smag_ec_before, &
        !$ACC   z_nabla2_e_before )

#ifdef __DSL_VERIFY
        !$ACC KERNELS DEFAULT(PRESENT) ASYNC(1)
        kh_smag_e_before(:, :, :) = kh_smag_e(:, :, :)
        kh_smag_ec_before(:, :, :) = kh_smag_ec(:, :, :)
        z_nabla2_e_before(:, :, :) = z_nabla2_e(:, :, :)
        !$ACC END KERNELS

call wrap_run_calculate_diagnostic_quantities_for_turbulence( &
    kh_smag_ec=kh_smag_ec(:, :, 1), &
    vn=p_nh_prog%vn(:, :, 1), &
    e_bln_c_s=p_int%e_bln_c_s(:, :, 1), &
    geofac_div=p_int%geofac_div(:, :, 1), &
    diff_multfac_smag=diff_multfac_smag(:), &
    wgtfac_c=p_nh_metrics%wgtfac_c(:, :, 1), &
    div_ic=p_nh_diag%div_ic(:, :, 1), &
    div_ic_before=div_ic_before(:, :, 1), &
    hdef_ic=p_nh_diag%hdef_ic(:, :, 1), &
    hdef_ic_before=hdef_ic_before(:, :, 1), &
    div_ic_abs_tol=1e-18_wp, &
    vertical_lower=2, &
    vertical_upper=nlev, &
    horizontal_lower=i_startidx, &
    horizontal_upper=i_endidx)

!$ACC EXIT DATA DELETE( &
!$ACC   div_ic_before, &
!$ACC   hdef_ic_before )

`!$DSL INSERT()`

This directive allows the user to generate any text that is placed between the parentheses. This is useful for situations where custom code generation is necessary. Note that, the INSERT`` statement is verbatim, such that there is no filtering or fortran formatting. Also, that line continuation with & is not provided forINSERT` statements.

`!$DSL START PROFILE()`

This directive allows generating an nvtx start profile data statement, and takes the stencil name as an argument.

`!$DSL END PROFILE()`

This directive allows generating an nvtx end profile statement.

`!$DSL START DELETE

This directive allows to disable code. The code is only disabled if both the fused mode and the substition mode are enabled. The START DELETE indicates the starting line from which on code is deleted.

`!$DSL END DELETE`

This directive allows to disable code. The code is only disabled if both the fused mode and the substition mode are enabled. The END DELETE indicates the ending line from which on code is deleted.

`!$DSL ENDIF()`

This directive generates an #endif statement.

ICON-Liskov integration style-guide

Check out tools/docs/ICON_Liskov_integration_style_guide.md for having a unique look and feel in the fortran integration.

`f2ser`

This tool is designed to parse a well-defined Fortran granule interface and generate ppser statements for each variable in the interface. It uses the f2py library to perform the parsing and liskov for the generation tasks.

Usage

f2ser [OPTIONS] GRANULE_PATH OUTPUT_FILEPATH

Arguments

GRANULE_PATH      A path to the Fortran source file to be parsed.
OUTPUT_FILEPATH   A path to the output Fortran source file to be generated.

Options

--dependencies PATH  Optional list of dependency paths.
--directory TEXT      The directory to serialise the variables to.
--prefix TEXT         The prefix to use for each serialised variable.

Note: The output of f2ser still has to be preprocessed using pp_ser.py, which then yields a compilable unit. The serialised files will have f2ser as their prefix in the default folder location of the experiment.

py2fgen

py2fgen is a command-line interface (CLI) tool designed to generate C and Fortran 90 (F90) wrappers, as well as a C library, for embedding a Python module into C and Fortran applications. This tool facilitates the embedding of Python code into Fortran programs by utilizing the CFFI library. CFFI instantiates a Python interpreter to execute Python code which is "frozen" into the dynamic library generated by CFFI.

Note: py2fgen has been used to embed the diffusion and dycore granule into ICON. It is important to remember that there are performance implications related to converting Fortran pointers to array-like objects that can be used in Python. It is also important to note that functions embedded into Fortran can only accept arguments with intrinsic types, as well as arrays. It is currently not possible to pass derived types to embedded Python functions.

Usage

py2fgen simplifies the process of embedding Python functions into C and Fortran codebases. Here's how to use it:

py2fgen [OPTIONS] MODULE_IMPORT_PATH FUNCTION_NAME

Arguments:
  MODULE_IMPORT_PATH  The Python module import path to the module where the functions to be embedded are defined.
  FUNCTIONS           A comma-separated list of functions to be embedded in the case of multiple, otherwise just the function name.
  PLUGIN_NAME         The name of the plugin used for creating the shared library and bindings.
Options:
  -o, --output-path PATH          Specify the directory for generated code and
                                  compiled libraries.
  -b, --backend [CPU|GPU|ROUNDTRIP]
                                  Set the backend to use, thereby unpacking
                                  Fortran pointers into NumPy or CuPy arrays
                                  respectively.
  -d, --debug-mode                Enable debug mode to log additional Python
                                  runtime information.
  -p, --profile                   Profile granule runtime and unpacking
                                  Fortran pointers into NumPy or CuPy arrays.
  --limited-area                  Enable limited area mode.

Initialising the grid

When embedding granules it may be necessary to have access to the representation of the ICON grid inside the granule. In order to initialise the grid for each granule there exists a grid_init_<granule_name> function which must also be embedded and called from Fortran. Each granule has access to a module state which is defined in a dictionary at the top of the module, for example diffusion_wrapper_state in the diffusion wrapper.

Environment variables

In order to run the embedded code from Fortran it is necessary to set environment variables based on whether you are running in a CPU or GPU context. For more information on these, as well as other information on how to build and integrate embedded code into ICON using py2fgen see this document.

Example

To create a Fortran interface along with the dynamic library for a Python function named square within the module example.functions, execute:

py2fgen example.functions square

It is also possible to generate bindings for more than one function at a time by using a comma-separated list of function names:

py2fgen example.functions square,square2

py2fgen can accept two types of functions:

Simple Function: Any Python function can be exposed.
GT4Py Program: Specifically, a Python function decorated with a @program decorator.

Important: All arguments in the exposed functions must use GT4Py style type hints. These are used by the parser to map GT4Py types to C and Fortran types in the generated bindings.

Generated Files

Running py2fgen generates five key files:

.c File: Contains the generated CFFI code and the frozen Python code.
.so File: The compiled dynamic C library containing the CFFI code.
.h File: Declares the function signature of your exposed function.
.f90 File: Contains a Fortran interface to the C function in the dynamic library.
.o File: Represents the object code of the CFFI plugin.
(Optional) .py File: Contains the Python code frozen into the dynamic library (available with --debug-mode).

Running from Fortran

To use the generated CFFI plugin in a Fortran program, call the subroutine defined in the .f90 interface file. Ensure that any arrays passed to the subroutine are in column-major order.

Examples can be found under tools/tests/py2fgen/fortran_samples.

Compilation

Compiling your Fortran driver code requires a Fortran compiler, such as gfortran or nvfortran. Follow these steps:

Compile and link the Fortran driver code along with the Fortran interface and dynamic library:

gfortran -I. -Wl,-rpath=. -L. <function_name>_plugin.f90 <fortran_driver>.f90 -l<function_name>_plugin -o <executable_name>

Replace <function_name>, <fortran_driver>, and <executable_name> with the appropriate names for your project.

Note: When executing the compiled binary make sure that you have sourced a Python virtual environment where all required dependencies to run the embedded Python code are present.

Error handling

All generated Python wrapper code is wrapped in a try: ... except: ... block, with the wrapper function returning an error code 1 if an Exception ocurred and 0 otherwise. In case of an exception the error message is written to the py2f_cffi.log file, which is located in the same directory as the generated bindings. This means that on the Fortran side we can handle errors gracefully as follows:

integer(c_int) :: rc
real(c_double), dimension(:, :), allocatable :: input, result

call square(input, result, rc)

! handle the Python error here
 if (rc /= 0) then
     print *, "Python failed with exit code = ", rc
     call exit(1)
 end if

Other requirements

Embedded Python functions must have type hints for all function parameters, as these are used to derive the corresponding C and Fortran types.
Embedded Python functions are assumed to modify function parameters in-place. Explicitly returning anything is currently not supported.

Files

README.md

Latest commit

History

README.md

File metadata and controls

ICON4PyTools

Description

Installation instructions

Command-line tools

icon4pygen

Usage

Autocomplete

icon_liskov

Usage

Integration mode

Serialisation mode

Preprocessor directives

!$DSL IMPORTS()

!$DSL START CREATE()

!$DSL END CREATE()

!$DSL DECLARE()

!$DSL START STENCIL()

!$DSL END STENCIL()

!$DSL FUSED START STENCIL()

!$DSL END FUSED STENCIL()

!$DSL INSERT()

!$DSL START PROFILE()

!$DSL END PROFILE()

`!$DSL START DELETE

!$DSL END DELETE

!$DSL ENDIF()

ICON-Liskov integration style-guide

f2ser

Usage

Arguments

Options

py2fgen

Usage

Initialising the grid

Environment variables

Example

Generated Files

Running from Fortran

Compilation

Error handling

Other requirements

`icon4pygen`

`icon_liskov`

`!$DSL IMPORTS()`

`!$DSL START CREATE()`

`!$DSL END CREATE()`

`!$DSL DECLARE()`

`!$DSL START STENCIL()`

`!$DSL END STENCIL()`

`!$DSL FUSED START STENCIL()`

`!$DSL END FUSED STENCIL()`

`!$DSL INSERT()`

`!$DSL START PROFILE()`

`!$DSL END PROFILE()`

`!$DSL END DELETE`

`!$DSL ENDIF()`

`f2ser`