Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify global model #315

Merged
merged 53 commits into from
Dec 15, 2023
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
5e7af70
remove unusde fields from metrics savepoint
halungge Nov 9, 2023
c3acfd8
remove dependency diffusion in common/test_utils
halungge Nov 10, 2023
853f5c9
use system epsilon
halungge Nov 10, 2023
8afbb4e
remove unused fields from serialbox_utils.py
halungge Nov 10, 2023
0237b20
fix imports in io_utils.py
halungge Nov 10, 2023
1b8271b
remove unused typed definition
halungge Nov 10, 2023
e0e8398
fix wrong import in test_diffusion_states.py
halungge Nov 10, 2023
acfc1cc
Merge branch 'main' into verify_global_model
halungge Nov 15, 2023
050cbc9
use dallclose in diffusion
halungge Nov 15, 2023
8516c51
diffusion: make choice of configuration flexible in tests
halungge Nov 15, 2023
85583a9
- pass experiment name a argument to datapath
halungge Nov 16, 2023
877f199
WIP diffusion single_step test for exclaim APE
halungge Nov 16, 2023
d745e5d
WIP (1) make velocity advection and solve nonhdyro tests parametrizab…
halungge Nov 17, 2023
e2d3583
WIP (2) make velocity advection and solve nonhdyro tests parametrizab…
halungge Nov 17, 2023
fcbf9ca
Merge branch 'main' into verify_global_model
halungge Nov 17, 2023
d7bd316
WIP (1) make solve_nonhydro.py choose configuration dynamically
halungge Nov 17, 2023
cf5a1a0
run test_solve_nonhydro.py::test_run_nonhydro_single_step for exclaim…
halungge Nov 20, 2023
c23d7e5
add experiment fixture to common and driver
halungge Nov 20, 2023
0b14265
run on specified backend for test_diffusion_utils.py
halungge Nov 21, 2023
fdbfe63
enable both experiments in test_solve_nonhydro.py
halungge Nov 21, 2023
085ad42
WIP: fix assert in test_diffusion.py
halungge Nov 22, 2023
f0dfaa1
merge main
halungge Nov 28, 2023
a4206ca
test_velocity_advection.py: for both experiments
halungge Nov 29, 2023
ac21490
test_diffusion.py: for both experiments
halungge Nov 29, 2023
e0f4298
fix tests: test_diffusion_utils.py, test_vertical.py
halungge Nov 29, 2023
c3df5aa
test_solve_nonhydro.py: invert args in dallclose, reference should be…
halungge Nov 29, 2023
d1efb33
add missing config parameters in Diffusion config
halungge Nov 30, 2023
eed4bc3
fix imports
halungge Nov 30, 2023
68c8b77
- make data loading fixture dependent
halungge Nov 30, 2023
aed9489
- add logging to solve_nonhydro.py
halungge Nov 30, 2023
da7b2d2
remove wrong and unused grid.nflat_gradp
halungge Nov 30, 2023
8eacbb9
set grid.limited_area according to experiment
halungge Nov 30, 2023
d1eb281
assert bounds in test
halungge Nov 30, 2023
7e49286
pre-commit fixes
halungge Dec 1, 2023
ac8fea5
fix typo in stencil name
halungge Dec 1, 2023
b108aa5
fix typo in stencil name
halungge Dec 1, 2023
767d64d
diffusion_tests:
halungge Dec 1, 2023
e36f8ae
add slow_tests marker to pytest_config.py
halungge Dec 1, 2023
64c001d
get limited_area from grid savepoint metadata
halungge Dec 1, 2023
3a43679
add rtol to diffusion asserts
halungge Dec 1, 2023
d09dfc3
merge from main
halungge Dec 1, 2023
1a9243b
test_velocity_advection.py: fix order of reference and values in dall…
halungge Dec 1, 2023
6ac6d0e
merge main
halungge Dec 5, 2023
450761e
add TODOs for released tolerances
halungge Dec 5, 2023
93c6a64
pre-commit fixes
halungge Dec 5, 2023
ef8be6d
cleanups
halungge Dec 5, 2023
b54f476
fix import of experiment fixture in model/common
halungge Dec 5, 2023
b96b7b4
fix nsubsteps in multistep test
halungge Dec 5, 2023
3149f36
pre-commit
halungge Dec 5, 2023
c720607
merge main
halungge Dec 7, 2023
23e78e4
merge main, resolve conflicts
halungge Dec 15, 2023
7425855
fix merge errors:
halungge Dec 15, 2023
f5fd1e7
fix merge errors: test_velocity_advection.py
halungge Dec 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
import logging
import math
import sys
from collections import namedtuple
from dataclasses import InitVar, dataclass, field
from enum import Enum
from typing import Final, Optional
Expand Down Expand Up @@ -43,8 +42,8 @@
zero_field,
)
from icon4py.model.atmosphere.diffusion.stencils.apply_diffusion_to_vn import apply_diffusion_to_vn
from icon4py.model.atmosphere.diffusion.stencils.apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance import (
apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance,
from icon4py.model.atmosphere.diffusion.stencils.apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence import (
apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence,
)
from icon4py.model.atmosphere.diffusion.stencils.calculate_diagnostic_quantities_for_turbulence import (
calculate_diagnostic_quantities_for_turbulence,
Expand Down Expand Up @@ -81,11 +80,15 @@
from icon4py.model.common.states.prognostic_state import PrognosticState


"""
Diffusion module ported from ICON mo_nh_diffusion.f90.

Supports only diffusion_type (=hdiff_order) 5 from the diffusion namelist.
"""

# flake8: noqa
log = logging.getLogger(__name__)

VectorTuple = namedtuple("VectorTuple", "x y")

cached_backend = run_gtfn_cached
compiled_backend = run_gtfn
imperative_backend = run_gtfn_imperative
Expand All @@ -95,6 +98,7 @@
class DiffusionType(int, Enum):
"""
Order of nabla operator for diffusion.

Note: Called `hdiff_order` in `mo_diffusion_nml.f90`.
Note: We currently only support type 5.
"""
Expand All @@ -106,6 +110,23 @@ class DiffusionType(int, Enum):
SMAGORINSKY_4TH_ORDER = 5 #: Smagorinsky diffusion with fourth-order background diffusion


class TurbulenceShearForcingType(int, Enum):
"""
Type of shear forcing used in turbulance.

Note: called `itype_sher` in `mo_turbdiff_nml.f90`
"""

VERTICAL_OF_HORIZONTAL_WIND = 0 #: only vertical shear of horizontal wind
VERTICAL_HORIZONTAL_OF_HORIZONTAL_WIND = (
1 #: as `VERTICAL_ONLY` plus horizontal shar correction
)
VERTICAL_HORIZONTAL_OF_HORIZONTAL_VERTICAL_WIND = (
2 #: as `VERTICAL_HORIZONTAL_OF_HORIZONTAL_WIND` plus shear form vertical velocity
)
VERTICAL_HORIZONTAL_OF_HORIZONTAL_WIND_LTHESH = 3 #: same as `VERTICAL_HORIZONTAL_OF_HORIZONTAL_WIND` but scaling of coarse-grid horizontal shear production term with 1/sqrt(Ri) (if LTKESH = TRUE)


class DiffusionConfig:
"""
Contains necessary parameter to configure a diffusion run.
Expand All @@ -132,11 +153,14 @@ def __init__(
smagorinski_scaling_factor: float = 0.015,
n_substeps: int = 5,
zdiffu_t: bool = True,
thslp_zdiffu: float = 0.025,
thhgtd_zdiffu: float = 200.0,
hdiff_rcf: bool = True,
velocity_boundary_diffusion_denom: float = 200.0,
temperature_boundary_diffusion_denom: float = 135.0,
max_nudging_coeff: float = 0.02,
nudging_decay_rate: float = 2.0,
shear_type: TurbulenceShearForcingType = TurbulenceShearForcingType.VERTICAL_OF_HORIZONTAL_WIND,
):
"""Set the diffusion configuration parameters with the ICON default values."""
# parameters from namelist diffusion_nml
Expand Down Expand Up @@ -176,13 +200,18 @@ def __init__(
self.hdiff_w_efdt_ratio: float = hdiff_w_efdt_ratio

#: Scaling factor for Smagorinsky diffusion at height hdiff_smag_z and below
#: Called `hdiff_smag_fac` inmo_diffusion_nml.f90
#: Called `hdiff_smag_fac` in mo_diffusion_nml.f90
self.smagorinski_scaling_factor: float = smagorinski_scaling_factor

#: If True, apply truly horizontal temperature diffusion over steep slopes
#: Called 'l_zdiffu_t' in mo_nonhydrostatic_nml.f90
self.apply_zdiffusion_t: bool = zdiffu_t

#:slope threshold (temperature diffusion): is used to build up an index list for application of truly horizontal diffusion in mo_vertical_grid.f89
self.thslp_zdiffu = thslp_zdiffu
#: threshold [m] for height difference between adjacent grid points, defaults to 200m (temperature diffusion)
self.thhgtd_zdiffu = thhgtd_zdiffu

# from other namelists:
# from parent namelist mo_nonhydrostatic_nml

Expand Down Expand Up @@ -219,6 +248,10 @@ def __init__(
#: Called `nudge_efold_width` in mo_interpol_nml.f90
self.nudge_efold_width: float = nudging_decay_rate

#: Type of shear forcing used in turbulence
#: Called itype_shear in `mo_turbdiff_nml.f90
self.shear_type = shear_type

self._validate()

def _validate(self):
Expand All @@ -237,8 +270,15 @@ def _validate(self):
self.apply_to_temperature = True
self.apply_to_horizontal_wind = True

if not self.apply_zdiffusion_t:
raise NotImplementedError("zdiffu_t = False is not implemented (leaves out stencil_15)")
if self.shear_type not in (
TurbulenceShearForcingType.VERTICAL_OF_HORIZONTAL_WIND,
TurbulenceShearForcingType.VERTICAL_HORIZONTAL_OF_HORIZONTAL_VERTICAL_WIND,
):
raise NotImplementedError(
f"Turbulence Shear only {TurbulenceShearForcingType.VERTICAL_OF_HORIZONTAL_WIND} "
f"and {TurbulenceShearForcingType.VERTICAL_HORIZONTAL_OF_HORIZONTAL_VERTICAL_WIND} "
f"implemented"
)

@functools.cached_property
def substep_as_float(self):
Expand Down Expand Up @@ -618,7 +658,7 @@ def _do_diffusion_step(

# 2. HALO EXCHANGE -- CALL sync_patch_array_mult u_vert and v_vert
log.debug("communication rbf extrapolation of vn - start")
h = self._exchange.exchange_and_wait(VertexDim, self.u_vert, self.v_vert)
self._exchange.exchange_and_wait(VertexDim, self.u_vert, self.v_vert)
log.debug("communication rbf extrapolation of vn - end")

log.debug("running stencil 01(calculate_nabla2_and_smag_coefficients_for_vn): start")
Expand Down Expand Up @@ -649,30 +689,38 @@ def _do_diffusion_step(
},
)
log.debug("running stencil 01 (calculate_nabla2_and_smag_coefficients_for_vn): end")
log.debug("running stencils 02 03 (calculate_diagnostic_quantities_for_turbulence): start")
calculate_diagnostic_quantities_for_turbulence.with_backend(backend)(
kh_smag_ec=self.kh_smag_ec,
vn=prognostic_state.vn,
e_bln_c_s=self.interpolation_state.e_bln_c_s,
geofac_div=self.interpolation_state.geofac_div,
diff_multfac_smag=self.diff_multfac_smag,
wgtfac_c=self.metric_state.wgtfac_c,
div_ic=diagnostic_state.div_ic,
hdef_ic=diagnostic_state.hdef_ic,
horizontal_start=cell_start_nudging,
horizontal_end=cell_end_local,
vertical_start=1,
vertical_end=klevels,
offset_provider={
"C2E": self.grid.get_offset_provider("C2E"),
"C2CE": self.grid.get_offset_provider("C2CE"),
"Koff": KDim,
},
)
log.debug("running stencils 02 03 (calculate_diagnostic_quantities_for_turbulence): end")

# HALO EXCHANGE IF (discr_vn > 1) THEN CALL sync_patch_array -> false for MCH
if (
self.config.shear_type
>= TurbulenceShearForcingType.VERTICAL_HORIZONTAL_OF_HORIZONTAL_WIND
):
log.debug(
"running stencils 02 03 (calculate_diagnostic_quantities_for_turbulence): start"
)
calculate_diagnostic_quantities_for_turbulence.with_backend(backend)(
kh_smag_ec=self.kh_smag_ec,
vn=prognostic_state.vn,
e_bln_c_s=self.interpolation_state.e_bln_c_s,
geofac_div=self.interpolation_state.geofac_div,
diff_multfac_smag=self.diff_multfac_smag,
wgtfac_c=self.metric_state.wgtfac_c,
div_ic=diagnostic_state.div_ic,
hdef_ic=diagnostic_state.hdef_ic,
horizontal_start=cell_start_nudging,
horizontal_end=cell_end_local,
vertical_start=1,
vertical_end=klevels,
offset_provider={
"C2E": self.grid.get_offset_provider("C2E"),
"C2CE": self.grid.get_offset_provider("C2CE"),
"Koff": KDim,
},
)
log.debug(
"running stencils 02 03 (calculate_diagnostic_quantities_for_turbulence): end"
)

# HALO EXCHANGE IF (discr_vn > 1) THEN CALL sync_patch_array
# TODO (magdalena) move this up and do asynchronous exchange
if self.config.type_vn_diffu > 1:
log.debug("communication rbf extrapolation of z_nable2_e - start")
self._exchange.exchange_and_wait(EdgeDim, self.z_nabla2_e)
Expand Down Expand Up @@ -731,17 +779,18 @@ def _do_diffusion_step(
handle_edge_comm = self._exchange.exchange(EdgeDim, prognostic_state.vn)

log.debug(
"running stencils 07 08 09 10 (apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance): start"
"running stencils 07 08 09 10 (apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence): start"
)
# TODO (magdalena) get rid of this copying. So far passing an empty buffer instead did not verify?
copy_field.with_backend(backend)(prognostic_state.w, self.w_tmp, offset_provider={})
apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance.with_backend(backend)(
apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence.with_backend(backend)(
area=self.cell_params.area,
geofac_n2s=self.interpolation_state.geofac_n2s,
geofac_grg_x=self.interpolation_state.geofac_grg_x,
geofac_grg_y=self.interpolation_state.geofac_grg_y,
w_old=self.w_tmp,
w=prognostic_state.w,
type_shear=int32(self.config.shear_type.value),
dwdx=diagnostic_state.dwdx,
dwdy=diagnostic_state.dwdy,
diff_multfac_w=self.diff_multfac_w,
Expand All @@ -762,7 +811,7 @@ def _do_diffusion_step(
},
)
log.debug(
"running stencils 07 08 09 10 (apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance): end"
"running stencils 07 08 09 10 (apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence): end"
)

log.debug(
Expand Down Expand Up @@ -807,29 +856,30 @@ def _do_diffusion_step(
log.debug(
"running stencil 15 (truly_horizontal_diffusion_nabla_of_theta_over_steep_points): start"
)
truly_horizontal_diffusion_nabla_of_theta_over_steep_points.with_backend(backend)(
mask=self.metric_state.mask_hdiff,
zd_vertoffset=self.metric_state.zd_vertoffset,
zd_diffcoef=self.metric_state.zd_diffcoef,
geofac_n2s_c=self.interpolation_state.geofac_n2s_c,
geofac_n2s_nbh=self.interpolation_state.geofac_n2s_nbh,
vcoef=self.metric_state.zd_intcoef,
theta_v=prognostic_state.theta_v,
z_temp=self.z_temp,
horizontal_start=cell_start_nudging,
horizontal_end=cell_end_local,
vertical_start=0,
vertical_end=klevels,
offset_provider={
"C2CEC": self.grid.get_offset_provider("C2CEC"),
"C2E2C": self.grid.get_offset_provider("C2E2C"),
"Koff": KDim,
},
)
if self.config.apply_zdiffusion_t:
truly_horizontal_diffusion_nabla_of_theta_over_steep_points.with_backend(backend)(
mask=self.metric_state.mask_hdiff,
zd_vertoffset=self.metric_state.zd_vertoffset,
zd_diffcoef=self.metric_state.zd_diffcoef,
geofac_n2s_c=self.interpolation_state.geofac_n2s_c,
geofac_n2s_nbh=self.interpolation_state.geofac_n2s_nbh,
vcoef=self.metric_state.zd_intcoef,
theta_v=prognostic_state.theta_v,
z_temp=self.z_temp,
horizontal_start=cell_start_nudging,
horizontal_end=cell_end_local,
vertical_start=0,
vertical_end=klevels,
offset_provider={
"C2CEC": self.grid.get_offset_provider("C2CEC"),
"C2E2C": self.grid.get_offset_provider("C2E2C"),
"Koff": KDim,
},
)

log.debug(
"running fused stencil 15 (truly_horizontal_diffusion_nabla_of_theta_over_steep_points): end"
)
log.debug(
"running fused stencil 15 (truly_horizontal_diffusion_nabla_of_theta_over_steep_points): end"
)
log.debug("running stencil 16 (update_theta_and_exner): start")
update_theta_and_exner.with_backend(backend)(
z_temp=self.z_temp,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,13 @@


@field_operator
def _apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance(
def _apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence(
area: Field[[CellDim], wpfloat],
geofac_n2s: Field[[CellDim, C2E2CODim], wpfloat],
geofac_grg_x: Field[[CellDim, C2E2CODim], wpfloat],
geofac_grg_y: Field[[CellDim, C2E2CODim], wpfloat],
w_old: Field[[CellDim, KDim], wpfloat],
type_shear: int32,
dwdx: Field[[CellDim, KDim], vpfloat],
dwdy: Field[[CellDim, KDim], vpfloat],
diff_multfac_w: wpfloat,
Expand All @@ -50,11 +51,14 @@ def _apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance(
Field[[CellDim, KDim], vpfloat],
]:
vert_idx = broadcast(vert_idx, (CellDim, KDim))

dwdx, dwdy = where(
int32(0) < vert_idx,
_calculate_horizontal_gradients_for_turbulence(w_old, geofac_grg_x, geofac_grg_y),
(dwdx, dwdy),
dwdx, dwdy = (
where(
int32(0) < vert_idx,
_calculate_horizontal_gradients_for_turbulence(w_old, geofac_grg_x, geofac_grg_y),
(dwdx, dwdy),
)
if type_shear == int32(2)
else (dwdx, dwdy)
)

z_nabla2_c = _calculate_nabla2_for_w(w_old, geofac_n2s)
Expand All @@ -78,13 +82,14 @@ def _apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance(


@program
def apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance(
def apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence(
area: Field[[CellDim], wpfloat],
geofac_n2s: Field[[CellDim, C2E2CODim], wpfloat],
geofac_grg_x: Field[[CellDim, C2E2CODim], wpfloat],
geofac_grg_y: Field[[CellDim, C2E2CODim], wpfloat],
w_old: Field[[CellDim, KDim], wpfloat],
w: Field[[CellDim, KDim], wpfloat],
type_shear: int32,
dwdx: Field[[CellDim, KDim], vpfloat],
dwdy: Field[[CellDim, KDim], vpfloat],
diff_multfac_w: wpfloat,
Expand All @@ -99,12 +104,13 @@ def apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance(
vertical_start: int32,
vertical_end: int32,
):
_apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulance(
_apply_diffusion_to_w_and_compute_horizontal_gradients_for_turbulence(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for changing this!

area,
geofac_n2s,
geofac_grg_x,
geofac_grg_y,
w_old,
type_shear,
dwdx,
dwdy,
diff_multfac_w,
Expand Down
Loading