[WIP] pilot parameter refactor

Project.toml

@@ -4,6 +4,8 @@ authors = ["Charles Kawczynski <[email protected]>"]
 version = "0.4.3"
 
 [deps]
+DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
+TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [compat]

docs/make.jl

@@ -1,6 +1,11 @@
 using CLIMAParameters, Documenter
 
-pages = Any["Home" => "index.md", "API" => "API.md"]
+pages = Any[
+    "Home" => "index.md",
+    "TOML file interface" => "toml.md",
+    "Parameter structures" => "parameter_structs.md",
+    "API" => "API.md",
+]
 
 mathengine = MathJax(
     Dict(

docs/src/API.md

@@ -4,6 +4,34 @@
 CurrentModule = CLIMAParameters
 ```
 
+## Parameter struct
+
+```@docs
+AbstractParamDict
+ParamDict
+AliasParamDict
+```
+
+## File parsing and parameter logging
+
+### User facing functions:
+```@docs
+create_parameter_struct
+get_parameter_values!
+get_parameter_values
+float_type
+log_parameter_information
+```
+
+### Internal functions:
+```@docs
+log_component!
+get_values
+check_override_parameter_usage
+write_log_file
+merge_override_default_values
+```
+
 ## Types
 
 ```@docs

docs/src/parameter_structs.md

@@ -0,0 +1,170 @@
+# Parameter Structures
+
+Parameters are stored in objects that reflect the model component construction. Definitions should be inserted into the model component source code
+
+## An example from `Thermodynamics.jl` 
+
+### In the user-facing driver file
+```julia
+import CLIMAParameters
+import Thermodynamics
+
+parameter_struct = CLIMAParameters.create_parameter_struct(;dict_type="alias") 
+thermo_params = Thermodynamics.ThermodynamicsParameters(parameter_struct)
+```
+
+### In the source code for `Thermodynamics.jl`
+
+```julia
+Base.@kwdef struct ThermodynamicsParameters{FT}
+    gas_constant::FT
+    molmass_dryair::FT
+    ...
+    R_d::FT
+end
+```
+- The struct is parameterized by `{FT}` which is a user-determined float precision
+- Only relevant parameters used in `Thermodynamics` are stored here.
+- A keyword based `struct` so we do not rely on parameter order
+
+The constructor is as follows
+```julia
+function ThermodynamicsParameters(parameter_struct)
+
+    # Used in thermodynamics, from parameter file
+    aliases = [ ..., "gas_constant", "molmass_dryair"]
+
+    param_pairs = CLIMAParameters.get_parameter_values!(
+        param_struct,
+        aliases,
+        "Thermodynamics",
+    )
+    nt = (; param_pairs...)
+
+    # derived parameters from parameter file
+    R_d = nt.gas_constant / nt.molmass_dryair
+
+    FT = CP.float_type(param_struct)
+    return ThermodynamicsParameters{FT}(; nt..., R_d)
+end
+```
+
+- The constructor takes in a `parameter_struct` produced from reading the TOML file
+- We list the aliases of parameters required by `Thermodynamics.jl`
+- We obtain parameters (in the form of a list of (alias,value) Pairs) from `get_parameter_values!(parameter_struct,aliases,component_name)` The `component_name` is a string used for the parameter log.
+- We convert to namedtuple for ease of extraction
+- We create any `derived parameters` i.e. commonly used simple functions of parameters that are treated as parameters. here we create the dry air gas constant `R_d`
+- We return the `ThermodynamicsParameters{FT}`, where FT is an enforced float type (e.g. single or double precision)
+
+
+## An example with modular components from `CloudMicrophysics.jl`
+
+### In the user-facing driver file
+
+Here we build a `CloudMicrophysics` parameter set. In this case, the user wishes to use a
+0-moment microphysics parameterization scheme.
+```julia
+import CLIMAParameters
+import Thermodynamics
+import CloudMicrophysics
+
+#load defaults
+parameter_struct = CLIMAParameters.create_parameter_struct(; dict_type="alias")
+
+#build the low level parameter set
+param_therm = Thermodynamics.ThermodynamicsParameters(parameter_struct)
+param_0M = CloudMicrophysics.Microphysics_0M_Parameters(parameter_struct)
+
+#build the hierarchical parameter set
+parameter_set = CloudMicrophysics.CloudMicrophysicsParameters(
+    parameter_struct,
+    param_0M,
+    param_therm
+)
+```
+!!! note
+    The exact APIs here are subject to change
+
+### In the source code for `CloudMicrophysics.jl`
+
+Build the different options for a Microphysics parameterizations
+```julia
+abstract type AbstractMicrophysicsParameters end
+struct NoMicrophysicsParameters <: AbstractMicrophysicsParameters end
+Base.@kwdef struct Microphysics_0M_Parameters{FT} <: AbstractMicrophysicsParameters
+    τ_precip::FT
+    qc_0::FT
+    S_0::FT
+end
+Base.@kwdef struct Microphysics_1M_Parameters{FT} <: AbstractMicrophysicsParameters
+    ...
+end
+```
+We omit their constructors (see above). The `CloudMicrophysics` parameter set is built likewise
+
+```julia
+Base.@kwdef struct CloudMicrophysicsParameters{FT, AMPS <: AbstractMicrophysicsParameters}
+    K_therm::FT
+    ...
+    MPS::AMPS
+    TPS::ThermodynamicsParameters{FT}
+end
+
+
+function CloudMicrophysicsParameters(
+    parameter_struct,
+    MPS::AMPS,
+    TPS::ThermodynamicsParameters{FT},
+) where {FT, AMPS <: AbstractMicrophysicsParameters}
+
+    aliases = [ "K_therm", ... ]
+
+    param_pairs  = CLIMAParameters.get_parameter_values!(
+        parameter_struct,
+        aliases,
+        "CloudMicrophysics",
+    )
+
+    nt = (; param_pairs...)
+    #derived parameters
+    ...
+    FT = CP.float_type(parameter_struct)
+
+    return CloudMicrophysicsParameters{FT, AMPS}(;
+            nt...,
+            ...
+            MPS,
+            TPS,
+        )
+end
+```
+
+## Calling parameters from `src`
+
+!!! note
+    The exact APIs here are subject to change
+
+When building the model components, parameters are extracted by calling `param_set.name` or `param_set.alias` (currently)
+```julia
+function example_cloudmicrophysics_func(param_set::CloudMicrophysicsParameters,...)
+    K_therm = param_set.K_therm
+    ...
+end
+```
+When calling functions from dependent packages, simply pass the relevant lower_level parameter struct
+```julia
+function example_cloudmicrophysics_func(param_set::CloudMicrophysicsParameters,...)
+    thermo_output = Thermodynamics.thermo_function(param_set.TPS,...)
+    cm0_output = Microphysics_0m.microphys_function(param_set.MPS,...)
+    ...
+end
+```
+These functions should be written with this in mind (dispatching)
+```julia
+function microphys_function(param_set::Microphysics_0M_parameters,...)
+   qc_0 = param_set.qc_0
+   ...
+end
+```
+
+

docs/src/toml.md

@@ -0,0 +1,128 @@
+# The TOML parameter file interface
+
+The complete user interface consists of two files in `TOML` format
+1. A user-defined experiment file - in the local experiment directory
+2. A defaults file - in `src/` directory of `ClimaParameters.jl`
+
+## Parameter style-guide
+
+A parameter is determined by its unique name. It has possible attributes
+1. `alias`
+2. `value`
+3. `type`
+4. `description`
+5. `prior`
+6. `transformation`
+
+!!! warn
+    Currently we only support `float` and `array{float}` types. (option-type flags and string switches are not considered CLIMAParameters.)
+
+### Minimal parameter requirement to run in CliMA
+
+```TOML
+[molar_mass_dry_air]
+value = 0.03
+type = "float"
+```
+
+### A more informative parameter (e.g. found in the defaults file)
+
+```TOML
+[molar_mass_dry_air]
+alias = "molmass_dryair"
+value = 0.02897
+type = "float"
+description = "Molecular weight dry air (kg/mol)"
+```
+
+### A more complex parameter for calibration
+
+```TOML
+[neural_net_entrainment]
+alias = "c_gen"
+value = [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
+type = "array"
+description = "NN weights to represent the non-dimensional entrainment function"
+prior = "MvNormal(0,I)"
+```
+
+### Interaction of the files
+
+On read an experiment file, the default file is also read and any duplicate parameter attributes are overwritten
+e.g. If the minimal example above was loaded from an experiment file, and the informative example above was in the defaults file, then the loaded parameter would look as follows:
+``` TOML
+[molar_mass_dry_air]
+alias = "molmass_dryair"
+value = 0.03
+type = "float"
+description = "Molecular weight dry air (kg/mol)"
+```
+Here, the `value` field has been overwritten by the experiment value
+
+## File and parameter interaction on with CliMA
+
+`ClimaParameters.jl` provides several methods to parse, merge, and log parameter information.
+
+
+### Loading from file
+We provide the following methods to load parameters from file
+```julia
+create_parameter_struct(Float64;override_filepath, default_filepath, dict_type="alias")
+create_parameter_struct(Float64;override_filepath ; dict_type="alias")
+create_parameter_struct(Float64; dict_type="name")
+```
+- The `dict_type = "name"` or `"alias"` determines the method of lookup of parameters (by `name` or by `alias` attributes).
+- The `Float64` (or `Float32`) defines the requested precision of the returned parameters.
+
+Typical usage involves passing the local parameter file
+```julia
+import CLIMAParameters
+local_exp_file = joinpath(@__DIR__,"local_exp_parameters.toml")
+parameter_struct = CLIMAParameters.create_parameter_struct(;local_exp_file)
+```
+If no file is passed it will use only the defaults from `ClimaParameters.jl` (causing errors if required parameters are not within this list).
+
+!!! note
+    Currently we search by the `alias` field (`dict_type="alias"` by default), so all parameters need an `alias` field, if in doubt, set alias and name to match the current code name convention
+
+The parameter struct is then used to build the codebase (see relevant Docs page)
+
+### Logging parameters
+
+Once the CliMA components are built, it is important to log the parameters. We provide the following methodd
+```julia
+log_parameter_information(parameter_struct, filepath; strict=false)
+```
+
+Typical usage will be after building components and before running
+```julia
+import Thermodynamics
+therm_params = Thermodynamics.ThermodynamicsParameters(parameter_struct)
+#... build(thermodynamics model,therm_params)
+
+log_file = joinpath(@__DIR__,"parameter_log.toml")
+CLIMAParameters.log_parameter_information(parameter_struct,log_file)
+
+# ... run(thermodynamics_model)
+```
+
+This function performs two tasks
+1. It writes a parameter log file to `log_file`.
+2. It performs parameter sanity checks.
+
+Continuing our previous example, imagine `molar_mass_dry_air` was extracted in `ThermodynamicsParameters`. Then the log file will contain:
+``` TOML
+[molar_mass_dry_air]
+alias = "molmass_dryair"
+value = 0.03
+type = "float"
+description = "Molecular weight dry air (kg/mol)"
+used_in = ["Thermodynamics"]
+```
+The additional attribute `used_in` displays every CliMA component that used this parameter.
+
+!!! note
+    Log files are written in TOML format, and can be read back into the model
+
+!!! warn
+    It is assumed that all parameters in the local experiment file should be used, if not a warning is displayed when calling `log_parameter_information`. this is upgraded to an error exception by changing `strict`

src/CLIMAParameters.jl

@@ -3,6 +3,7 @@ module CLIMAParameters
 export AbstractParameterSet
 export AbstractEarthParameterSet
 
+include("file_parsing.jl")
 include("types.jl")
 include("UniversalConstants.jl")