Add new CLIMAParameters interface

Enable get_parameter_values to deal with array-valued parameters

- `get_parameter_values` extracts the requested parameter values and returns
  them as an array. The returned array contains the values corresponding
  to the given list of parameter names (unless a single scalar  parameter is
  requested, in which case the function returns a float), regardless of whether
  these parameters are all of the same type.
- E.g., given a list of parameter names ["scalar_param", "array_param"] with
  `scalar_param` having a scalar value and `array_param` being array-valued,
  the first element of the returned array will be a float, and the second
  element will be an array
- Added more tests to `toml_consistency.jl` to check if parsing, extracting,
  and writing of array-valued parameters works. Also added a test for
  `merge_override_default_values`

Enforce types and move `array_parameters.toml` to test folder

Add V0 of TOML file parsing for UQ parameters

Add functionality to write parameter ensembles to toml files

Move docstrings above functions

Add improvements and updates

- save_parameter_ensemble now creates a separate subdirectory for
  each ensemble member, while the name of the saved parameter file
  after each ensemble Kalman update is the same for all members.
  Example:
    iteration_01/member_01/test_parameters.toml
    iteration_01/member_02/test_parameters.toml
    etc
- correct saving of multidimensional parameters using parameter slices

Move dict version of `write_log_file` from file_parsing.jl to file_parsing_uq.jl

Within the CES API, `write_log_file` will take a parameter dictionary as input
(rather than a parameter struct containing a parameter dictionary, as in the
running-the-climate-model API). Since the two APIs will go separate ways in the
future, the CES version of `write_log_file` should be located in
`file_parsing_uq.jl`.

added typo-checks for overrides, provides warnings or error on parameter logs

slightly more revealing test

documentation and guides

Modify parsing of UQ parameters to work with new `ParameterDistribution` API

Make tests work

Add regularization flags

Revise based on Ollie's comments, and remove print commands from tests

moved toml and old files into directories

parameter box example (as test set)

Improve create_parameter_struct interface

Project.toml

@@ -4,6 +4,8 @@ authors = ["Charles Kawczynski <[email protected]>"]
 version = "0.4.3"
 
 [deps]
+DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
+TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [compat]

docs/make.jl

@@ -2,6 +2,8 @@ using CLIMAParameters, Documenter
 
 pages = Any[
     "Home" => "index.md",
+    "TOML file interface" => "toml.md",
+    "Parameter structures" => "parameter_structs.md",
     "API" => "API.md",
 ]
 

docs/src/API.md

@@ -4,6 +4,33 @@
 CurrentModule = CLIMAParameters
 ```
 
+## Parameter struct
+
+```@docs
+ParamDict
+```
+
+## File parsing and parameter logging
+
+### User facing functions:
+```@docs
+create_parameter_struct
+get_parameter_values!
+get_parameter_values
+float_type
+log_parameter_information
+```
+
+### Internal functions:
+```@docs
+iterate_alias
+log_component!(param_set::ParamDict,names,component)
+get_values(param_set::ParamDict, names)
+check_override_parameter_usage
+write_log_file
+merge_override_default_values
+```
+
 ## Types
 
 ```@docs

docs/src/parameter_structs.md

@@ -0,0 +1,163 @@
+# Parameter Structures
+
+Parameters are stored in objects that reflect the model component construction. Definitions should be inserted into the model component source code
+
+## An example from `Thermodynamics.jl` 
+
+### In the user-facing driver file
+```julia
+import CLIMAParameters
+import Thermodynamics
+parameter_struct = CLIMAParameters.create_parameter_struct(;dict_type="alias") 
+thermo_params = Thermodynamics.ThermodynamicsParameters(parameter_struct)
+```
+
+### In the source code for `Thermodynamics.jl`
+
+```julia
+struct ThermodynamicsParameters{FT}
+    ...
+    R_d::FT
+    gas_constant::FT
+    molmass_dryair::FT
+end
+```
+- The struct is parameterized by `{FT}` which is a user-determined float precision
+- Only relevant parameters used in `Thermodynamics` are stored here.
+
+The constructor is as follows
+```julia
+function ThermodynamicsParameters(parameter_struct)
+
+    # Used in thermodynamics, from parameter file
+    aliases = [ ..., "gas_constant", "molmass_dryair"]
+
+    (gas_constant, molmass_dryair,) = CLIMAParameters.get_parameter_values!(
+        param_struct,
+        aliases,
+        "Thermodynamics",
+    )
+
+    # derived parameters from parameter file
+    R_d = gas_constant / molmass_dryair
+
+    return ThermodynamicsParameters{
+        CLIMAParameters.float_type(param_struct)}(...,R_d, gas_constant, molmass_dryair)
+end
+```
+
+- The constructor takes in a `parameter_struct` produced from reading the TOML file
+- We list the aliases of parameters required by `Thermodynamics.jl` 
+- We obtain parameters by calling the function `CLIMAParameters.get_parameter_values!(parameter_struct,aliases,component_name)` The `component_name` is a string used for the parameter log.
+- We then create any `derived parameters` e.g. commonly used simple functions of parameters that are treated as parameters. here we create the dry air gas constant `R_d`
+- We end by returning creating the `ThermodynamicsParameters{FT}`.
+
+
+## An example with modular components from `CloudMicrophysics.jl`
+
+### In the user-facing driver file
+
+Here we build a `CloudMicrophysics` parameter set. In this case, the user wishes to use a
+0-moment microphysics parameterization scheme.
+```julia
+import CLIMAParameters
+import Thermodynamics
+import CloudMicrophysics
+
+#load defaults
+parameter_struct = CLIMAParameters.create_parameter_struct(;dict_type="alias")
+
+#build the low level parameter set
+param_therm = Thermodynamics.ThermodynamicsParameters(parameter_struct)
+param_0M = CloudMicrophysics.Microphysics_0M_Parameters(parameter_struct)
+
+#build the hierarchical parameter set
+parameter_set = CloudMicrophysics.CloudMicrophysicsParameters(
+    parameter_struct,
+    param_0M,
+    param_therm
+)
+```
+!!! note
+    The exact APIs here are subject to change
+
+### In the source code for `CloudMicrophysics.jl`
+
+Build the different options for a Microphysics parameterizations
+```julia
+abstract type AbstractMicrophysicsParameters end
+struct NoMicrophysicsParameters <: AbstractMicrophysicsParameters end 
+struct Microphysics_0M_Parameters{FT} <: AbstractMicrophysicsParameters
+    τ_precip::FT
+    qc_0::FT
+    S_0::FT
+end
+struct Microphysics_1M_Parameters{FT} <: AbstractMicrophysicsParameters
+    ...
+end
+```
+We omit their constructors (see above). The `CloudMicrophysics` parameter set is built likewise
+
+```julia
+struct CloudMicrophysicsParameters{FT, AMPS <: AbstractMicrophysicsParameters}
+    K_therm::FT
+    ...
+    MPS::AMPS
+    TPS::ThermodynamicsParameters{FT}
+end
+
+
+function CloudMicrophysicsParameters(
+    param_set,
+    MPS::AMPS,
+    TPS::ThermodynamicsParameters{FT},
+) where {FT, AMPS <: AbstractMicrophysicsParameters}
+
+    aliases = [ "K_therm", ... ]
+
+    ( K_therm,... ) = CLIMAParameters.get_parameter_values!(
+        param_set,
+        aliases,
+        "CloudMicrophysics",
+    )
+
+    #derived parameters 
+    ...
+
+    return CloudMicrophysicsParameters{
+        CLIMAParameters.float_type(param_set), AMPS}(
+            K_therm,
+            ...     
+            MPS,
+            TPS,
+        )
+end
+```
+
+## Calling parameters from `src`
+
+
+When building the model components, parameters are extracted by calling `param_set.name` or `param_set.alias` (currently)
+```julia
+function example_cloudmicrophysics_func(param_set::CloudMicrophysicsParameters,...)
+    K_therm = param_set.K_therm
+    ...
+end
+```
+When calling functions from dependent packages, simply pass the relevant lower_level parameter struct
+```julia
+function example_cloudmicrophysics_func(param_set::CloudMicrophysicsParameters,...)
+    thermo_output = Thermodynamics.thermo_function(param_set.TPS,...)
+    cm0_output = Microphysics_0m.microphys_function(param_set.MPS,...)
+    ...
+end
+```
+These functions should be written with this in mind (dispatching)
+```julia
+microphys_function(param_set::Microphysics_0M_parameters,...)
+   qc_0 = param_set.qc_0
+   ...
+end
+```
+
+

docs/src/toml.md

@@ -0,0 +1,128 @@
+# The TOML parameter file interface
+
+The complete user interface consists of two files in `TOML` format
+1. A user-defined experiment file - in the local experiment directory
+2. A defaults file - in `src/` directory of `ClimaParameters.jl`
+
+## Parameter style-guide 
+
+A parameter is determined by its unique name. It has possible attributes
+1. `alias` 
+2. `value`  
+3. `type`
+4. `description`
+5. `prior`
+6. `transformation`
+
+!!! warn
+    Currently we only support `float` and `array{float}` types. (option-type flags and string switches are not considered CLIMAParameters.)
+
+### Minimal parameter requirement to run in CliMA
+
+```TOML
+[molar_mass_dry_air] 
+value = 0.03
+type = "float"
+```    
+
+### A more informative parameter (e.g. found in the defaults file) 
+
+```TOML
+[molar_mass_dry_air] 
+alias = "molmass_dryair"
+value = 0.02897
+type = "float"
+description = "Molecular weight dry air (kg/mol)"
+```
+
+### A more complex parameter for calibration
+
+```TOML
+[neural_net_entrainment] 
+alias = "c_gen"
+value = [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
+type = "array"
+description = "NN weights to represent the non-dimensional entrainment function"
+prior = "MvNormal(0,I)"
+```
+
+### Interaction of the files
+
+On read an experiment file, the default file is also read and any duplicate parameter attributes are overwritten 
+e.g. If the minimal example above was loaded from an experiment file, and the informative example above was in the defaults file, then the loaded parameter would look as follows: 
+``` TOML
+[molar_mass_dry_air] 
+alias = "molmass_dryair"
+value = 0.03
+type = "float"
+description = "Molecular weight dry air (kg/mol)"
+```
+Here, the `value` field has been overwritten by the experiment value
+
+## File and parameter interaction on with CliMA
+
+`ClimaParameters.jl` provides several methods to parse, merge, and log parameter information. 
+
+
+### Loading from file
+We provide the following methods to load parameters from file
+```julia
+create_parameter_struct(;override_filepath, default_filepath; dict_type="alias",value_type=Float64)
+create_parameter_struct(;override_filepath ; dict_type="alias",value_type=Float64) 
+create_parameter_struct(; dict_type="name",value_type=Float64) 
+```
+- The `dict_type = "name"` or `"alias"` determines the method of lookup of parameters (by `name` or by `alias` attributes).
+- The  `value_type` defines the requested precision of the returned parameters (e.g. `Float64` or `Float32`)
+
+Typical usage involves passing the local parameter file
+```julia
+import CLIMAParameters
+local_exp_file = joinpath(@__DIR__,"local_exp_parameters.toml")
+parameter_struct = CLIMAParameters.create_parameter_struct(;local_exp_file) 
+```
+If no file is passed it will use only the defaults from `ClimaParameters.jl` (causing errors if required parameters are not within this list).
+
+!!! note
+    Currently we search by the `alias` field (`dict_type="alias"` by default), so all parameters need an `alias` field, if in doubt, set alias and name to match the current code name convention
+
+The parameter struct is then used to build the codebase (see relevant Docs page)
+
+### Logging parameters
+
+Once the CliMA components are built, it is important to log the parameters. We provide the following methodd
+```julia
+log_parameter_information(parameter_struct, filepath; warn_else_error="warn")
+```
+
+Typical usage will be after building components and before running
+```julia
+import Thermodynamics
+therm_params = Thermodynamics.ThermodynamicsParameters(parameter_struct)
+#... build(thermodynamics model,therm_params)
+
+log_file = joinpath(@__DIR__,"parameter_log.toml")
+CLIMAParameters.log_parameter_information(parameter_struct,log_file)
+
+# ... run(thermodynamics_model)
+```
+
+This function performs two tasks
+1. It writes a parameter log file to `log_file`.
+2. It performs parameter sanity checks.
+
+Continuing our previous example, imagine `molar_mass_dry_air` was extracted in `ThermodynamicsParameters`. Then the log file will contain:
+``` TOML
+[molar_mass_dry_air] 
+alias = "molmass_dryair"
+value = 0.03
+type = "float"
+description = "Molecular weight dry air (kg/mol)"
+used_in = ["Thermodynamics"]
+```
+The additional attribute `used_in` displays every CliMA component that used this parameter. 
+
+!!! note
+    Log files are written in TOML format, and can be read back into the model
+
+!!! warn
+    It is assumed that all parameters in the local experiment file should be used, if not a warning is displayed when calling `log_parameter_information`. this is upgraded to an error exception by changing `warn_else_error`

src/CLIMAParameters.jl

@@ -3,6 +3,7 @@ module CLIMAParameters
 export AbstractParameterSet
 export AbstractEarthParameterSet
 
+include("file_parsing.jl")
 include("types.jl")
 include("UniversalConstants.jl")