An opinionated way of setting up configuration for data science projects.
Gamma config provides a standard and flexible way of implementing application configuration primarily based on YAML files. It promotes best practices by:
- simplify separation of code and config data;
- breaking large, complex configuration into multiple files;
- providing a way to write environment-aware config files;
- facilitating security best-practices, like proper secrets management;
- providing a immutable central global object storing all contextual data.
Another benefit of a standard config mechanism is allowing Gamma extension libraries to provide configuration in a common way.
- Configuration expressed as a set of YAML files (by defaul) inside a
config
folder in the root of the project. - Multiple YAML files merged following simple rules. Simple file ordering convention using two digit prefixes.
- Builtin support for environment specific parameters (production, development, etc.)
- Support for
.env
files viapython-dotenv
, including.local.env
that can be added to.gitignore
- Dynamic evaluation via YAML tags. For instance, the
!ref
tag allow you to reference other parameters in any config file. - Custom tag support via simple and cool multiple dispatch mechanism.
- Round-trip dump of config back into YAML. Support for hiding sensitive data on dump.
- Simplified key access via dot (
.
). Eg. forconfig: {foo: {bar: 100}}
, this is True:config.foo.bar == 100
Click here to view the full documentation
Using pip:
pip install gamma-config
In most cases, you'll want to use the !j2
tag to interpolate values using Jinja2.
This requires manually installing the jinja2
package or using the jinja2
extras.
pip install gamma-config[jinja2]
You must install pydantic
if using the structured configuration feature.
pip install gamma-config[pydantic]
The package comes with "scaffolding" to help you get started. In your project folder:
python -m gamma.config.scaffold
Remove the sample files, then create yourself a config/20-myconfig.yaml
file
with the contents:
foo: 1
user: !env USER
To access the config from within your Python program:
import os
from gamma.config import get_config
def run():
# it's safe and efficient to call this multiple times
config = get_config()
# get static value using the dict keys or attribute access
assert config["foo"] == 1
assert config.foo == 1
# get dynamic variables
assert config["user"] == os.getenv("USER")
assert config.user == os.getenv("USER")
Most of the magic happen via tags. Look at the documentation for info on the built-in tags available.
-
We DEPRECATED
!py
and!obj
tags, replacing them by the cleaner/simpler!call
tag. Those are schedule to be removed by release 1.0. -
We DEPRECATED "Structured configuration" support, We recommend directly using Pydantic's V2 discriminated unions as an alternative.
-
By default,
!env
,!expr
and!call
will not dump the contents on ato_yaml
call. We provide!env:dump
,!expr:dump
,!call:dump
to force dumping. Tag!env_secret
is still supported but use is discouraged. -
We have
!j2:secret
in addition to!j2_secret
for consistency. Use of!j2_secret
is discouraged.
- We've DEPRECATED our homegrown multiple dispatch system
gamma.dispatch
, replacing it byplum
. Unless you were usinggamma.dispatch
directly, or extending via custom tags, no need to worry. Thegamma.dispatch
package will be removed by release 1.0.
- We have a new home in https://github.com/cjalmeida/gamma-config !
- Strict support for YAML 1.2 Core Schema.
In practice, unquoted ISO8610 dates (eg.
2022-12-20
) won't get converted todatetime.date
ordatetime.datetime
objects. Use!date
or!datetime
if needed. .env
files are loaded automatically and get precedence overconfig.env
andconfig.local.env
.- Use of
config.env
andconfig.local.env
is deprecated. - Default scaffolded
include_folder
interpretENVIRONMENT
variable string likefoo bar
as two separate environment subfolders. - (dispatch)
Val
arguments passed as class (eg.foo(Val['bar'])
) will be converted to instance, as if it were calledfoo(Val['bar']())
- The
!py:<module>:<func>
will no longer a singleNone
argument
- Support for YAML Anchors and Aliases
- When using the dot (
.
) syntax, missing values raiseAttributeError
instead of returning a false-y object. - Dropped support for Python 3.7
- We're now in PyPI!
- Options for installing extra dependencies (eg.
jinja2
,pydantic
)