linting

.pre-commit-config.yaml

@@ -25,7 +25,7 @@ repos:
     rev: 22.3.0
     hooks:
       - id: black
-  - repo: https://github.com/pre-commit/mirrors-isort
-    rev: v4.3.21
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
     hooks:
     -   id: isort

docs/built-in-tags.md

@@ -49,7 +49,9 @@ level0:
   bar1: !j2 Custom = {{ custom_var }}
 ```
 
-The value of `foo1` is the string `Number = 100`. The value of `level0.bar1` is `Custom = myvalue` because we defined `_context` in a parent node. See the section on [extending the render context](tags?id=extending-the-render-context) to add your own variables.
+The value of `foo1` is the string `Number = 100`. The value of `level0.bar1` is `Custom
+= myvalue` because we defined `_context` in a parent node. See the section on [extending
+the render context](tags?id=extending-the-render-context) to add your own variables.
 
 We also provide `!j2_secret` to be used when dealing with sensitive data
 

docs/custom-tags.md

@@ -4,39 +4,46 @@ We use [multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) for
 
 ```py
 import os
-from gamma.dispatch import dispatch                             # (1)
-from gamma.config import ScalarNode, Tag, render_node           # (2) (3)
+from gamma.config import dispatch, ScalarNode, Tag               # (1) (2)
 
-MyEnvTag = Tag["!myenv"]                                        # (4)
+MyEnvTag = Tag["!myenv"]                                         # (3)
 
 @dispatch
-def render_node(node: ScalarNode, tag: MyEnvTag, **ctx):        # (5)
+def render_node(node: ScalarNode, tag: MyEnvTag, **ctx):         # (4)
     """Simpler clone of !env"""
-    return os.getenv(node.value)                                # (6)
+    return os.getenv(node.value)                                 # (5)
 ```
 
 In more details:
 
-1. We import `gamma.dispatch.dispatch`, that register our function as a dispatchable function.
+1. We import `gamma.config.dispatch`, that register our function as a dispatchable
+   function.
 
-2. We import `gamma.config.render_node`, to indicate we want to add another function to the dispatch table. A **very common mistake** is to forget to import the function, thus creating a _new_ dispatchable function instead of extending the existing one.
+2. We import `ScalaNode` and `Tag` types that we'll use to _specialize_ the
+   `render_node` function.
 
-3. We import `ScalaNode` and `Tag` types that we'll use to _specialize_ the `render_node` function.
+3. Because we want to dispatch on a **specific** tag, we create a specialized `MyEnvTag`
+   type from the general `Tag` type using a **parameter**. We call these _parametric_ or
+   _value_ types.
 
-4. Because we want to dispatch on a **specific** tag, we create a specialized `MyEnvTag` type from the general `Tag` type using a **parameter**. We call these _parametric_ or _value_ types.
+4. We annotate our function with the specific types. The name (eg. `render_node`) and
+   order of the arguments matter! The dispatch mechanics ignore keyword-only or untyped
+   arguments.
 
-5. We annotate our function with the specific types. The order of the arguments matter! The dispatch mechanics ignore keyword-only or untyped arguments.
-
-6. Whatever we return will be the value returned when accessing the configuration.
+5. Whatever we return will be the value returned when accessing the configuration.
 
 And that's it! You just need to ensure the code is loaded before rendering
 the config value.
 
 ### Render node arguments
 
-The `node` argument comes directly from [ruaml.yaml](https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/nodes.py#l12) package. For `ScalarNode`s usually you're interested in the `node.value`, that provides the exact string in the YAML file.
+The `node` argument comes directly from
+[ruaml.yaml](https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/nodes.py#l12)
+package. For `ScalarNode`s usually you're interested in the `node.value`, that provides
+the exact string in the YAML file.
 
-If you want to use the standard YAML inference from `ruamel.yaml` package, you can do as follows:
+If you want to use the standard YAML inference from `ruamel.yaml` package, you can do as
+follows:
 
 ```py
 # ... other imports as above ...
@@ -51,7 +58,10 @@ def render_node(node: ScalarNode, tag: MyEnvTag, **ctx):
                            # float, bool, timestamp, null)
 ```
 
-For `MappingNode` or `SequenceNode`, the `node.value` object is more complex. To avoid having to write your own recursive parsing logic, you can use the `to_dict` dump function to get a rendered object, including child nodes. (yes, it works with `Sequence`s as well)
+For `MappingNode` or `SequenceNode`, the `node.value` object is more complex. To avoid
+having to write your own recursive parsing logic, you can use the `to_dict` dump
+function to get a rendered object, including child nodes. (it works with
+`Sequence`s as well)
 
 ```py
 # ... other imports as above ...
@@ -75,9 +85,14 @@ The `ctx` kwargs dict allows you to access contextual information when rendering
 
 ### URI-style tags
 
-The default tag dispatch mechanism is to dispatch on the resolved tag value using a parameterized `Tag` subtype. For instance, `foo: !mytag 1` will only dipatch on `(node: ScalarNode, tag: Tag["!mytag"])`.
+The default tag dispatch mechanism is to dispatch on the resolved tag value using a
+parameterized `Tag` subtype. For instance, `foo: !mytag 1` will only dipatch on `(node:
+ScalarNode, tag: Tag["!mytag"])`.
 
-However, if the tag contains a `:` (colon), we're assuming the tag is an _URI-style tag_ of the format (`![scheme]:[path]`). Besides the default tag dispatch, URI-style tags will also dispatch on the _scheme_ part only, and pass a `path` keywork arg to the `render_node` function.
+However, if the tag contains a `:` (colon), we're assuming the tag is an _URI-style tag_
+of the format (`![scheme]:[path]`). Besides the default tag dispatch, URI-style tags
+will also dispatch on the _scheme_ part only, and pass a `path` keyword argument to the
+`render_node` function.
 
 An example:
 
@@ -104,7 +119,7 @@ catalog:
     inputs: s3://mybucket/myproject/inputs
 
   datasets:
-    customers: !j2 "{{ inputs }}/customers"   # will reference the context above
+    customers: !j2 "{{ inputs }}/customers"   # will reference the _context map above
 ```
 
 If you need to extend the render context, please refer to docstrings in the source file

gamma/config/__init__.py

@@ -1,5 +1,10 @@
 # flake8: noqa
+# isort: skip_file
+
 from gamma.config.__version__ import __version__
+
+# register a scoped dispatcher
+from plum import Dispatcher
 from ruamel.yaml.nodes import MappingNode, Node, ScalarNode, SequenceNode
 
 from .builtin_tags import yaml
@@ -10,3 +15,5 @@
 from .render import render_node
 from .render_context import ContextVar, context_providers
 from .tags import Tag
+
+dispatch = Dispatcher()

gamma/config/builtin_tags.py

@@ -5,12 +5,13 @@
 import threading
 from typing import Any, Union
 
-from gamma.config.confignode import ConfigNode
-from gamma.config.dump_dict import to_dict
-from gamma.dispatch import dispatch
 from ruamel.yaml import YAML
 from ruamel.yaml.nodes import MappingNode, Node, ScalarNode, SequenceNode
 
+from gamma.config import dispatch
+from gamma.config.confignode import ConfigNode
+from gamma.config.dump_dict import to_dict
+
 from .findconfig import get_config_root
 from .render import render_node
 from .render_context import get_render_context
@@ -34,7 +35,7 @@
 
 
 @dispatch
-def render_node(node: Node, tag: EnvTag, **ctx) -> str:
+def render_node(node: Node, tag: EnvTag, **ctx):
     """[!env] Maps the value to an environment variable of the same name.
 
     You can provide a default using the ``|`` (pipe) character after the variable
@@ -57,7 +58,7 @@ def render_node(node: Node, tag: EnvTag, **ctx) -> str:
 
 # process: !env_secret
 @dispatch
-def render_node(node: Node, tag: EnvSecretTag, *, dump=False, **ctx) -> str:
+def render_node(node: Node, tag: EnvSecretTag, *, dump=False, **ctx):
     """[!env_secret] Similar to !env, but never returns the value when dumping."""
     if dump:
         return node
@@ -66,7 +67,7 @@ def render_node(node: Node, tag: EnvSecretTag, *, dump=False, **ctx) -> str:
 
 # process: !expr
 @dispatch
-def render_node(node: Node, tag: ExprTag, **ctx) -> Any:
+def render_node(node: Node, tag: ExprTag, **ctx):
     """[!expr] Uses ``eval()`` to render arbitrary Python expressions.
 
     By default, we add the root configuration as `c` variable.
@@ -81,7 +82,7 @@ def render_node(node: Node, tag: ExprTag, **ctx) -> Any:
 
 
 @dispatch
-def render_node(node: Node, tag: J2Tag, *, config=None, key=None, **ctx) -> Any:
+def render_node(node: Node, tag: J2Tag, *, config=None, key=None, **ctx):
     """[!j2] Treats the value a Jinj2 Template
 
     See ``gamma.config.render_context.context_providers`` documentation to add your
@@ -135,7 +136,7 @@ def render_node(node: Node, tag: J2Tag, *, config=None, key=None, **ctx) -> Any:
 
 # process: !j2_secret
 @dispatch
-def render_node(node: Node, tag: J2SecretTag, *, dump=False, **ctx) -> Any:
+def render_node(node: Node, tag: J2SecretTag, *, dump=False, **ctx):
     """[!j2_secret] Similar to !j2, but never returns the value when dumping."""
 
     if dump:
@@ -144,7 +145,7 @@ def render_node(node: Node, tag: J2SecretTag, *, dump=False, **ctx) -> Any:
 
 
 @dispatch
-def render_node(node: Node, tag: RefTag, *, config=None, recursive=False, **ctx) -> Any:
+def render_node(node: Node, tag: RefTag, *, config=None, recursive=False, **ctx):
     """[!ref] References other entries in the config object.
 
     Navigate the object using the dot notation. Complex named keys can be accessed
@@ -176,7 +177,7 @@ def render_node(node: Node, tag: RefTag, *, config=None, recursive=False, **ctx)
 
 # process: !py
 @dispatch
-def render_node(node: ScalarNode, tag: PyTag, *, path=None, **ctx) -> Any:
+def render_node(node: ScalarNode, tag: PyTag, *, path=None, **ctx):
     """[!py] Pass the node value to a Python callable.
 
     This tag should be used as a URI-style tag on the form `!py:<module>:<callable>`

gamma/config/confignode.py

@@ -4,10 +4,11 @@
 from pathlib import Path
 from typing import Any, Dict, Iterable, Optional
 
-from gamma.config.load import load_node
-from gamma.dispatch import dispatch
 from ruamel.yaml.nodes import MappingNode, Node, SequenceNode
 
+from gamma.config import dispatch
+from gamma.config.load import load_node
+
 from . import tags
 from .merge import merge_nodes
 from .rawnodes import get_entry, get_id, get_keys, get_values
@@ -339,7 +340,7 @@ def create_last_entry_key(cfg: RootConfig) -> str:
 
 @dispatch
 @contextmanager
-def config_context(cfg: RootConfig, partial) -> None:
+def config_context(cfg: RootConfig, partial) -> Any:
     entry_key = create_last_entry_key(cfg)
     push_entry(cfg, entry_key, partial, _allow_unsafe=True)
     try:
@@ -350,7 +351,7 @@ def config_context(cfg: RootConfig, partial) -> None:
 
 @dispatch
 @contextmanager
-def config_context(partial) -> None:
+def config_context(partial) -> Any:
     from gamma.config.globalconfig import get_config
 
     cfg = get_config()

gamma/config/dump_dict.py

@@ -1,7 +1,8 @@
-from gamma.config.confignode import ConfigNode
-from gamma.dispatch import dispatch
 from ruamel.yaml.nodes import MappingNode, SequenceNode
 
+from gamma.config import dispatch
+from gamma.config.confignode import ConfigNode
+
 from .render import render_node
 from .tags import Map, Seq
 

gamma/config/dump_yaml.py

@@ -1,10 +1,11 @@
 import copy
 from io import StringIO
 
-from gamma.dispatch import dispatch
 from ruamel.yaml import YAML
 from ruamel.yaml.nodes import MappingNode, ScalarNode, SequenceNode
 
+from gamma.config import dispatch
+
 from .confignode import ConfigNode, RootConfig
 from .merge import merge_nodes
 from .rawnodes import as_node

gamma/config/findconfig.py

@@ -5,7 +5,9 @@
 from pathlib import Path
 from typing import Any, List, Optional, Tuple
 
-from gamma.dispatch import Val, dispatch
+from plum import Val
+
+from gamma.config import dispatch
 
 from .confignode import ConfigNode
 from .load import load_node

gamma/config/load.py

@@ -3,9 +3,11 @@
 from pathlib import Path
 from typing import Dict
 
-from gamma.dispatch import dispatch, parametric
+from plum import parametric
 from ruamel.yaml.nodes import Node
 
+from gamma.config import dispatch
+
 
 @parametric
 class ContentType:

gamma/config/merge.py

@@ -4,9 +4,10 @@
 from functools import reduce
 from typing import List, Tuple, Union
 
-from gamma.dispatch import dispatch
 from ruamel.yaml.nodes import MappingNode, Node, SequenceNode
 
+from gamma.config import dispatch
+
 from .rawnodes import get_item, get_keys, get_values, is_in, union_nodes
 
 hints_pattern = re.compile("^.*@hint: *?([A-Za-z0-9_]+) ?.*$", re.MULTILINE)

gamma/config/rawnodes.py

@@ -1,9 +1,10 @@
 """Module implementing convenience methods for dealing with `ruamel.yaml` `Node`s"""
 from typing import Any, Hashable, Iterable, Optional, Tuple
 
-from gamma.dispatch import dispatch
 from ruamel.yaml.nodes import MappingNode, Node, ScalarNode, SequenceNode
 
+from gamma.config import dispatch
+
 from . import tags
 
 Entry = Tuple[Node, Optional[Node]]
@@ -17,7 +18,7 @@ def get_keys(node: MappingNode) -> Iterable[Node]:
 
 
 @dispatch
-def get_item(node: MappingNode, key, *, default=...) -> Node:
+def get_item(node: MappingNode, key, *, default=...) -> Optional[Node]:
     """Get a single child node item from `map` node"""
     for item_key, item_value in node.value:
         if is_equal(key, item_key):
@@ -74,11 +75,16 @@ def is_equal(a, b) -> bool:
 @dispatch
 def is_equal(a: SequenceNode, b: SequenceNode) -> bool:
     """Check if `a` is equal to `b`"""
-    a_values = get_values(a)
-    b_values = get_values(b)
-    for value in a_values:
-        if value not in b_values:
+    a_values = list(get_values(a))
+    b_values = list(get_values(b))
+
+    if len(a_values) != len(b_values):
+        return False
+
+    for i in range(len(a_values)):
+        if not is_equal(a_values[i], b_values[i]):
             return False
+
     return True
 
 

gamma/config/render.py

@@ -3,10 +3,11 @@
 import logging
 from typing import Any, Optional
 
-from gamma.config.confignode import ConfigNode, RootConfig  # noqa
-from gamma.dispatch import dispatch
 from ruamel.yaml.nodes import MappingNode, Node, ScalarNode, SequenceNode
 
+from gamma.config import dispatch
+from gamma.config.confignode import ConfigNode, RootConfig  # noqa
+
 from . import tags
 from .rawnodes import get_entries, get_values