clconf

clconf provides a utility for merging multiple config files and extracting values using a path string. clconf is both a library, and a command line application.

For details, see clconf --help.

Background

clconf was primarily designed for use in containers where you inject secrets as files/environment variables, but need to convert them to application specific configuration files.

The 12 factor app states:

The twelve-factor app stores config in environment variables

But many existing applications/frameworks expect configuration files. This application helps serve to bridge this gap. clconf can be used by itself, or combined with other tools like confd (note, this is my fork of confd as they chose a different direction).

Configuration

Using clconf requires one or more yaml files (or strings) to merge together. They are specified using either using environment variables or command line options, as either files or base64 encoded strings. The order they are processed in is as follows:

1. --yaml: One or more files.
2. YAML_FILES environment variable: A comma separated list of files.
3. --yaml-base64: One or more base64 encoded strings containing yaml.
4. YAML_VARS environment variable: A comma separated list of environment variable names, each a base64 encoded string containing yaml.
5. --stdin/--pipe: One or more --- separated yaml files read from stdin.
6. --var: One or more path overrides of the form /foo="bar". Key is a path, an value is json/yaml encoded.
7. --patch: One or more rfc 6902 json/yaml patch files to apply to the result of merging all the config sources.
8. --patch-string: One or more rfc 6902 json/yaml patches to apply to the result of merging all the config sources.

All of these categories of input will be appended to each other and the last defined value of any key will take precedence. For example:

YAML_FILES="c.yml,d.yml"
YAML_VARS="G_YML_B64,H_YML_B64"
E_YML_B64"$(echo -e "c:\n  foo: bar" | base64 -w 0)
F_YML_B64"$(echo -e "d:\n  foo: bar" | base64 -w 0)

G_YML_B64="$(echo -e "g:\n  foo: bar" | base64 -w 0)
H_YML_B64="$(echo -e "h:\n  foo: bar" | base64 -w 0)

clconf \
  --yaml a.yml \
  --yaml b.yml \
  --yaml-base64 "$E_YML_B64" \
  --yaml-base64 "$F_YML_B64" \
  --var '/foo="bar"' \
  --patch patch.json \
  --patch-string '[{"op": "replace", "path": "/foo", "value": "baz"}]' \
  <<<"---\nfoo: baz"

Would be processed in the following order:

1. a.yml
2. b.yml
3. c.yml
4. d.yml
5. E_YML_B64
6. F_YML_B64
7. G_YML_B64
8. H_YML_B64
9. stdin
10. /foo="bar"
11. patch.json
12. [{"op": "replace", "path": "/foo", "value": "baz"}]

Use Cases

Helper in Scripts

Getv as JSON

When using the --output json option, the value obtained at the indicated path will be serialized to json. For example, if you have foo.yml:

applications:
- a
- b
- c

You could use:

clconf --yaml foo.yml getv /applications --output json

To get:

["a","b","c"]

Safe iteration of YAML/JSON elements

The --output json-lines option will convert each top level element to a JSON lines object. This allows you to safely process the results one line at a time with a simple line oriented reader. For example:

readarray -t dbs < <(clconf --pipe jsonpath '$[*]' --output json-lines <<'EOF'
---
foodb:
  host: foo.example.com
  credentials:
    username: foouser
    password: foopass
bardb:
  host: bar.example.com
  credentials:
    username: baruser
    password: barpass
EOF
)
for db in "${dbs[@]}"; do
  host="$(clconf --pipe getv /host <<<"${db}")"
  user="$(clconf --pipe getv /credentials/username <<<"${db}")"
  pass="$(clconf --pipe getv /credentials/password <<<"${db}")"

  echo "connecting to ${host} with ${user}/${pass}"
done
# connecting to bar.example.com with baruser/barpass
# connecting to foo.example.com with foouser/foopass

If the top level element is a map, then each JSON lines object will contain two top level keys: key, value. For example:

clconf --var '/foo=["bar","baz"]' getv / --output json-lines # {"key":"foo","value":["bar","baz"]}

Convert Bash Array to JSON

arr=(a b)

# always use the `--` to ensure none of the arguments get consumed by clconf
clconf var -- /foo "${arr[@]}" # /foo=["a","b"]

# note that with --value-only the / is ignored and can be anything
clconf --value-only -- / "${arr[@]}" # ["a","b"]

# can force single values into array
clconf --force-array -- /root/arr "foo" # /root/arr=["foo"]

Convert JSON Array to Bash

Allows for iteration:

# clconf getv --as-bash array will print out '([0]="foo bar" [1]="hip hop")'
# which bash's declare -a can turn into an array.  using --var here for
# simplicity but any yaml/json source will do.
declare -a arr="$(clconf --var '/a=["foo bar", "hip hop"]' getv /a --output bash-array)"
for i in "${arr[@]}"; do
  printf '<<<%s>>>' "$i"
done
# <<<foo bar>>><<<hip hop>>>

Also allows for iteration over maps:

declare -a arr="$(clconf --var '/a={"foo": "bar","hip":"hop"}' getv /a --output bash-array)"
for i in "${arr[@]}"; do
  printf '<<<%s>>>' "$i"
done
# <<<{"key":"foo","value":"bar"}>>><<<{"key":"hip","value":"hop"}>>>

Get Value Using JSON Path

The jsonpath subcommand allows you to use jsonpath syntax to locate values. The values obtained have the same output formatting options as getv does.

clconf --pipe jsonpath "$..credentials" <<'EOF'
foodb:
  host: foo.example.com
  credentials:
    username: foouser
    password: foopass
EOF
# - password: foopass
#   username: foouser

Getv Templates

Templates allow you to apply your configuration to golang template plus some additional custom functions

Note that when used in conjunction with the --output go-template* options, getv templates see a one-level key-value map, not the map represented by the yaml. For example, this yaml (foo.yml):

applications:
- a
- b
- c
credentials:
  username: foo
  password: bar

Would be seen by inside the templates as:

/applications/0: a
/applications/1: b
/applications/2: c
/credentials/username: foo
/credentials/password: bar

A simple bash program to utilizes this might look something like:

(
  clconf --pipe \
    getv / \
    --output go-template-file \
    --template <(cat <<'EOF'
{{- range (getvs "/applications/*")}}
echo {{.}} --user {{getv "/credentials/username"}} --pass {{getv "/credentials/password"}}
{{- end}}
EOF
      ) \
    <<'EOF'
---
applications:
- command_a
- command_b
- command_c
credentials:
  username: foo
  password: bar
EOF
) | bash
# command_a --user foo --pass bar
# command_b --user foo --pass bar
# command_c --user foo --pass bar

Kubernetes/OpenShift

This is my primary use case. It is a natural extension of the built-in ConfigMap and Secret objects. With clconf you can provide non-sensitive environment configuration in a ConfigMap:

db:
  url: jdbc.mysql:localhost:3306/mydb

and sensitive configuration in a Secret:

db:
  username: mydbuser
  password: youllneverguess

Then with:

clconf \
  --yaml /etc/myapp/config.yml \
  --yaml /etc/myapp/secrets.yml \
  getv \
  > /app/config/application.yml

You would have a file containing:

db:
  url: jdbc.mysql:localhost:3306/mydb
  username: mydbuser
  password: youllneverguess

Which can be written to an in-memory emptyDir:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: my-springboot-app
      volumeMounts:
        - name: app-config
          mountPath: /app/config
  volumes:
  - name: app-config
    emptyDir:
      medium: "Memory"

So that the sensitive information never touches disk and would not be exposed by a ps command.

Secret Management

clconf can encrypt and decrypt values as well, similar in nature to ansible-vault. This allows you to commit your secrets alongside the code that uses them. For example, you could create a new config file:

db:
  url: jdbc.mysql:localhost:3306/mydb

Then add your secrets:

clconf \
  --secret-keyring testdata/test.secring.gpg \
  --yaml C:/Temp/config.yml \
  csetv /db/username dbuser
clconf \
  --secret-keyring testdata/test.secring.gpg \
  --yaml C:/Temp/config.yml \
  csetv /db/password dbpass

Which would result in something safe to commit with your source code:

db:
  password: wcBMA5B5A4w5Zw+rAQgAJ9bR77oJi0P7X5qtnN+soUCszYTy6VGvNutHInE0QCugyXhVeovm+iPaFo/K5D8IO9QJnRL4D9PCiuqVslhsP54b7Qpep/1R/1HEbw9XNMv+uTh9CQDnT1FMer9i+samZ6poTT5uWMJtdTnwa187V5TUGKQdSwoz82CgQ8zQYq0aI15kZp4VziN9eQV1jrphG2+aJdtyIuIouafuEMSnrRz+bb8xAWu3I1INfEP0MuttTYdoY9W3xEU7L4IGvzhw8rnJPNhkK5LKTtvlOCDpKSs1ESReBHYSPNSAAlKBOTHwZ1MHKnypiWVzGACzq+Yh0K+UGtb8dGRiFhwMAn9jfdLgAeRkS/i2wGBjd3suaPzadgW84a0e4L3g3+FKo+Co4k3c3CHgB+OodVAQ2+LoReD54e6X4HbjH52aGIGkSKbg5eLA4qGv4Dnjwf422VOoqubgTOQV3gjv0NTKLF9IXaFPyhtj4joDyk/hwo8A
  url: jdbc.mysql:localhost:3306/mydb
  username: wcBMA5B5A4w5Zw+rAQgAAH1FM4x/FAjmspKbyHJvvaMwmFjGOMOKIle1oe0tpewzaUaEoYZ2trx8nerbWqtIxf4rnB9kNA2YyKs6CLka1q6jnN2U4KI3EjXQaaf6sL5qg/g3Hlak937Wf8+fK1tpghGuFJXTcRjqOgAyV8LfZtQ7MDfgoIy30bihjQz/0TzNi3IZlezqsgvLqoRsgP4b5S9liR/8EaQQ9BepaAgjl3c37QJf/qQK1mkPTOGzlTzZ7dcicpycxRwU8mMlYMq4qN0RR8ZMuiPshYJOdb3OVbNZq08MVzRbuMcPo+SbJsckD+V7EvOn3Km7jefblZsx2fzRPrAG23zZYkAPsUUuE9LgAeTO9rtOh0NQhkYL+9nJzCE+4dpv4K3gCOGkxOBR4o5q737gIuOVjW3r5vC/cuCA4ciT4JDjUV+uW8+IzSfgceKckR304HrjfbEkfn2gljvgAuSCU2yJMaO1aVjs225Rhw7q4pq3xL3hDV4A

These values can be decrypted using:

clconf \
  --secret-keyring testdata/test.secring.gpg \
  --yaml C:/Temp/config.yml \
  cgetv /db/username
clconf \
  --secret-keyring testdata/test.secring.gpg \
  --yaml C:/Temp/config.yml \
  cgetv /db/password

Or in conjunction with templates

clconf \
  --secret-keyring testdata/test.secring.gpg \
  --yaml C:/Temp/config.yml \
  getv / \
  --output go-template \
  --template '{{ cgetv "/db/username" }}:{{ cgetv "/db/password" }}'

Templating

clconf has a template operation that functions as a confd replacement but supports only yaml as a value store. It uses command line arguments in place of confd's toml files to determine where templates are found and output placed.

clconf supports additional functions above what confd provides.

All of the options for getv are available for specifying yaml sources, and the templates behave as outlined above. The template operation takes it a step further by templating many files in a single run. The template function's --help provides examples.

See the template function documentation for the available template functions.