Feature: Allow config params to be passable at the CLI and have a parameter_default

[johncmerfeld]

From this Monday item:

Earthmover doesn't support the following construction:

config:
  output_dir: ${OUTPUT_DIR}
  parameter_defaults:
    OUTPUT_DIR: './output/'

(result is the creation of a directory named ${OUTPUT_DIR})

This PR adds support for this. The use case is for things we want to allow advanced users to configure but usually has a sensible default (e.g. OUTPUT_DIR, STATE_DIR)

Behavior notes:

• Params passed at runtime still take precedence, and the default is only used if no param is passed. If the user provides an OUTPUT_DIR, the default is ignored.
• Only "typical" environment variables are allowed, i.e. ${VALUE_X}. No spaces, dashes, etc. Just write it the way we normally do and you're fine.

You can test this easily with earthmover -t, which has been updated to utilize this construction. You can also modify the earthmover.yaml of an assessment bundle and run with its sample data.

[tomreitz]

Could we use string templates here instead? something like

from string import template

for key, val in configs.items():
    if isinstance(val, str):
        template = Template(val)
        configs[key]= template.substitute(self.params)

This would eliminate the need for regex matching, and also support using an env var as part of the config value but not the whole value, like

config:
  output_dir: ${OUTPUT_BASE}transformed/
  parameter_defaults:
    OUTPUT_BASE: './output/'

[johncmerfeld]

WOW, dramatically simpler, this is awesome, thanks! Tested that it works like a charm

[tomreitz]

Looks great - thanks @johncmerfeld.