STALE: conditional substitutions and formulas?

[o-smirnov]

@SpheMakh FYI. While implementing my DDF mosaicing recipe, I've come across a use case that {}-substitutions don't quite serve. That is, "please set this step's parameter X to the value of previous Y if previous Y was set, or leave X unset if previous Y was unset".

I propose to implement it as another kind of substitution, the <<-substitution. Here's an example. This is a generic template for a DDF predict step where I want to reuse some settings from the previous cleaning step -- and leave them unset if they were unset in the previous step.

      predict-previous:
        info: "predicts sky model from imaging step directly preceding this one"
        _use: lib.steps.ddfacet.base
        params:
          Output-Mode: Predict 
          DDESolutions-DDSols: << previous.DDESolutions-DDSols?
          Predict-InitDicoModel: << previous.Predict-InitDicoModel?
          Facets-CatNodes: << previous.Facets-CatNodes?
          Predict-ColName: MODEL_DATA
          Deconv-Mode: << previous.Deconv-Mode

Here's the proposed logic:

• a string value of the form << a.b.c[?] is a <<-substitution. (Escaping: if you want a literal "<<" at the start of your string, use "<<<").
  • space between << and a is optional
• a.b.c is looked up in the substitution namespace, and inserted for the parameter's value. The same as '{a.b.c}', so far.
• If the lookup of a.b.c fails:
  • if there's no traling ?, return a substitution error (as '{a.b.c}' currently does)
  • if there is a trailing ?, leave the parameter unset <-- this is the crucial feature exploited in the example above!

So in the simplest case, without the trailing ?, <<a.b.c works exactly the same as '{a.b.c}'. (Footnote: there is still an advantage to using << in this simple case, since it actually specifies a semantic connection -- almost like a pipe -- between one step and another step. Future stimelas may take advantage of this information. By contrast, an '{a.b.c}' substitution is just a "soft" string substitution that "just happens" to evaluate to something, there are no semantic connections that stimela knows about there.)

I also propose some extended syntax for <<-substitutions, making it into a true conditional:

params:
  foo: << a.b.c[?] [?IF_TRUE] [:IF_FALSE] [!IF_UNSET]

In this case, a.b.c is evaluated. If it evaluates to true (in the Pythonic sense), then IF_TRUE is substituted in. If it evaluates to false, IF_FALSE is substituted in. If it is unset, IF_UNSET is substituted in.

Note that any of IF_TRUE, IF_FALSE, IF_UNSET may be omitted. Also note that they may themselves be <<- or {}-substitutions.

This makes it possible to specify things like "if parameter X is set, then use the value of parameter Y, else use parameter Z", which I think will simplify the life of generic recipes (thinking 1GC here @landmanbester).

[o-smirnov]

Anyway I implemented this on the rich branch of scabha2. Pull to test. Working example here: https://github.com/o-smirnov/jovial/blob/master/ddf-mosaic-lib.yaml#L49

(Top-level recipe is https://github.com/o-smirnov/jovial/blob/master/ddf-mosaic.yaml)

[o-smirnov]

Actually, having slept on it, I'm leaning against the funky syntax, and more towards spreadsheet-style (don't want to say Excel 😨) formulas for this. So instead of the above, have:

  x: =previous.y                  # set to 'y 'of previous step, error if that is unset
  x: =IFSET(previous.y)        # set to 'y 'of previous step, leave unset if that is unset
  x: =IFSET(previous.y, A, B)     # if previous.y is set, use A, if unset, use B
  x: =IFSET(previous.y,, B)      # if previous.y is set, use previous.y, if unset, use B
  x: =IF(previous.y, A, B)      # if previous.y is true, use A, else use B. Throw error if y is not set.
  x: =IF(previous.y, A, B, C)   # if previous.y is true, use A, else use B, or use C if y is not set.

I think it's a little more human-readable, and extensible to more formulas in the future (since we were already talking about arithmetic...) In fact, my doubts in #21 can have the same kind of solution:

   output_files: =GLOB(*.fits)

Also, the suggestion in #25 can be addressed elegantly. Rather than having a separate skip_if_exists attribute on a step, keep using the existing skip attribute, just make it subject to formula expansion too:

  skip: =EXISTS(current.output_files)

...which actually gets 80% of the way towards conditionals-in-recipes, something I've thought about but couldn't figure out how to do neatly.

[o-smirnov]

OK all done in #27.

[o-smirnov]

Since this is all mainstream now, I'm marking this as stale.