Access child component from root pipeline

[trducng]

Thinking points:

• Should this mechanism internal of the Composable implementation?
• Should this mechanism external of the Composable implementation?
• Nevertheless, it seems composition rather than inheritance.
• This ability will open up functionalities like:
  • Disable certain middleware in certain levels
  • Change parameters of some specific components
  • Serve as a kind of a dashboard to control the behavior of a pipeline.
• Should this being constructed and passed around by the context?
  • Use contextvars?

Initialization

Basic initialization:

# way 1
params = {
  "args1": 1,
  "args2": 2,
  "node1": {
      "__name__": "x",
      "arg2": "some-arg2"
  },
}
pipeline = Pipeline(**params)

# which is similar to way 2
pipeline = Pipeline(args1=1, args2=2, node1={"__name__": "x", "arg2": "some-arg2"})

Pros:

• Toward serializable params. Easier to perform plug-n-play.

Cons:

• Too much nesting, hard to read and examine. The params might benefit from being flattened out (e.g. "node1.name", "node1.arg2"...), but this approach only works if we pass params as a dictionary. When we pass the params as __init__'s kwargs, we have to use the dictionary notation.

For complicated and nested pipeline, people don't usually initialize with hard-coded values in __init__ (way 1), since it is by nature will be very long, hard to observe, and does not allow plug-n-play. Instead, they would likely want something like way 2, because they can swap out values easily (changing params or swapping nodes is just a matter of manipulating a dictionary from some UI).

Update params or nodes dynamically

Update params and nodes of an existing pipeline.

pipeline = Pipeline(arg1="arg1", arg2="arg2")
pipeline.node1.node2.paramxyz = "1000"    # reload the module `node2` with new value for `paramxyz`
pipeline.node1 = cls                      # reload the module `node1` with new value for  class