How to dynamically register a pipeline

[niartnelis]

Description

How to complete the dynamic registration of pipelines without registering pipelines through the register_pipelines method of pipeline_registry.py

Context

I hope to be able to dynamically add nodes or pipelines with the help of API, instead of building fixed pipelines through code. At present, it is possible to dynamically add nodes and pipelines, and dynamically execute the above pipelines, but I cannot dynamically register the pipelines, which makes it impossible to execute through the command line. Or visualize the process

Possible Implementation

I hope that the directory structure is not too fixed. Can I remove the pipeline directory and the pipeline.py and node.py below? In fact, I don’t quite understand the role of this layer of fixed directory structure. Structural organization impact, e.g. on deployment, or operational impact

Possible Alternatives

(Optional) Describe any alternative solutions or features you've considered.

[deepyaman]

I haven't heard of this use case previously, but I think I can point you in the right direction:

• The registry references the pipelines variable in kedro.framework.project; see

  kedro/kedro/framework/project/__init__.py

  Line 235 in 86ada23

  pipelines = _ProjectPipelines()

• This variable call the register_pipelines method, as you stated; however, nothing technically says that function can't access some other data (for reference, here's the code where register_pipelines is picked up:

  kedro/kedro/framework/project/__init__.py

  Lines 170 to 173 in 86ada23

  	register_pipelines = self._get_pipelines_registry_callable(
  	self._pipelines_module
  	)
  	project_pipelines = register_pipelines()

  )
• Finally, you would have to call pipelines.configure() each time to pick up the latest state; see

  kedro/kedro/framework/project/__init__.py

  Lines 170 to 173 in 86ada23

  	register_pipelines = self._get_pipelines_registry_callable(
  	self._pipelines_module
  	)
  	project_pipelines = register_pipelines()

I hope that the directory structure is not too fixed. Can I remove the pipeline directory and the pipeline.py and node.py below? In fact, I don’t quite understand the role of this layer of fixed directory structure. Structural organization impact, e.g. on deployment, or operational impact

In order to work with some built-in functionality, a certain structure is expected, but you can technically change it (e.g. compare the "standard" structure in spaceflights to the "simplified" one in the pandas-iris starter).

Sorry this isn't a complete answer; please feel free to follow up if it's not helpful in terms of next steps to investigate. Perhaps sharing a bit more context on what the dynamic pipeline definition and registration workflow would look like in practice would also help.

[niartnelis]

I have another question to ask you, in the newly created project "kedro new --starter=pandas-iris", in the src/{projectname}/main.py file, how to pass in a custom DataCatalog without configuring The file catalog.yml is defined, which is also the step for me to realize the dynamic construction of the pipeline

When I pass in params, the internal feed_dict does add the param prefix and cannot become a valid variable

[deepyaman]

I'm not sure I fully follow your question. If you want to define a catalog (and pipelines) dynamically, maybe you shouldn't use the CLI that expects a more standard Kedro project structure.

Instead, you can construct a runner object and pass your pipeline and catalog objects to it, as in the examples in https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/run_a_pipeline.html#run-pipelines-with-io (if you expand the hidden block).

[niartnelis]

Thank you very much for your reply. It is indeed possible to do as you said. I will ask you if you encounter any problems in the future.

[deepyaman]

Great! I'm going to close this, then; please feel free to open a new issue (and reference this one, if necessary), should you run into related issues down the road!