Diagram to pipeline structure

[nickofca-mck]

Description

An integration to Miro, Figma, adapted kedro viz, or one of the other diagramming tools where we can create a diagram (with designated elements) and the pipeline and nodes automatically populate themselves in the kedro pipeline structure.

Context

Would drastically reduce the overhead in setting up pipelines, nodes, and the connections between them within the code structure. Usually I need to create a diagram within Miro or Figma anyways to write notes and plan pipeline creation anyways.

Possible Implementation

Create a Miro and/or Figma template to construct a pipeline within their platform. Export the results and read in the connectivity/constituency of nodes and pipelines. Parse results into kedro pipeline and nodes created and outlined within the local directory.

Possible Alternatives

Implement something similar to above but in the kedro viz platform

[astrojuanlu]

Hi @nickofca-mck, thanks for opening this feature request. Do you know if Miro or Figma allow exporting their diagrams in a structured format?

[antonymilne]

I've wanted to do this for ages so am a huge fan of the idea 👍 IMO kedro-viz is ultimately the right place for this to live, but having something which converts from Miro to a kedro pipeline would be cool also. I wonder if these days you even need to export from Miro to a structured format? Maybe it's possible to just screenshot your Miro diagram and put it into some GenAI tool that outputs a rough attempt at the kedro code for the pipeline definition. The result probably won't be perfect but might not be far off.

[astrojuanlu]

Adding some context here, there have been a number of visual pipeline editors over the years. See for example

• Elyra https://elyra.readthedocs.io/en/latest/getting_started/overview.html#ai-pipelines-visual-editor
• Orchest (defunct) https://github.com/orchest/orchest/

Probably integration with Miro or Figma is hard if they don't provide a structured format that can be parsed (unless maybe we can rely on GenAI to look at the image and produce a text description of a graph?). But a lightweight, visual pipeline editor would be cool.

I know @NeroOkwa has mentioned informally that he'd like that functionality in Kedro Viz. In any case, this is not a priority at the moment. Leaving the issue open.

[astrojuanlu]

Turning this into a discussion.