Column-level lineage in Marquez UI

[vivekramaswamy1]

This is more of a question/feature request than an in issue.

Does the Marquez UI show column-lineage tracking/information. For example, in the UI, we show dataset A -> transformation -> dataset B, and we are tracking a certain column 'col1' from dataset A to 'col_new' in dataset B, is this currently displayable other than in the json dump /spec? If no, can we do it? I feel a lot of the work has already been done in terms of tracking the column lineage, showing each column as node and edges between the different nodes will be extremely useful

[boring-cyborg]

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

[pawel-big-lebowski]

Hi @vivekramaswamy1,

As a backend HTTP API, there exists an endpoint that returns column-lineage graph, but currently Marquez UI is not able to draw column-lineage graph the same shiny way it draws datasets & jobs lineage graph.

However, it is capable of presenting column-lineage of a dataset as a JSON, which has been done here -> #2293 (you can see the screenshots)

[wslulciuc]

@vivekramaswamy1, with PR #2293 now merged, Marquez 0.30.0 will have support for displaying column-level lineage in the UI. There more that we can do here in terms of visual rendering, but it's a great start!