Is there a way to track data dependency in Spline for spark in down stream jobs in one place?

[leadawsprofessional]

Can we track the data dependency (Forward data source level lineage) ? Consider Data set A ---> Transformation --> Data Set B --> Transformation --> Data Set C .
There are almost 5 jobs that take the data from "Data Set C" and use them in some other Spark job. My question: Is there a way we can see the lineage of the Data Set C with these jobs in one place, not in separate job sets.

This question is similar to #1081.

Could anyone please let me know when this feature will be available as I can not see the "Forward data source level lineage for a given data source" in the current Spline UI.

[mariobarbosa10]

Hi bro, maybe you have any advance with this task? Can you help me, please?

[wajda]

Yes, absolutely you can. As I explained in the other topic this feature isn't on the UI yet, but the data currently captured is sufficient to build that kind of graph. We are very small team (and are hiring BTW) and currently lack of a UI dev, thus features are added rather slowly, sorry about that. I tried to explain in short how to modify the query to build the forward lineage in the #1081. We would gladly accept a PR on this feature. Otherwise I'll probably be only able to get to it somewhere in middle of June at the earliest.