OpenLineage integration

[yonil7]

now that OpenLineage is formally released and is part of LF AI & Data it would be nice to see some kind of integration between it and dvc

[efiop]

Hi @yonil7

Could you please elaborate? I'm not really familiar with OpenLineage. How do you see the integration, btw?

[yonil7]

OpenLineage is a new open standard for lineage metadata collection. (inspired by opentelemetry - standard for telemetry data (metrics, logs, and traces) - https://openlineage.io/blog/openlineage-takes-inspiration-from-opentelemetry/)

OpenLineage spec is a definition of a single json object - RunEvent - this event should be generated by a pipelines/workflows frameworks during a pipeline/workflow run to report the run state: START / COMPLETE / ABORT / FAIL / OTHER and additional data about the run: input datasets identification, output datasets identification, job version that runs, ...
here is the full definition of this json schema: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json

On top of this json schema definition, OpenLineage defines HTTP API and provide a python/java client library implementation for using this (very simple) API

On the other hand, OpenLineage provides integrations with pipelines/workflows frameworks (Airflow, Spark, dbt and other frameworks are on the roadmap).
These integrations, allows one to use these framework normally but when the framework will run a pipeline/workflow they will automatically send RunEvent to report on their run progress and other run data (input dataset, output dataset, ...) to any system that implements the OpenLineage HTTP API

There are currently 2 systems that implements the OpenLineage HTTP API that I know of:

• marquez
• egeria

Some ways dvc can be involved in this initiative:

1. Participate in the standard spec definition
2. Add ability for dvc repro or dvc exp run to report their run state during the run to any system that implements the OpenLineage HTTP API (e.g. marquez) (like the integrations provided here: https://openlineage.io/integration)
3. Add ability to export a pipeline run data (after it already done) in this standard json schema (then the user can import this data into other systems for run analysis)
4. Other integrations with DVCLive / CML / Studio and this standard