Update lineage query to only look at jobs with inputs or outputs

[collado-mike]

Signed-off-by: Michael Collado [email protected]

Problem

In many environments a large number of jobs reporting events have no inputs or outputs - e.g., PythonOperators in an Airflow deployment. If a Marquez installation has a lot of these, the lineage query spends a lot of its time searching for overlaps with jobs that have no inputs or outputs. In one installation, we have > 200K jobs, but only ~7000 jobs that have any inputs or outputs at all.

Solution

This changes the lineage query to query the job_versions_io_mapping table and INNER join with the jobs_view so that only jobs that have inputs or outputs are present in the jobs_io CTE. The impact of this is that table becomes very small and the recursive join in the lineage CTE is very fast.

Probably notable that the missing inputs/outputs are largely due to insufficient coverage by the OpenLineage integrations - e.g., those PythonOperators are likely reading data from somewhere. This is, at best, a short term fix until OL coverage increases, at which point, the query will have to be revisited again.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[codecov]

Codecov Report

Merging #2068 (4af5216) into main (476e472) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #2068   +/-   ##
=========================================
  Coverage     78.79%   78.79%           
  Complexity     1011     1011           
=========================================
  Files           200      200           
  Lines          5574     5574           
  Branches        422      422           
=========================================
  Hits           4392     4392           
  Misses          730      730           
  Partials        452      452           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[wslulciuc]

@collado-mike, though you consider this a quick "hack", I do view it as reasonable optimization to improve the lineage query performance. Now, as you pointed out, it is a short-term optimization change. That is, coverage for OpenLineage will improve, which, in turn, will negatively impact the lineage query. But we have spoken briefly on ways we can continue to improve lineage query performance by introducing materialized views, caching, etc. Anyways, great work 💯 🥇