Optimize Column Lineage Query Performance

[vinhnemo]

Problem

The current implementation of the dataset fields view query in ColumnLineageDao.java does not include a filter to narrow down the dataset fields to only those linked with the version UUIDs identified in selected_column_lineage. This results in processing a larger dataset than necessary.
The lack of this filter can cause the query execution time to increase, especially when dealing with large datasets.

Closes: #2802

Solution

I propose adding a filter condition to the CTE dataset_fields_view in ColumnLineageDao.java:
From:

        dataset_fields_view AS (
          SELECT d.namespace_name as namespace_name, d.name as dataset_name, df.name as field_name, df.type, df.uuid
          FROM dataset_fields df
          INNER JOIN datasets_view d ON d.uuid = df.dataset_uuid
        )

To

dataset_fields_view AS (
  SELECT 
    d.namespace_name as namespace_name, 
    d.name as dataset_name, 
    df.name as field_name, 
    df.type, 
    df.uuid 
  FROM 
    dataset_fields df 
    INNER JOIN (
      select 
        * 
      from 
        datasets_view 
      where 
        current_version_uuid IN (
          SELECT 
            DISTINCT output_dataset_version_uuid 
          FROM 
            selected_column_lineage 
          UNION 
          SELECT 
            DISTINCT input_dataset_version_uuid 
          FROM 
            selected_column_lineage
        )
    ) d ON d.uuid = df.dataset_uuid
)

This filter will ensure that only relevant dataset fields are processed, improving the overall efficiency of the query.

One-line summary: adding a filter condition to the CTE dataset_fields_view in ColumnLineageDao.java:

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[boring-cyborg]

Thanks for opening your first pull request in the Marquez project! Please check out our contributing guidelines (https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md).

[netlify]

✅ Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
🔨 Latest commit	a10360e
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/665ded01517e610008b0be57

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.55%. Comparing base (e54ffca) to head (e14f060).

❗ Current head e14f060 differs from pull request most recent head a10360e

Please upload reports for the commit a10360e to get more accurate results.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2821      +/-   ##
============================================
- Coverage     84.56%   84.55%   -0.01%     
+ Complexity     1441     1440       -1     
============================================
  Files           251      251              
  Lines          6504     6501       -3     
  Branches        303      302       -1     
============================================
- Hits           5500     5497       -3     
  Misses          851      851              
  Partials        153      153              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[boring-cyborg]

Great job! Congrats on your first merged pull request in the Marquez project!