OL facets - PR3 - migrate data to facet tables

[pawel-big-lebowski]

Signed-off-by: Pawel Leszczynski [email protected]

Problem

The last PR of Openlineage facets' PRs. It contains a migration that backfills newly created facets' tables (job_facets, dataset_facets and run_facets) with data contained within lineage_events tables. The migration has to be done manually for users with more than 100K lineage_events stored in Marquez.

Closes: #ISSUE-NUMBER

Solution

• Prepare migration script in Java that backfills facets' tables.
• Store in database migration_lock to be able to pause/continue migration and know if it has finished.
• At the end of successful migration, replace existing job_facets_vew, dataset_facets_view and run_facets_view to point at newly created tables. This will make application logic make use of new tables.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[codecov]

Codecov Report

Merging #2359 (98435c5) into ol-facets/PR2-read-data-from-views (45ecf15) will increase coverage by 0.26%.
The diff coverage is 87.01%.

@@                           Coverage Diff                            @@
##             ol-facets/PR2-read-data-from-views    #2359      +/-   ##
========================================================================
+ Coverage                                 76.85%   77.11%   +0.26%     
- Complexity                                 1208     1234      +26     
========================================================================
  Files                                       227      228       +1     
  Lines                                      5495     5572      +77     
  Branches                                    444      447       +3     
========================================================================
+ Hits                                       4223     4297      +74     
- Misses                                      774      775       +1     
- Partials                                    498      500       +2     

Impacted Files	Coverage Δ
api/src/main/java/marquez/MarquezApp.java	63.75% <0.00%> (-2.49%)	⬇️
...a/marquez/db/migrations/V57_1__BackfillFacets.java	90.54% <90.54%> (ø)
api/src/main/java/marquez/db/RunFacetsDao.java	80.95% <0.00%> (+4.76%)	⬆️
...main/java/marquez/service/models/LineageEvent.java	94.52% <0.00%> (+8.21%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[wslulciuc]

Minor comments, otherwise 🚀 💯