Add proposal for ol facet tables

[wslulciuc]

This PR adds the proposal: Optimize query performance for OpenLineage facets

[codecov]

Codecov Report

Merging #2076 (28a07e9) into main (1d28adf) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #2076   +/-   ##
=========================================
  Coverage     76.72%   76.72%           
  Complexity     1177     1177           
=========================================
  Files           222      222           
  Lines          5354     5354           
  Branches        429      429           
=========================================
  Hits           4108     4108           
  Misses          768      768           
  Partials        478      478           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[collado-mike]

Nice. Can you add specifics on how conflicts will be dealt with? Are these facet tables append-only? E.g., if two facets with the same name (I assume the name column in the tables refers to the facet name) but different contents are received for the same dataset version (e.g., two different GreatExpectations suites on the same data), will both records be added to the tables? If not, which one wins? First received? Last received? If both records are inserted, will they be merged at query time? Or just appended one after the other? Are there indexes on these tables? In particular, the runs table can get very, very large. The same will be true of the runs_facets table eventually.

[wslulciuc]

@collado-mike: All great questions that I'll elaborate on in the proposal!

[pawel-big-lebowski]

From what I understand, we load the events to memory in Marquez and do facet filtering there.
If so, the solution should be to offload filtering to database. The short term solution could be filtering json content in postgres as described here (please note indexing json content is also possible). In this particular example we could select output dataset facets from event json within postgresql, instead of selecting whole events.

But lineage_events will grow over time and querying for datasets' facets will slow down. Normalizing json facets, as described within this proposal, is a good way to go.

The only problem with separate tables is a backfill procedure which is a heavy operation. I am not sure whether lazy migration would work. The existence of some facets in new tables does not mean we collected all the existing facets, including those from lineage_events table.

[wslulciuc]

Very reasonable point, @pawel-big-lebowski. But, to get ahead of the issue, like you stated, we'll want to normalize facets.

[julienledem]

That looks great, I left some comments

[pawel-big-lebowski]

I think it's a great proposal which can have huge impact on Marquez performance 🚀
I've put some comments which are worth considering (if not already considered).

[mobuchowski]

In some cases it might make sense to accumulate content inside the facets themselves.

For example, streaming Flink job might report how many records were processed per checkpoint - sometimes sending multiple results per OL event. The result facet should return list of those reports.

I don't think we need to address this within this proposal though.

[wslulciuc]

I agree, certain facets can be handled differently (i.e. accumulated) based on some context. I'll make a note on the proposals scope / limitations.

[mobuchowski]

@wslulciuc let's also create follow-up issue, will be good to go then on this problem.

[pawel-big-lebowski]

Please also refer to this:
https://github.com/MarquezProject/marquez/pull/2152/files?short_path=17843f7#diff-17843f7d4567ca029ee0d63f56e6b75a000b384768d9e0352badffad66eeea3c

to see migration from a user's perspective.