Search service proposal

[pawel-big-lebowski]

Signed-off-by: Pawel Leszczynski [email protected]

Problem

Search service proposal

Closes #2180

Solution

Within PR.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

• You've signed-off your work
• [] Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[codecov]

Codecov Report

Merging #2203 (727e6a9) into main (7885c8c) will increase coverage by 0.01%.
The diff coverage is n/a.

❗ Current head 727e6a9 differs from pull request most recent head fc851d3. Consider uploading reports for the commit fc851d3 to get more accurate results

@@             Coverage Diff              @@
##               main    #2203      +/-   ##
============================================
+ Coverage     76.72%   76.73%   +0.01%     
- Complexity     1147     1148       +1     
============================================
  Files           219      219              
  Lines          5318     5318              
  Branches        423      423              
============================================
+ Hits           4080     4081       +1     
  Misses          763      763              
+ Partials        475      474       -1     

Impacted Files	Coverage Δ
...pi/src/main/java/marquez/db/mappers/RunMapper.java	90.56% <0.00%> (+1.88%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[wslulciuc]

What would be returned by querying postgres? Wouldn't the object be stored in ES and returned to the caller?

[pawel-big-lebowski]

right, it's not necessary. SearchResult can be evaluated from ElasticSearch backend.

[wslulciuc]

I think as a follow up, we can discuss how we might want to revisit the SearchResult model and was to extend it.

[mobuchowski]

I discussed this with Paweł offline, but I believe there are few things missing:

• we shouldn't index everything, but whitelisted facets
• we should know where we've found the keyword and somehow describe it in the search result

[pawel-big-lebowski]

I discussed this with Paweł offline, but I believe there are few things missing:

• we shouldn't index everything, but whitelisted facets

Blacklisting spark_plan and spark_unknown definitely makes sense. I would be in favor of blacklisting just to allow searching on user's facets.

• we should know where we've found the keyword and somehow describe it in the search result

As discussed, ElasticSeach highlight feature enriches document with <em> tag which is not super-clean and is tightly-coupled with ElasticSearch backend. I would prefer to start without it, we can add event preview on a list of search result to help user find out how the search lead to such results.

[wslulciuc]

A well thought-out and written proposal. Great job, @pawel-big-lebowski 💯 💯 🥇