-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Internal DB enrichment performance improvements #11981
Conversation
…ere is a problem with cyclic enrichment due to the outputProcessing, need to decide how to handle formulas on enrichment.
Codecov Report
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. @@ Coverage Diff @@
## master #11981 +/- ##
==========================================
+ Coverage 74.38% 74.62% +0.23%
==========================================
Files 326 326
Lines 13941 13925 -16
Branches 2937 2923 -14
==========================================
+ Hits 10370 10391 +21
+ Misses 3330 3299 -31
+ Partials 241 235 -6
... and 2 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
…ched and then we can ignore everything else, makes a big difference for enriching users (with a lot of relationships).
…al-db-enrich-perf
…al-db-enrich-perf
…ibase into fix/internal-db-enrich-perf
44c6401
to
5e00a0d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Nice work
…al-db-enrich-perf
Description
This PR really provides one major change to the enrich endpoint and then makes some minor adjustments to the link system to try and improve efficiency of the calls.
In general, this is pretty much as optimised as it can be - this does not make any headway to improving the self call, but there is very little we can do there as we don't know the scope of what is needed from the user structure - if it is a heavily modified user table (with a lot of relationships) we cannot improve the speed of this.
The biggest change here is that when the data provider requests a to enrich a row, it doesn't need all of the columns that come back, it generally only looks at a single field. In fact within the API functions of frontend-core it strips out everything else it gets back, so this is wasted if it is retrieved. I've added a
field
query string parameter to the enrichGET
call which allows restricting the call down to exactly what the data provider needs, significantly improving the performance of this call.Below is a before and after on a local network, note the difference in size as well, this makes a big difference in our cloud platform where the size creates extra latency ontop of the actual work done in the request.
Screenshots
Before (enrichment call)
After (enrichment call)