Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Transform] Transform optmize date histogram #54068

Merged

Conversation

hendrikmuhs
Copy link

@hendrikmuhs hendrikmuhs commented Mar 24, 2020

optimize transform for group_by on date_histogram by injecting an additional range query. This limits the number of search and index requests and avoids unnecessary updates. Only recent buckets get re-written.

fixes #54254

@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml/Transform)

Copy link
Member

@benwtrent benwtrent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optimization makes sense. If we are pivoting with a date_histogram, we shouldn't run through the entire index to update the destination.

The state machine changes were difficult to parse through. Will go through them again once this is no longer a draft.

@hendrikmuhs hendrikmuhs force-pushed the transform-optmize-date-histogram branch from 09085df to 9c2ed35 Compare March 24, 2020 21:26
@bpintea bpintea added v7.8.0 and removed v7.7.0 labels Mar 25, 2020
@hendrikmuhs hendrikmuhs force-pushed the transform-optmize-date-histogram branch from dbfd27c to 80d9413 Compare March 26, 2020 08:23
@hendrikmuhs hendrikmuhs marked this pull request as ready for review March 26, 2020 08:26
@hendrikmuhs
Copy link
Author

run elasticsearch-ci/2

Copy link
Member

@benwtrent benwtrent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another pair of 👀 would be good

Hendrik Muhs and others added 3 commits March 26, 2020 14:11
…transform/transforms/pivot/Pivot.java

Co-Authored-By: Benjamin Trent <[email protected]>
…transform/transforms/pivot/DateHistogramGroupSource.java

Co-Authored-By: Benjamin Trent <[email protected]>
@hendrikmuhs hendrikmuhs merged commit 69fe13c into elastic:master Mar 26, 2020
@hendrikmuhs hendrikmuhs deleted the transform-optmize-date-histogram branch March 26, 2020 20:38
hendrikmuhs pushed a commit that referenced this pull request Mar 26, 2020
optimize transform for group_by on date_histogram by injecting an additional range query. This limits the number of search and index requests and avoids unnecessary updates. Only recent buckets get re-written.

fixes #54254
hendrikmuhs pushed a commit that referenced this pull request Mar 26, 2020
optimize transform for group_by on date_histogram by injecting an additional range query. This limits the number of search and index requests and avoids unnecessary updates. Only recent buckets get re-written.

fixes #54254
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Transform] continuous transform date_histogram group_by performance
5 participants