-
Notifications
You must be signed in to change notification settings - Fork 24.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider query when optimizing date rounding #63403
Merged
Merged
Commits on Oct 7, 2020
-
Consider query when optimizing date rounding
Before this change we inspected the index when optimizing `date_histogram` aggregations, precalculating the divisions for the buckets for the entire range of dates on the index so long as there aren't a ton of these buckets. This works very well when you query all of the dates in the index which is quite common - after all, folks frequently want to query a week of data and have daily indices. But it doesn't work as well when the index is much larger than the query. This is quite common when dumping data into ES just to investigate it but less common in the traditional time series use case. But even there it still happens, it is just less impactful. Consider the default query produced by Kibana's Discover app: a range of 15 minutes and a interval of 30 seconds. This optimization saves something like 3 to 12 nanoseconds per document, so that 15 minutes would have to have hundreds of millions of documents for it to be impactful. Anyway, this commit takes the query into account when precalculating the buckets. Mostly this is good when you have "dirty data". Immagine loading 80 billion docs in an index to investigate them. Most of them have dates around 2015 and 2016 but some have dates in 1970 and others have dates in 2030. These outlier dates are "dirty" "garbage". Well, without this change a `date_histogram` across many of these docs is significantly slowed down because we don't precalculate the range due to the outliers. That's just rude! So this change takes the query into account. The bulk of the code change here is plumbing the query into place. It turns out that its a *ton* of plumbing, so instead of just adding a `Query` member in hundreds of args replace `QueryShardContext` with a new `AggregationContext` which does two things: 1. Has the top level `Query`. 2. Exposes just the parts of `QueryShardContext` that we actually need to run aggregation. This lets us simplify a few tests now and will let us simplify many, many tests later.
Configuration menu - View commit details
-
Copy full SHA for e4c4f68 - Browse repository at this point
Copy the full SHA e4c4f68View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3c6953c - Browse repository at this point
Copy the full SHA 3c6953cView commit details -
Configuration menu - View commit details
-
Copy full SHA for b25aae8 - Browse repository at this point
Copy the full SHA b25aae8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 788846f - Browse repository at this point
Copy the full SHA 788846fView commit details
Commits on Oct 12, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 569e0e7 - Browse repository at this point
Copy the full SHA 569e0e7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d7e639 - Browse repository at this point
Copy the full SHA 3d7e639View commit details -
Configuration menu - View commit details
-
Copy full SHA for 04caadd - Browse repository at this point
Copy the full SHA 04caaddView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3316c9f - Browse repository at this point
Copy the full SHA 3316c9fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4b3073b - Browse repository at this point
Copy the full SHA 4b3073bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d8cd95 - Browse repository at this point
Copy the full SHA 6d8cd95View commit details -
Configuration menu - View commit details
-
Copy full SHA for 18ac471 - Browse repository at this point
Copy the full SHA 18ac471View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.