Setting to disable rewrite-to-filters optimization #73426
Comments
elasticmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
|
The big question I had in my mind was over the default for the switch - whether folks would explicitly choose to opt in for the whizzy date optimisation or opt out of scenarios when it goes wrong. |
I talked to @markharwood about this earlier. I think the right thing is to give folks an out, but try to be fairly aggressive about turning on the optimizations. We benchmark and stuff, but we'll never hit all the cases. And this'd give us an out. I've got the makings of a cluster setting you can use to disable the optimization when the top level query contains some kind of query. You can specify the kind of query. You specify |
Sometimes our fancy "run this agg as a Query" optimizations end up slower than running the aggregation in the old way. We know that and use heuristics to dissable the optimization in that case. But it turns out that the process of running the heuristics itself can be slow, depending on the query. Worse, changing the heuristics requires an upgrade, which means waiting. If the heurisics make a terrible choice folks need a quick way out. This adds such a way: a cluster level setting that contains a list of queries that are considered "too expensive" to try and optimize. If the top level query contains any of those queries we'll disable the "run as Query" optimization. The default for this settings is wildcard and term-in-set queries, which is fairly conservative. There are certainly wildcard and term-in-set queries that the optimization works well with, but there are other queries of that type that it works very badly with. So we're being careful. Better, you can modify this setting in a running cluster to disable the optimization if we find a new type of query that doesn't work well. Closes elastic#73426
Sometimes our fancy "run this agg as a Query" optimizations end up slower than running the aggregation in the old way. We know that and use heuristics to dissable the optimization in that case. But it turns out that the process of running the heuristics itself can be slow, depending on the query. Worse, changing the heuristics requires an upgrade, which means waiting. If the heurisics make a terrible choice folks need a quick way out. This adds such a way: a cluster level setting that contains a list of queries that are considered "too expensive" to try and optimize. If the top level query contains any of those queries we'll disable the "run as Query" optimization. The default for this settings is wildcard and term-in-set queries, which is fairly conservative. There are certainly wildcard and term-in-set queries that the optimization works well with, but there are other queries of that type that it works very badly with. So we're being careful. Better, you can modify this setting in a running cluster to disable the optimization if we find a new type of query that doesn't work well. Closes #73426
Sometimes our fancy "run this agg as a Query" optimizations end up slower than running the aggregation in the old way. We know that and use heuristics to dissable the optimization in that case. But it turns out that the process of running the heuristics itself can be slow, depending on the query. Worse, changing the heuristics requires an upgrade, which means waiting. If the heurisics make a terrible choice folks need a quick way out. This adds such a way: a cluster level setting that contains a list of queries that are considered "too expensive" to try and optimize. If the top level query contains any of those queries we'll disable the "run as Query" optimization. The default for this settings is wildcard and term-in-set queries, which is fairly conservative. There are certainly wildcard and term-in-set queries that the optimization works well with, but there are other queries of that type that it works very badly with. So we're being careful. Better, you can modify this setting in a running cluster to disable the optimization if we find a new type of query that doesn't work well. Closes elastic#73426
Sometimes our fancy "run this agg as a Query" optimizations end up slower than running the aggregation in the old way. We know that and use heuristics to dissable the optimization in that case. But it turns out that the process of running the heuristics itself can be slow, depending on the query. Worse, changing the heuristics requires an upgrade, which means waiting. If the heurisics make a terrible choice folks need a quick way out. This adds such a way: a cluster level setting that contains a list of queries that are considered "too expensive" to try and optimize. If the top level query contains any of those queries we'll disable the "run as Query" optimization. The default for this settings is wildcard and term-in-set queries, which is fairly conservative. There are certainly wildcard and term-in-set queries that the optimization works well with, but there are other queries of that type that it works very badly with. So we're being careful. Better, you can modify this setting in a running cluster to disable the optimization if we find a new type of query that doesn't work well. Closes elastic#73426
Sometimes our fancy "run this agg as a Query" optimizations end up slower than running the aggregation in the old way. We know that and use heuristics to dissable the optimization in that case. But it turns out that the process of running the heuristics itself can be slow, depending on the query. Worse, changing the heuristics requires an upgrade, which means waiting. If the heurisics make a terrible choice folks need a quick way out. This adds such a way: a cluster level setting that contains a list of queries that are considered "too expensive" to try and optimize. If the top level query contains any of those queries we'll disable the "run as Query" optimization. The default for this settings is wildcard and term-in-set queries, which is fairly conservative. There are certainly wildcard and term-in-set queries that the optimization works well with, but there are other queries of that type that it works very badly with. So we're being careful. Better, you can modify this setting in a running cluster to disable the optimization if we find a new type of query that doesn't work well. Closes #73426
* Add setting to disable aggs optimization (backport of #73620) Sometimes our fancy "run this agg as a Query" optimizations end up slower than running the aggregation in the old way. We know that and use heuristics to dissable the optimization in that case. But it turns out that the process of running the heuristics itself can be slow, depending on the query. Worse, changing the heuristics requires an upgrade, which means waiting. If the heurisics make a terrible choice folks need a quick way out. This adds such a way: a cluster level setting that contains a list of queries that are considered "too expensive" to try and optimize. If the top level query contains any of those queries we'll disable the "run as Query" optimization. The default for this settings is wildcard and term-in-set queries, which is fairly conservative. There are certainly wildcard and term-in-set queries that the optimization works well with, but there are other queries of that type that it works very badly with. So we're being careful. Better, you can modify this setting in a running cluster to disable the optimization if we find a new type of query that doesn't work well. Closes #73426
|
We just hit a situation where we had to disable this option to keep query performance up. Is it useful to share our configuration somewhere? |
|
Probably! I've been working on a change.to better detect the bad cases up
front but I'd love to see what you hit.
You can post here or email me directly if you'd like. My email is in my
commit metadata.
If you send the _search request or an anonymized copy of it that is
*probably* enough for me to figure out if my later work was enough.
Sorry for all this!
…On Wed, Jun 23, 2021, 06:17 Robert Kenny ***@***.***> wrote:
We just hit a situation where we had to disable this option to keep query
performance up. Is it useful to share our configuration somewhere?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#73426 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABUXIW3D6A32OJ6QMARXMTTUGYB7ANCNFSM45SMKMOQ>
.
|
|
#73620 (comment) should have been maybe placed here |
|
@nik9000 so we've had the same issue which caused quite a lot of panick! Upgraded from 7.9 to 7.13.3 , but actually the slowness almost "only" affected spatial queries which is why we didn't detect at the beginning. Non-spatial queries seem to work fine but spatial queries (using the geospatial type) would take forever and cause our ES to crash. Our index is > 200GB in size, mostly due to geospatial data. And of course, we would want our spatial queries to work much faster, so I wonder if it's something you can investigate? |
|
Hi @aj7, In order to help you can you please add information about the mapping of the geospatial field and a query example where you have experience the slowdown. Thanks! |
|
hi @iverase , of course, I will aim to do this; but also thought of mentioning it here since this i didn't see any mention of spatial queries being affected specifically, and therefore could be something for the ES team to check out.... |
It turns out that sometimes my beautiful date_histogram optimizations aren't always optimizations. Sometimes they make the
date_histogramsuper ultra slow. We've known that and have heuristics that disable the optimization. Those heuristics generally work because, like, we worked on them. But it turns out that they don't work for everyone. Sometimes folks have queries or data or something that doesn't work well with the heuristics we have. I'll be filing a separate issue with the cases we've heard about and fix the heuristics. But I think this is a forever problem. We just won't cover all cases in our tests. So we're going to need an escape hatch to disable the optimization entirely. I believe a cluster level setting is the most sensible thing here. Its a "big hammer", but its one that you can easily modify if you hit the issue without restarting anything and you can disable once you've upgraded to a version that includes the fix.The text was updated successfully, but these errors were encountered: