Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kibana 4 unable to utilize query cache #9526

Closed
jimmyjones2 opened this issue Feb 2, 2015 · 5 comments
Closed

Kibana 4 unable to utilize query cache #9526

jimmyjones2 opened this issue Feb 2, 2015 · 5 comments
Labels
>enhancement high hanging fruit :Search/Search Search-related issues that do not fall into other categories

Comments

@jimmyjones2
Copy link
Contributor

Currently Kibana 4 is unable to use the query cache, see elastic/kibana#2577, which is a massive shame as the query cache should be great for the logging use case.

The crux of the issue, is when Kibana queries last 7 days of eg. daily logstash indices, the start and end dates change with every query so cannot be cached. However the result from the "middle" 5 indices won't change as the timerange always covers the full index, so in theory could be cached. However currently there seems no way to do this, as the indices are queried together and the same query is sent to each.
@clintongormley clintongormley added >enhancement :Search/Search Search-related issues that do not fall into other categories discuss labels Feb 5, 2015
@clintongormley
Copy link
Contributor

Hi @jimmyjones2

Yes, you're absolutely right. We've been bouncing ideas around including:

  • supporting a post-cache filter for timestamp which isn't included in the cached query
  • rewriting the range filter before caching to replace the min/max with the actual min/max present in the segment
  • supporting a different way of specifying now which can avoid cache misses

We'd very much like to improve this issue, but the best way forward is not entirely clear yet.

@s1monw
Copy link
Contributor

s1monw commented Feb 13, 2015

I think we are all in agreement that this is desireable. The only problem is that it's technically not very simple to do. It boils down to comparing two queries that are not the same and make a decision if the query results are the same before we execute the second query. I will mark this as a high-hanging fruit.

s1monw added a commit to s1monw/elasticsearch that referenced this issue Feb 26, 2015
@s1monw
[PROTOTYPE] Merge TermQueryParser and TermQueryBuilder into TermQuery
d812396 
This refactoring splits the parsing and the creation of the lucene query
  This has a couple of advantages
   * XContent parsing creation are in one file and can be tested more easily
   * the class allows a typed in-memory representation of the query that can be modified before a lucene query is build
   * the query can be normalized and serialized via Streamable to be used as a normalized cache key (not depending on the order of the keys in the XContent)
   * the query can be parsed on the coordinating node to allow document prefetching etc. forwarding to the executing nodes would work via Streamable binary representation --> elastic#8150
   * for the query cache a query tree can be "walked" to rewrite range queries into match all queries with MIN/MAX terms to get cache hits for sliding windows --> elastic#9526
   * code wise two classes are merged into one which is nice
   * filter and query can maybe share once class and we add a `toFilter(QueryParserContenxt ctx)` method that returns a filter and by default return a `new QueryWrapperFilter(toQuery(context));`
@s1monw s1monw mentioned this issue Feb 26, 2015
Closed
@clintongormley clintongormley mentioned this issue Mar 23, 2015
Closed
@evilezh
Copy link

evilezh commented Oct 18, 2015

Just an idea ....
do like cron job ...
define at what time to refresh (which should give exact time ranges e.g. every 10th second of minute.. every minute of hour etc.).
That would produce exact time ranges. if we run over last 7 days .. and we have per day index ... then for 6 previous days query should match ...
So .. if kibana can produce it .. then we should get cache hits i guess.

@s1monw s1monw mentioned this issue Mar 2, 2016
Merged
@ebuildy
Copy link
Contributor

ebuildy commented Oct 18, 2016

Nothing ready yet? It's very crucial when querying huge indices, maybe a small blog post about putting a cache-proxy between Kibana and elasticsearch could help?

@colings86
Copy link
Contributor

colings86 commented Oct 18, 2016 •
edited
Loading

@ebuildy This issue was closed by #16870 as referenced above and will be available in 5.0. There is also a three part blog series starting with the following post which explains what we did, why we did it and the effect is has on query performance: https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>enhancement high hanging fruit :Search/Search Search-related issues that do not fall into other categories
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@clintongormley @colings86 @s1monw @ebuildy @jimmyjones2 @evilezh