Kibana 4 unable to utilize query cache

[jimmyjones2]

Currently Kibana 4 is unable to use the query cache, see elastic/kibana#2577, which is a massive shame as the query cache should be great for the logging use case.

The crux of the issue, is when Kibana queries last 7 days of eg. daily logstash indices, the start and end dates change with every query so cannot be cached. However the result from the "middle" 5 indices won't change as the timerange always covers the full index, so in theory could be cached. However currently there seems no way to do this, as the indices are queried together and the same query is sent to each.

[clintongormley]

Hi @jimmyjones2

Yes, you're absolutely right. We've been bouncing ideas around including:

• supporting a post-cache filter for timestamp which isn't included in the cached query
• rewriting the range filter before caching to replace the min/max with the actual min/max present in the segment
• supporting a different way of specifying now which can avoid cache misses

We'd very much like to improve this issue, but the best way forward is not entirely clear yet.

[s1monw]

I think we are all in agreement that this is desireable. The only problem is that it's technically not very simple to do. It boils down to comparing two queries that are not the same and make a decision if the query results are the same before we execute the second query. I will mark this as a high-hanging fruit.

[evilezh]

Just an idea ....
do like cron job ...
define at what time to refresh (which should give exact time ranges e.g. every 10th second of minute.. every minute of hour etc.).
That would produce exact time ranges. if we run over last 7 days .. and we have per day index ... then for 6 previous days query should match ...
So .. if kibana can produce it .. then we should get cache hits i guess.

[ebuildy]

Nothing ready yet? It's very crucial when querying huge indices, maybe a small blog post about putting a cache-proxy between Kibana and elasticsearch could help?

[colings86]

@ebuildy This issue was closed by #16870 as referenced above and will be available in 5.0. There is also a three part blog series starting with the following post which explains what we did, why we did it and the effect is has on query performance: https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters