Large change in cache hit rate for 0.9.1

[drcrallen]

We are seeing a significant change in cache hit rate and response rate for 0.9.1 vs 0.9.0.

The above screenshot shows cache hit rate over time. Vertical lines are day boundaries. The discontinuity and drop in cache hit rate directly correlates with the 0.9.1 rollout to this node. Rolling back to 0.9.0 causes an increase in reported cache hit rate.

The Memcached metric [MEM] Response Rate: All (Failure + Success + Retry) correlates with the change in cache hit rate.

The only queries registered on this node are of type dataSourceMetadata, topN, or timeseries.

I don't have enough data yet to determine if there is an impact on query performance.

[gianm]

@drcrallen When the hit rates are depressed, is the majority of the cluster running 0.9.1 or 0.9.0? #2690 changed cache keys for all of the filters, so I would expect a single 0.9.1 node amongst a sea of 0.9.0s to experience depressed hit rates.

[gianm]

If that's what's going on, you should see a difference if you split the hit rate by "has filters" vs "does not have filters".

Also if you're using approximate histograms, #2891 adjusted cache keys for those.

[drcrallen]

@gianm some historicals are on 0.9.1 but not all of them. I'll update this when I have everything on 0.9.1

[drcrallen]

General FYI, this is a communicative measure rather than a bug report, since it is not obvious if this is actually a bug or an artifact of configuration, or a release not that simply needs to be written.

[drcrallen]

Early results are positive that this is an upgrade side-effect that needs a callout in the release notes and not a regression or drop in caching.

[drcrallen]

Will have more information after peak.

[xvrl]

@gianm is it possible to make the cache keys backwards compatible? Is that something we should strive for between releases?

[gianm]

@xvrl In #2891 no, although in #2690 probably yes? Although in general I don't strive for it (& I think others generally don't either) on the grounds that it would potentially clutter the code, the impact is nil if you use local cache, and probably not too awful if you use remote cache.

[drcrallen]

Values have returned to nominal levels. Will update release notes.