Bigger chunks

[tomwilkie]

I'd like to build a new chunk format that results in reliably bigger chunks. Effectively I want to build a "meta chunk" which itself is a series of chunks. These chunks won't be fixed to 1kb in size, but instead to 12hrs in length.

Any thoughts on this? Going to experiment today.

If this works I suspect it will solve:

• Grow chunks in memory #300 Grow chunks in memory
• We're flushing virtually empty chunks in prod #448 We're flushing virtually empty chunks in prod
• Use Prom2 chunk code #766 Use Prom2 chunk code (I'm going to investigate using this for the new code)

[bboreham]

I have spoken (in Slack and/or the community meeting) about "multi-chunks", which can cover a number of cases.

For example putting all buckets of a histogram in one block in the storage, for efficiency.
This needs other changes to indexing on ingest and query; just noting it here for now.

Also possibly doing a mass rewrite of an entire day to put all data for the same series/bucket in one multi-chunk. This collapses the v9 schema to about the same as v6; removes the need to do N(series) queries.

[bboreham]

One of the reasons I added the "event sampling" code was to gather data on why I was getting short chunks. Recommend you turn it on and do some counts, e.g. reason=idle vs reason=chunk full.

[tomwilkie]

Evidence I'm looking at is number of chunks per series on some of our slow queries. We're seeing 30k chunks for a query covering only 512 series (and a single day) - meaning there are ~60 chunks per day, or almost 1 per hour per ingester.

One of the reasons I added the "event sampling" code was to gather data on why I was getting short chunks. Recommend you turn it on and do some counts, e.g. reason=idle vs reason=chunk full.

This seem useful beyond event logging, so I'll add a counter for this.

For example putting all buckets of a histogram in one block in the storage, for efficiency.
This needs other changes to indexing on ingest and query; just noting it here for now.

This probably deserves an issue in its own right - I'm just planning on looking chunks for a single series here.

Also possibly doing a mass rewrite of an entire day to put all data for the same series/bucket in one multi-chunk. This collapses the v9 schema to about the same as v6; removes the need to do N(series) queries.

Probably also deserves it own ticket; IMO this would be too expensive to do right now, but once we put TSDB in we should be able to do this with "compactions".

IFAICT there are a few reasons why a chunk might be "smaller" (in time) than expected:

1. The series stopped being written to, for instance because a pod went away.
2. The chunk became too old.
3. The chunk became full.

I'm looking at the last case; I think most of our cases are where the chunk is "full" mainly because of variation in the values - varbit won't transcode the chunk if the value doesn't fit in the chosen encoding and the chunk is more than half full.

Anyway, will add the counters to confirm.

[bboreham]

IFAICT there are a few reasons

https://github.com/cortexproject/cortex/blob/master/pkg/ingester/flush.go#L105-L108

[tomwilkie]

Deployed the counters to one of our test clusters and its predominantly MultipleChunksInSeries, which is when chunk are full I think:

sum by (reason) (increase(cortex_ingester_flush_reasons{reason!="NoFlush"}[10m]))

{reason="Aged"}                   | 232.8205128205128
{reason="Idle"}                   | 1734.3335791868276
{reason="MultipleChunksInSeries"} | 6255.868100944053