We're flushing virtually empty chunks in prod

[tomwilkie]

Somethings not right:

50th percentile chunk length is ~10!

I suspect https://github.com/weaveworks/cortex/pull/415/files might be to blame.

[tomwilkie]

This also causes the ingesters to not shutdown cleanly, as they try and flush their flush queue after successfully transferring (which they shouldn't).

[aaron7]

Fixed by #449

[tomwilkie]

Worth keeping this open until we actually understand why we are doing this.

[bboreham]

What were the findings of the experiment to set the value to 1 hour?

[bboreham]

If the underlying series change a lot, e.g. in Kubernetes if you start and stop a lot of pods, you will get a lot of very short timeseries.

[bboreham]

Note the "event sampling" feature gives you raw data to understand why series are being flushed.

[bboreham]

I think this issue applies to a specific incident so long ago that we won't make any more progress; closing.