Panic in querier

[gouthamve]

Panic in querier:

level=error ts=2019-04-25T18:31:39.817197708Z caller=engine.go:696 msg="runtime panic in parser" err="runtime error: index out of range" stacktrace="goroutine 1292 [running]:\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).recover(0xc021b498b0, 0xc008308d00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:694 +0xd8\npanic(0x19a6f40, 0x30eb240)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/cortexproject/cortex/pkg/chunk/encoding.(*bigchunk).NewIterator(0xc015a25a70, 0xc004fa5380, 0xc003f41040)\n\t/go/src/github.com/cortexproject/cortex/pkg/chunk/encoding/bigchunk.go:175 +0x16d\ngithub.com/cortexproject/cortex/pkg/querier/batch.(*chunkIterator).reset(...)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/batch/chunk.go:19\ngithub.com/cortexproject/cortex/pkg/querier/batch.newNonOverlappingIterator(0xc01789a380, 0x1, 0x2, 0xc003f41040)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/batch/non_overlapping.go:24 +0xd9\ngithub.com/cortexproject/cortex/pkg/querier/batch.newMergeIterator(0xc001aa5ea0, 0x2, 0x2, 0xc01edd4c20)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/batch/merge.go:28 +0x105\ngithub.com/cortexproject/cortex/pkg/querier/batch.NewChunkMergeIterator(0xc001aa5ea0, 0x2, 0x2, 0x16a55b70960, 0x16a55c38c80, 0x1, 0x1)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/batch/batch.go:37 +0x43\ngithub.com/cortexproject/cortex/pkg/querier.(*chunkSeries).Iterator(0xc0014d5770, 0x13880, 0x1)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/chunk_store_queryable.go:90 +0x57\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc021b498b0, 0x1f43740, 0xc00f7fc2a0, 0xc019385050, 0xc019384f01)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:995 +0xac9\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).rangeEval(0xc021b498b0, 0xc00f7fc720, 0xc008308138, 0x2, 0x2, 0x0, 0x1, 0x1)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:778 +0x147a\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc021b498b0, 0x1f436c0, 0xc021b494a0, 0x1, 0xc019384fc0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:919 +0x2f7c\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).rangeEval(0xc021b498b0, 0xc01edd4c00, 0xc00f7fc2c0, 0x2, 0x2, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:778 +0x147a\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc021b498b0, 0x1f43740, 0xc00f7fc2e0, 0xc008308d00, 0xc021b498b0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:960 +0x123c\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).Eval(0xc021b498b0, 0x1f43740, 0xc00f7fc2e0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:705 +0x80\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc00264c780, 0x1f5e980, 0xc019384b70, 0xc02a4a5e60, 0xc021b494f0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:477 +0xd27\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*Engine).exec(0xc00264c780, 0x1f5e980, 0xc019384b70, 0xc02a4a5e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:389 +0x55b\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*query).Exec(0xc02a4a5e60, 0x1f5e8c0, 0xc0077ce540, 0xc00ded0160)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:179 +0x94\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1.(*API).queryRange(0xc0002b7220, 0xc01d5c2000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:371 +0xb26\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x1f43980, 0xc00f7fc1a0, 0xc01d5c2000)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:205 +0x8e\nnet/http.HandlerFunc.ServeHTTP(0xc0016434e0, 0x1f43980, 0xc00f7fc1a0, 0xc01d5c2000)\n\t/usr/local/go/src/net/http/server.go:1995 +0x44\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x1f19ea0, 0xc0016434e0, 0x1f43dc0, 0xc00f7fc160, 0xc01d5c2000)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x1f43dc0, 0xc00f7fc160, 0xc0002c3f00, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route/route.go:60 +0x276\ngithub.com/cortexproject/cortex/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc00264c840, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3f00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6be\ngithub.com/cortexproject/cortex/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc001643400, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3f00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route/route.go:98 +0x4c\ngithub.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.glob..func1.1(0x1f43dc0, 0xc00f7fc160, 0xc0002c3e00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x123\nnet/http.HandlerFunc.ServeHTTP(0xc001643e40, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3e00)\n\t/usr/local/go/src/net/http/server.go:1995 +0x44\ngithub.com/cortexproject/cortex/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005e4bd0, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3e00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:162 +0xec\ngithub.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0x1f43d80, 0xc0077ce500, 0xc0002c3c00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:49 +0x167\nnet/http.HandlerFunc.ServeHTTP(0xc00023f200, 0x1f43d80, 0xc0077ce500, 0xc0002c3c00)\n\t/usr/local/go/src/net/http/server.go:1995 +0x44\ngithub.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.Log.Wrap.func1(0x1f42ec0, 0xc00f7fc140, 0xc0002c3c00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:41 +0x1a6\nnet/http.HandlerFunc.ServeHTTP(0xc00023f230, 0x1f42ec0, 0xc00f7fc140, 0xc0002c3c00)\n\t/usr/local/go/src/net/http/server.go:1995 +0x44\ngithub.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x1f505c0, 0xc0077ce380, 0xc0002c3b00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x517\nnet/http.HandlerFunc.ServeHTTP(0xc00023f260, 0x1f505c0, 0xc0077ce380, 0xc0002c3b00)\n\t/usr/local/go/src/net/http/server.go:1995 +0x44\ngithub.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.Tracer.Wrap.func2(0x1f505c0, 0xc0077ce380, 0xc0002c3b00)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_tracing.go:39 +0x8c\nnet/http.HandlerFunc.ServeHTTP(0xc00023f290, 0x1f505c0, 0xc0077ce380, 0xc0002c3b00)\n\t/usr/local/go/src/net/http/server.go:1995 +0x44\ngithub.com/cortexproject/cortex/vendor/github.com/weaveworks/common/httpgrpc/server.Server.Handle(0x1f19ea0, 0xc00023f290, 0x1f5e8c0, 0xc00023a940, 0xc021b49360, 0xc00841ee80, 0x0, 0x0)\n\t/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/httpgrpc/server/server.go:50 +0x352\ngithub.com/cortexproject/cortex/pkg/querier/frontend.(*worker).process(0xc001e18240, 0x1f5e8c0, 0xc00023a940, 0x1f7f460, 0xc0005a1d20, 0x0, 0x1f7f460)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/frontend/worker.go:193 +0x96\ngithub.com/cortexproject/cortex/pkg/querier/frontend.(*worker).runOne(0xc001e18240, 0x1f5e8c0, 0xc00023a940, 0x1f153a0, 0xc00000e310)\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/frontend/worker.go:175 +0x3e4\ncreated by github.com/cortexproject/cortex/pkg/querier/frontend.(*worker).runMany\n\t/go/src/github.com/cortexproject/cortex/pkg/querier/frontend/worker.go:157 +0xf6\n"

Some cleaning up later:

goroutine 1292 [running]:
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).recover(0xc021b498b0, 0xc008308d00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:694 +0xd8
panic(0x19a6f40, 0x30eb240)
    /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/cortexproject/cortex/pkg/chunk/encoding.(*bigchunk).NewIterator(0xc015a25a70, 0xc004fa5380, 0xc003f41040)
    /go/src/github.com/cortexproject/cortex/pkg/chunk/encoding/bigchunk.go:175 +0x16d
github.com/cortexproject/cortex/pkg/querier/batch.(*chunkIterator).reset(...)
    /go/src/github.com/cortexproject/cortex/pkg/querier/batch/chunk.go:19
github.com/cortexproject/cortex/pkg/querier/batch.newNonOverlappingIterator(0xc01789a380, 0x1, 0x2, 0xc003f41040)
    /go/src/github.com/cortexproject/cortex/pkg/querier/batch/non_overlapping.go:24 +0xd9
github.com/cortexproject/cortex/pkg/querier/batch.newMergeIterator(0xc001aa5ea0, 0x2, 0x2, 0xc01edd4c20)
    /go/src/github.com/cortexproject/cortex/pkg/querier/batch/merge.go:28 +0x105
github.com/cortexproject/cortex/pkg/querier/batch.NewChunkMergeIterator(0xc001aa5ea0, 0x2, 0x2, 0x16a55b70960, 0x16a55c38c80, 0x1, 0x1)
    /go/src/github.com/cortexproject/cortex/pkg/querier/batch/batch.go:37 +0x43
github.com/cortexproject/cortex/pkg/querier.(*chunkSeries).Iterator(0xc0014d5770, 0x13880, 0x1)
    /go/src/github.com/cortexproject/cortex/pkg/querier/chunk_store_queryable.go:90 +0x57
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc021b498b0, 0x1f43740, 0xc00f7fc2a0, 0xc019385050, 0xc019384f01)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:995 +0xac9
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).rangeEval(0xc021b498b0, 0xc00f7fc720, 0xc008308138, 0x2, 0x2, 0x0, 0x1, 0x1)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:778 +0x147a
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc021b498b0, 0x1f436c0, 0xc021b494a0, 0x1, 0xc019384fc0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:919 +0x2f7c
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).rangeEval(0xc021b498b0, 0xc01edd4c00, 0xc00f7fc2c0, 0x2, 0x2, 0x0, 0x0, 0x0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:778 +0x147a
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc021b498b0, 0x1f43740, 0xc00f7fc2e0, 0xc008308d00, 0xc021b498b0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:960 +0x123c
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*evaluator).Eval(0xc021b498b0, 0x1f43740, 0xc00f7fc2e0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:705 +0x80
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc00264c780, 0x1f5e980, 0xc019384b70, 0xc02a4a5e60, 0xc021b494f0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:477 +0xd27
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*Engine).exec(0xc00264c780, 0x1f5e980, 0xc019384b70, 0xc02a4a5e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:389 +0x55b
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql.(*query).Exec(0xc02a4a5e60, 0x1f5e8c0, 0xc0077ce540, 0xc00ded0160)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/promql/engine.go:179 +0x94
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1.(*API).queryRange(0xc0002b7220, 0xc01d5c2000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:371 +0xb26
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x1f43980, 0xc00f7fc1a0, 0xc01d5c2000)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:205 +0x8e
net/http.HandlerFunc.ServeHTTP(0xc0016434e0, 0x1f43980, 0xc00f7fc1a0, 0xc01d5c2000)
    /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x1f19ea0, 0xc0016434e0, 0x1f43dc0, 0xc00f7fc160, 0xc01d5c2000)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c
github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x1f43dc0, 0xc00f7fc160, 0xc0002c3f00, 0x0, 0x0, 0x0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route/route.go:60 +0x276
github.com/cortexproject/cortex/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc00264c840, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3f00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6be
github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc001643400, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3f00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route/route.go:98 +0x4c
github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.glob..func1.1(0x1f43dc0, 0xc00f7fc160, 0xc0002c3e00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x123
net/http.HandlerFunc.ServeHTTP(0xc001643e40, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3e00)
    /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/cortexproject/cortex/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005e4bd0, 0x1f43dc0, 0xc00f7fc160, 0xc0002c3e00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:162 +0xec
github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0x1f43d80, 0xc0077ce500, 0xc0002c3c00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:49 +0x167
net/http.HandlerFunc.ServeHTTP(0xc00023f200, 0x1f43d80, 0xc0077ce500, 0xc0002c3c00)
    /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.Log.Wrap.func1(0x1f42ec0, 0xc00f7fc140, 0xc0002c3c00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:41 +0x1a6
net/http.HandlerFunc.ServeHTTP(0xc00023f230, 0x1f42ec0, 0xc00f7fc140, 0xc0002c3c00)
    /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x1f505c0, 0xc0077ce380, 0xc0002c3b00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x517
net/http.HandlerFunc.ServeHTTP(0xc00023f260, 0x1f505c0, 0xc0077ce380, 0xc0002c3b00)
    /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware.Tracer.Wrap.func2(0x1f505c0, 0xc0077ce380, 0xc0002c3b00)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_tracing.go:39 +0x8c
net/http.HandlerFunc.ServeHTTP(0xc00023f290, 0x1f505c0, 0xc0077ce380, 0xc0002c3b00)
    /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/httpgrpc/server.Server.Handle(0x1f19ea0, 0xc00023f290, 0x1f5e8c0, 0xc00023a940, 0xc021b49360, 0xc00841ee80, 0x0, 0x0)
    /go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/httpgrpc/server/server.go:50 +0x352
github.com/cortexproject/cortex/pkg/querier/frontend.(*worker).process(0xc001e18240, 0x1f5e8c0, 0xc00023a940, 0x1f7f460, 0xc0005a1d20, 0x0, 0x1f7f460)
    /go/src/github.com/cortexproject/cortex/pkg/querier/frontend/worker.go:193 +0x96
github.com/cortexproject/cortex/pkg/querier/frontend.(*worker).runOne(0xc001e18240, 0x1f5e8c0, 0xc00023a940, 0x1f153a0, 0xc00000e310)
    /go/src/github.com/cortexproject/cortex/pkg/querier/frontend/worker.go:175 +0x3e4
created by github.com/cortexproject/cortex/pkg/querier/frontend.(*worker).runMany

[bboreham]

Do you know how it got into this state? Chunks are normally created when we have a sample, so how did we manage to get an empty one?

[gouthamve]

You're right. I have been going through the codebase for the past 30mins trying to figure it out. We call newBigchunk()` and then add a sample to it. We lock the series for the entire duration so I cannot find how we can use that empty chunk in a query.

Parts where chunk creation and addition happen: https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester.go#L264-L283 (don't see a reason we error out on an empty chunk though).

Code which we call on query path:

cortex/pkg/ingester/user_state.go

Lines 291 to 306 in 1d0ff21

	u.fpLocker.Lock(fp)
	series, ok := u.fpToSeries.get(fp)
	if !ok {
	u.fpLocker.Unlock(fp)
	continue
	}
	for _, filter := range filters {
	if !filter.Matches(series.metric.Get(filter.Name)) {
	u.fpLocker.Unlock(fp)
	continue outer
	}
	}
	err := add(ctx, fp, series)
	u.fpLocker.Unlock(fp)

Not sure how we can query an empty chunk :/

[gouthamve]

Figured out, so the chunks being empty is in the querier and if you're using QueryStream (-querier.ingester-streaming=true), then there is a chance that empty series are sent: https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester.go#L387-L393

[bboreham]

Can you be more explicit how the series get to be empty?
Is it when the series exists but all chunks are outside of the time-window requested?