Split incoming requests by day and run them in parallel.#995
Split incoming requests by day and run them in parallel.#995tomwilkie wants to merge 1 commit intocortexproject:masterfrom
Conversation
tomwilkie
force-pushed
the
parallelise-multi-day-queries
branch
from
14e98f5 to
ae29f0f
Compare
tomwilkie
changed the title
| f.IntVar(&cfg.MaxOutstandingPerTenant, "querier.max-outstanding-requests-per-tenant", 100, "Maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429.") | ||
| f.IntVar(&cfg.MaxRetries, "querier.max-retries-per-request", 5, "Maximum number of retries for a single request; beyon this, the downstream error is returned.") | ||
| f.IntVar(&cfg.MaxRetries, "querier.max-retries-per-request", 5, "Maximum number of retries for a single request; beyond this, the downstream error is returned.") | ||
| f.BoolVar(&cfg.SplitQueriesByDay, "querier.split-queries-by-day", false, "Split queries by day and execute in parallel.") |
There was a problem hiding this comment.
Why day? It's a nice time boundary, but I'm curious if there was reasoning here that makes it the off/on choice, versus being able to specify the range to split on. If someone was using periodic tables with a week range, is there a difference to splitting the query over that time boundary instead?
There was a problem hiding this comment.
Good question! The rows in the index are organised by day, and sub day queries pretty much have to read an entire days index entries anyway. Therefore might as well parallelise by day IMO.
We're running this in prod now and it actually looks like for high-cardinality queries where the execution of the PromQL is the dominant latency, sub-day parallelism might be worth while. In the future we should probably make this tuneable.
- Generic code to parse incoming query_range requests, mutate them and round trip them. - Split queries along day boundaries, modulo step. - Run queries in parallel and combine their results. - Ensure we propagate org ids correctly; add e2e tests. - Take care to ensure we propagate trace IDs correctly; involved updating weaveworks/common. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
tomwilkie
force-pushed
the
parallelise-multi-day-queries
branch
from
af16d46 to
3502fb4
Compare
|
Going to roll this into #1029, as there are review comments there. |
2 participants
Continuation of the work proposed in https://docs.google.com/document/d/1lsvSkv0tiAMPQv-V8vI2LZ8f4i9JuTRsuPI_i-XcAqY/edit?usp=drive_web&ouid=103586900408483314805.
Fixes #963, fixes #266
Need to merge https://github.com/weaveworks/common/pull/ firstSigned-off-by: Tom Wilkie tom.wilkie@gmail.com