Use TSDB's WAL for writes.#1103
Conversation
tomwilkie
force-pushed
the
wal
branch
5 times, most recently
from
047b710 to
4390f26
Compare
- Add github.com/promtheus/tsdb/wal - Update github.com/prometheus/client_golang for WrapRegistererWith function. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
|
@tomwilkie do you plan to fix pr? |
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Rebase with master
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
|
Currently, the idea is to only write the WAL. Reading the WAL on startup without any downtime of ingesters is a little tricky, as reading WAL takes a lot of time - hence need to have ingesters dedicated only to read WAL before it takes any writes. So that would be a follow-up work after this. As for using the written WAL, we can read the WAL and directly flush the chunks to the chunk store in case ingester crashes. A tool/script for to do it would again be a follow up of this (or should I add in the same PR?) We (@gouthamve, @tomwilkie and I) also discussed regarding prometheus's tsdb vs only WAL, and concluded that having tsdb would be very tricky right now as it requires a lot of changes and need to think of a way to handle the churn. Hence planned to go ahead with WAL for now. |
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
codesome
force-pushed
the
wal
branch
3 times, most recently
from
673b0bc to
76dcda0
Compare
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
gouthamve
left a comment
gouthamve
left a comment
There was a problem hiding this comment.
LGTM with nits! Thanks for all the work and patience Ganesh!
| return nil, err | ||
| } | ||
| elapsed := time.Since(start) | ||
| level.Info(util.Logger).Log("msg", "recovery from WAL completed", "time", elapsed.String()) |
There was a problem hiding this comment.
Can we also make this a metric? So that we can compare the duration changes over releases and also correlate it with the number of series, etc.
pkg/ingester/ingester.go
Outdated
|
|
||
| // Push implements client.IngesterServer | ||
| func (i *Ingester) Push(ctx old_ctx.Context, req *client.WriteRequest) (*client.WriteResponse, error) { | ||
|
|
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
pkg/ingester/ingester.go
Outdated
| // A small number of chunks per series - 10*(8^(7-1)) = 2.6m. | ||
| Buckets: prometheus.ExponentialBuckets(10, 8, 7), | ||
| }), | ||
| walReplayDuration: prometheus.NewSummary(prometheus.SummaryOpts{ |
There was a problem hiding this comment.
This could be just a gauge as it doesn't change at all. No need of a summary.
|
@gouthamve in the last commit
|
pkg/ingester/ingester.go
Outdated
There was a problem hiding this comment.
I think this metric is not registered. You should register it below.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
pkg/ingester/ingester.go
Outdated
| } | ||
| } | ||
| client.ReuseSlice(req.Timeseries) | ||
| defer client.ReuseSlice(req.Timeseries) |
There was a problem hiding this comment.
Pointing #2000 here as it also includes the same fix, and there is discussion going on for the fix. I have found WAL to cause panics without that fix, so maybe we want to wait for some conclusion on that PR (merging this PR without this fix would make it unsafe to deploy WAL).
There was a problem hiding this comment.
It has been merged and I have rebased to remove the change from this PR.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
docs/guides/ingesters-with-wal.md
Outdated
| * `--ingester.wal-dir` to the directory where the WAL data should be stores and/or recovered from. | ||
| * `--ingester.wal-enabled` to `true` which enables writing to WAL during ingestion. | ||
| * `--ingester.checkpoint-enabled` to `true` to enable checkpointing of in-memory chunks to disk. This is optional which helps in speeding up the replay process. | ||
| * `--ingester.checkpoint-duration` to the interval at which checkpoints should be created. |
There was a problem hiding this comment.
Can we have a recommendation here (what is the default?)
There was a problem hiding this comment.
The recommendation depends on the need and the ingester size. The default is 30m, will add that here, but we have 15m in our dev.
docs/guides/ingesters-with-wal.md
Outdated
| * `--ingester.recover-from-wal` to `true` to recover data from an existing WAL. The data is recovered even if WAL is disabled and this is set to `true`. The WAL dir needs to be set for this. | ||
| * If you are going to enable WAL, it is advisable to always set this to `true`. | ||
|
|
||
| ## Stuff that is changed automatically when WAL is enabled |
There was a problem hiding this comment.
Changes in lifecycle when WAL is enabled
docs/guides/ingesters-with-wal.md
Outdated
|
|
||
| ## Stuff that is changed automatically when WAL is enabled | ||
|
|
||
| 1. Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there is no 1 ingester leaving and joining each at the same time, rather the same ingester is shut down and broght back again with updated config. Hence flushing is skipped and the data is recovered from the WAL. |
There was a problem hiding this comment.
there are no ingesters that are simultaneously leaving and joining.
docs/guides/ingesters-with-wal.md
Outdated
|
|
||
| To use WAL, there are some changes that needs to be made in the deployment. | ||
|
|
||
| ## Things to change |
There was a problem hiding this comment.
Changes to the deployment
docs/guides/ingesters-with-wal.md
Outdated
|
|
||
| Let's take an example of 4 ingesters. The migration would look something like this: | ||
|
|
||
| 1. Bring up a 1 stateful ingester `ingester-0` and wait till it's ready (accepting read and write requests). |
docs/guides/ingesters-with-wal.md
Outdated
|
|
||
| ### Scale up | ||
|
|
||
| Scaling up is same as what you would do without WAL or statefulsets. Add 1 ingester at a time. |
There was a problem hiding this comment.
Add one ingester. Why do we need to add one at a time? This limitation doesn't exist and I sometimes added several at the same time?
There was a problem hiding this comment.
I thought we should scale up 1 at a time (due to the ring logic?), if that is not the case then I will modify the description.
docs/guides/ingesters-with-wal.md
Outdated
| There are 2 ways to do it, with the latter being a fallback option. | ||
|
|
||
| **First option** | ||
| Consider you have 4 ingesters `ingester-0 ingester-1 ingester-2 ingester-3` and you want to scale down to 2 ingesters, the ingesters which will be shutdown according to statefulset rules are `ingester-2 ingester-3`. |
There was a problem hiding this comment.
ingester 2 ingester-3 --> ingester 3 and then ingester 2
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
9 participants
As per https://docs.google.com/document/d/1n1HcdgmsqaVwqxKg_POZqIREOSqZIAlrxuyx4YYV_pA/edit
Fixes #12