Option IO throttling of the merger.#2163
Conversation
fulmicoton
force-pushed
the
throttling-2
branch
8 times, most recently
from
1cd2afb to
9254c59
Compare
fulmicoton
force-pushed
the
throttling-2
branch
from
9254c59 to
25242ef
Compare
| use std::time::{Duration, Instant}; | ||
|
|
||
| use quickwit_common::metrics::IntCounter; | ||
| pub struct Throttle { |
There was a problem hiding this comment.
We might not need a custom implementation: the governor rate limiter wraps an AtomicU64, so cloneable with an Arc and has sync (check*) and async until* APIs.
There was a problem hiding this comment.
One thing I am scared with governor is that
a) the docs only talks about requests / operations etc. and does not talk about using it to throttle a throughput
b) the logic that throttles is a loop that have tasks wait for the capacity to be available. I am not fond of that:
I don't know what limits the number of times a thread will have to wake up, I am not sure if there can be some wait of time here.
c) I don't understand how it works so I am not able to reason with it:
For instance, if I have two writers, sharing a rate limiter, and being throttled. One is writing 1kB at a time, when the other is doing it 10kb at a time, what throughput do they achieve?
There was a problem hiding this comment.
created an issue on github: boinkor-net/governor#154
There was a problem hiding this comment.
created an issue on github: boinkor-net/governor#154
There was a problem hiding this comment.
https://github.com/tikv/async-speed-limit seems to do exactly what we need and more (can wrap async read/write directly).
|
Switching to draft, will replace with async-speed-limit |
|
Closed for #2205 |
2 participants
No description provided.