Added IO throttling for the merger pipeline.#2205
Merged
fulmicoton merged 2 commits intomainfrom Nov 3, 2022
Merged
Conversation
8fef013 to
26f2d19
Compare
This was referenced Oct 31, 2022
3def525 to
d68cba0
Compare
4df0511 to
7f26f4d
Compare
7f26f4d to
e89a6e7
Compare
d0c386f to
39c9ed2
Compare
fmassot
reviewed
Nov 1, 2022
39c9ed2 to
63b98e6
Compare
fmassot
reviewed
Nov 1, 2022
fmassot
reviewed
Nov 1, 2022
63b98e6 to
79ffe6f
Compare
fulmicoton
commented
Nov 2, 2022
Collaborator
Author
There was a problem hiding this comment.
this just became a method. Diff is not very good at that, but the changed is trivial.
fulmicoton
commented
Nov 2, 2022
fulmicoton
commented
Nov 2, 2022
Collaborator
Author
There was a problem hiding this comment.
index serialization used to happen here, which is why we took over the controlled directory in the packager.
Today the serialization logic was moved int he IndexSerializer so I moved these lines to the IndexSerializer.
fulmicoton
commented
Nov 2, 2022
fulmicoton
commented
Nov 2, 2022
Collaborator
Author
There was a problem hiding this comment.
That logic is replaced by the ControlledWrite in quickwit_common/src/io.rs
guilload
approved these changes
Nov 2, 2022
79ffe6f to
3afa97a
Compare
The `IoControls` object makes it possible to - limit the throughput of Write and AsyncWrite object, - abort a write if the killswitch is activated, - record progress when performing IO - record the number of writes write in a Prometheus counter. In this PR, the IoControls object is plugged into the the indexer, the merger and the delete pipeline. The merge and the delete pipeline can optionally get a separate write throughput limit, as defined in the index configuration. In the merge and the delete pipeline, split downloads and actual index writing share the same throughput limit, and attempt to consume it in a concurrent manner. We rely on async-speed-limit for the throttling. We could not use async_speed_limit::Resource directly as it is not compatible with tokio. This PR also does some minor refactoring on the merge pipeline parameters.
32cc4e9 to
8cfbe7a
Compare
8cfbe7a to
b8fac1b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.