Skip to content

Added IO throttling for the merger pipeline.#2205

Merged
fulmicoton merged 2 commits intomainfrom
throttling-3
Nov 3, 2022
Merged

Added IO throttling for the merger pipeline.#2205
fulmicoton merged 2 commits intomainfrom
throttling-3

Conversation

@fulmicoton
Copy link
Copy Markdown
Collaborator

@fulmicoton fulmicoton commented Oct 31, 2022

Added generic IO controls.

The `IoControls` object makes it possible to
- limit the throughput of Write and AsyncWrite object,
- abort a write if the killswitch is activated,
- record progress when performing IO
- record the number of writes write in a Prometheus counter.

In this PR, the IoControls object is plugged into the
the indexer, the merger and the delete pipeline.

The merge and the delete pipeline can optionally get a
separate write throughput limit, as defined in the index
configuration.

In the merge and the delete pipeline, split downloads and
actual index writing share the same throughput limit, and
attempt to consume it in a concurrent manner.

We rely on async-speed-limit for the throttling.
We could not use async_speed_limit::Resource directly as
it is not compatible with tokio.

This PR also does some minor refactoring on the merge pipeline
parameters.

throttle

@fulmicoton fulmicoton marked this pull request as draft October 31, 2022 02:35
@fulmicoton fulmicoton force-pushed the throttling-3 branch 7 times, most recently from 8fef013 to 26f2d19 Compare October 31, 2022 06:36
@fulmicoton fulmicoton changed the title Throttling 3 Added IO throttling for the merger pipeline. Oct 31, 2022
@fulmicoton fulmicoton marked this pull request as ready for review October 31, 2022 06:39
@fulmicoton fulmicoton requested a review from guilload October 31, 2022 07:42
@fulmicoton fulmicoton force-pushed the throttling-3 branch 2 times, most recently from 3def525 to d68cba0 Compare October 31, 2022 12:02
@fulmicoton fulmicoton marked this pull request as draft October 31, 2022 12:04
@fulmicoton fulmicoton force-pushed the throttling-3 branch 14 times, most recently from 4df0511 to 7f26f4d Compare November 1, 2022 04:07
@fulmicoton fulmicoton marked this pull request as ready for review November 1, 2022 04:13
@fulmicoton fulmicoton force-pushed the throttling-3 branch 3 times, most recently from d0c386f to 39c9ed2 Compare November 1, 2022 05:23
Comment thread quickwit/quickwit-indexing/src/actors/index_serializer.rs Outdated
Comment thread quickwit/quickwit-indexing/src/actors/index_serializer.rs Outdated
Comment thread quickwit/quickwit-indexing/src/actors/merge_pipeline.rs Outdated
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this just became a method. Diff is not very good at that, but the changed is trivial.

Comment thread quickwit/quickwit-indexing/src/actors/merge_pipeline.rs Outdated
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

index serialization used to happen here, which is why we took over the controlled directory in the packager.

Today the serialization logic was moved int he IndexSerializer so I moved these lines to the IndexSerializer.

Comment thread quickwit/quickwit-indexing/src/controlled_directory.rs Outdated
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That logic is replaced by the ControlledWrite in quickwit_common/src/io.rs

Comment thread quickwit/quickwit-common/src/io.rs Outdated
Comment thread quickwit/quickwit-common/src/io.rs Outdated
Comment thread quickwit/quickwit-common/src/io.rs Outdated
Comment thread quickwit/quickwit-common/src/io.rs Outdated
Comment thread quickwit/quickwit-config/src/index_config.rs Outdated
The `IoControls` object makes it possible to
- limit the throughput of Write and AsyncWrite object,
- abort a write if the killswitch is activated,
- record progress when performing IO
- record the number of writes write in a Prometheus counter.

In this PR, the IoControls object is plugged into the
the indexer, the merger and the delete pipeline.

The merge and the delete pipeline can optionally get a
separate write throughput limit, as defined in the index
configuration.

In the merge and the delete pipeline, split downloads and
actual index writing share the same throughput limit, and
attempt to consume it in a concurrent manner.

We rely on async-speed-limit for the throttling.
We could not use async_speed_limit::Resource directly as
it is not compatible with tokio.

This PR also does some minor refactoring on the merge pipeline
parameters.
@fulmicoton fulmicoton enabled auto-merge (squash) November 3, 2022 00:45
@fulmicoton fulmicoton merged commit ec3e5dd into main Nov 3, 2022
@fulmicoton fulmicoton deleted the throttling-3 branch November 3, 2022 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants