feat(layer): add throttle layer

[morristai]

Description

• closed: Add Throttle Support #743

Background

NOTE: This is still a early stage of this PR, I open this PR for RFC and guidance.

As there's no open-source repository that uses Governor to limit bandwidth so far IMK, it's mostly used to limit the number of requests, e.g. tower-governor (although Governor can theoretically limit any element).

There's an issue asked by Quickwit's author about how to use Governor to limit IO throughput, but he ended up using async-speed-limit for Quickwit (PR, which also seems to use a truncated byte stream implementation.).

My current implementation uses the answer provided in the above issue by the Governor author. However, it's rough and untested, and there are some uncertainties that I left comments inline. Things I need more information to confirm include:

• Is it possible to limit read (async/blocking) operations since the layer can't affect the inner.read() implementor's behavior?(I assume)?

Answer by Xuanwo: "We can get the read size and wait for them. But it may be complex for waiting in poll_read. We can finish it in a new PR."

• For my current implementations for Write (async/blocking) and Append operations, I use the Bytes truncation method to limit inner consumption. I don't know if this direction is the right approach so far.

Answer by Xuanwo: #2444 (comment)

[Xuanwo]

Mostly LGTM, only some naming related issues.

[Xuanwo]

Thanks a lot!

[morristai]

BTW, there are still 2 todo item comments inside this PR:

• Can we apply a throttle layer over oio::Read/oio::BlockingRead operations?
• When the query is valid but the Decider cannot accommodate it, should we lock the limiter and wait for the wait_time, or should we let other small requests go first?

If you believe that they are not essential, then I will remove them.

[Xuanwo]

• Can we apply a throttle layer over oio::Read/oio::BlockingRead operations?

We can get the read size and wait for them. But it may be complex for waiting in poll_read. We can finish it in a new PR.

• When the query is valid but the Decider cannot accommodate it, should we lock the limiter and wait for the wait_time, or should we let other small requests go first?

Current behavior LGTM.