feat(writer): configurable buffer size of unsized write

[wcy-fdu]

Currently only support configurable write buffer size for s3, gcs, and oss.
close #2087

[Xuanwo]

I don't like current API:

• Users should be able to specify the buffer size during write operation. (maybe not true?)
• There are different buffer size/limit here, we should make the behavior more clear.
  • S3's buffer size requires every write size >= size
  • GCS's buffer size requires every write size == size to align with 256 KiB

For my current understanding, there are at least the following buffer related sizes:

• the at_least_size for write operation: every write will have at least n bytes
• the exact_size for write operation: every write will have exactly n times of bytes
• the at_most_size for write operation: storage can buffer at most n bytes before write

I'm also seeking for better names. Any ideas?

---

Seems we do need to add value in config, the best value cloud be different for services.

[xyjixyjixyji]

I'm also seeking for better names. Any ideas?

I think it is ok to specify the buffer size for each write, since one writer is constructed per-write operation.

Then maybe this write pattern could be a field in OpWrite like

enum Buffersize {
   AtLeast(usize),
   Exact(usize),
   AtMost(usize),
}

[Xuanwo]

Then maybe this write pattern could be a field in OpWrite like

No, too much. And they could be set at the same time, for example, at least 4MiB, at most 16 MiB.

[suyanhanx]

Why not just upper/lower? 👀

[wcy-fdu]

I don't like current API:

• Users should be able to specify the buffer size during write operation. (maybe not true?)

• There are different buffer size/limit here, we should make the behavior more clear.

  • S3's buffer size requires every write size >= size
  • GCS's buffer size requires every write size == size to align with 256 KiB

For my current understanding, there are at least the following buffer related sizes:

• the at_least_size for write operation: every write will have at least n bytes
• the exact_size for write operation: every write will have exactly n times of bytes
• the at_most_size for write operation: storage can buffer at most n bytes before write

I'm also seeking for better names. Any ideas?

Seems we do need to add value in config, the best value cloud be different for services.

Seem that different services need different semantics, but maybe maintaining three related items is a little redundant. Waiting for an exact solution(naming scheme).

[wcy-fdu]

Any suggestion?
If there is no good solution for the time being, we can close this pr as well.

[xyjixyjixyji]

Any suggestion? If there is no good solution for the time being, we can close this pr as well.

I think this need some further discussion before we implement this.

[Xuanwo]

Great, thanks a lot!

[Xuanwo]

Mostly LGTM, the only thing left is to resolve the conflicts.