Skip to content

feat: Implement Lazy Reader#3395

Merged
Xuanwo merged 54 commits intomainfrom
lazy-reader
Oct 31, 2023
Merged

feat: Implement Lazy Reader#3395
Xuanwo merged 54 commits intomainfrom
lazy-reader

Conversation

@Xuanwo
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo commented Oct 26, 2023

This PR implements Lazy Reader which will only send real IO when users call read/seek/next.

Close #3359


The mainly changes in this PR:

RpRead's Metadata changes

Metadata (the content_length) is not required to implement read anymore. We use size: Option<u64> to replace the metadata to a optional size hint. Only service that known the content length need to fill it.

Lazy Reader

We implement lazy reader upon RangeReader and FileReader. To make it easier, we refactor some code for TokioReader and FuturesReader.

Xuanwo added 21 commits October 25, 2023 11:40
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
@github-actions github-actions Bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Oct 26, 2023
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Xuanwo added 11 commits October 30, 2023 11:27
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
@Xuanwo Xuanwo marked this pull request as ready for review October 30, 2023 10:08
@Xuanwo Xuanwo requested a review from PsiACE as a code owner October 30, 2023 10:08
Signed-off-by: Xuanwo <github@xuanwo.io>
@Xuanwo
Copy link
Copy Markdown
Member Author

Xuanwo commented Oct 31, 2023

CI failure related to #3429, not introduced by this PR.

Copy link
Copy Markdown
Member

@PsiACE PsiACE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total look good to me. Want to know if users have the opportunity to customize their own opendal read and write policies.

@Xuanwo
Copy link
Copy Markdown
Member Author

Xuanwo commented Oct 31, 2023

Want to know if users have the opportunity to customize their own opendal read and write policies.

For example?

@suyanhanx
Copy link
Copy Markdown
Member

For example?

Could this change be an opt-out?

@PsiACE
Copy link
Copy Markdown
Member

PsiACE commented Oct 31, 2023

For example?

I haven't thought it through yet, just exploring if there is such a possibility. Let's temporarily skip this additional issue. Once I have a clear idea, I will try to write something for discussion.

@Xuanwo
Copy link
Copy Markdown
Member Author

Xuanwo commented Oct 31, 2023

Could this change be an opt-out?

I'm gussing it's impossible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat"

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking issues of RFC-3356: Lazy Reader

3 participants