Skip to content

feat(core): adopt WebHDFS LISTSTATUS_BATCH for better performance#2499

Merged
Xuanwo merged 12 commits intoapache:mainfrom
morristai:feat/adopt_webhdfs_liststatus_batch
Jun 21, 2023
Merged

feat(core): adopt WebHDFS LISTSTATUS_BATCH for better performance#2499
Xuanwo merged 12 commits intoapache:mainfrom
morristai:feat/adopt_webhdfs_liststatus_batch

Conversation

@morristai
Copy link
Copy Markdown
Member

@morristai morristai commented Jun 20, 2023

Description

Background

Use the WebHDFS Iteratively_List_a_Directory API in WebHDFS as the default listing behavior, and fallback to One-time Listing API if the server doesn't support batch listing.

@github-actions github-actions Bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Jun 20, 2023
@morristai morristai marked this pull request as draft June 20, 2023 00:21
Xuanwo
Xuanwo reviewed Jun 20, 2023
View reviewed changes
Comment thread core/src/services/webhdfs/backend.rs Outdated
Comment thread core/src/services/webhdfs/backend.rs Outdated
Xuanwo
Xuanwo reviewed Jun 20, 2023
View reviewed changes
Comment thread core/src/services/webhdfs/backend.rs Outdated
@morristai morristai force-pushed the feat/adopt_webhdfs_liststatus_batch branch from a9ac40e to 3f044d6 Compare June 20, 2023 20:20
Xuanwo
Xuanwo reviewed Jun 21, 2023
View reviewed changes
Comment thread core/src/services/webhdfs/backend.rs Outdated
Comment thread core/src/services/webhdfs/backend.rs Outdated
Comment thread core/src/services/webhdfs/message.rs Outdated
Xuanwo
Xuanwo reviewed Jun 21, 2023
View reviewed changes
Comment thread core/src/services/webhdfs/backend.rs
@morristai morristai marked this pull request as ready for review June 21, 2023 04:08
Xuanwo
Xuanwo approved these changes Jun 21, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@Xuanwo Xuanwo merged commit 04852a4 into apache:main Jun 21, 2023
@morristai
Copy link
Copy Markdown
Member Author

morristai commented Jun 21, 2023

BTW, same dilemma on whether to use #[derive(default)] or impl Default. My first implementation used the derive macro, but it required four other sub struct/enum to implement the default macro. Anyway, I've learned a lot from these PR practices. I'll ask for help when I'm stuck again.

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Jun 21, 2023

I'll ask for help when I'm stuck again.

Feel free to ask for help. I am willing to answer any questions you may have!

@PsiACE PsiACE mentioned this pull request Jun 27, 2023
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Xuanwo Xuanwo Xuanwo approved these changes

@PsiACE PsiACE Awaiting requested review from PsiACE

@ClSlaid ClSlaid Awaiting requested review from ClSlaid

@sundy-li sundy-li Awaiting requested review from sundy-li

Assignees

No one assigned

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat"

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance

2 participants

@morristai @Xuanwo