Skip to content

fix(service/hdfs): enable hdfs append support#3600

Merged
Xuanwo merged 5 commits intoapache:mainfrom
qingwen220:hdfs_blocking_append
Nov 17, 2023
Merged

fix(service/hdfs): enable hdfs append support#3600
Xuanwo merged 5 commits intoapache:mainfrom
qingwen220:hdfs_blocking_append

Conversation

@qingwen220
Copy link
Copy Markdown
Contributor

@qingwen220 qingwen220 commented Nov 16, 2023 •
edited by Xuanwo
Loading

support append in block_write

close #2500

@qingwen220 qingwen220 requested a review from Xuanwo as a code owner November 16, 2023 09:29
@qingwen220 qingwen220 requested a review from PsiACE as a code owner November 16, 2023 11:24
@qingwen220 qingwen220 force-pushed the hdfs_blocking_append branch 5 times, most recently from 5f76495 to 48eb645 Compare November 16, 2023 13:23
@qingwen220
Copy link
Copy Markdown
Contributor Author

qingwen220 commented Nov 16, 2023 •
edited
Loading

@Xuanwo After some tests, default and cluster tests fail with different reasons. Below are my guess.

  1. default single node may not support append.
  2. for the distributed cluster, we should add configuration 'dfs_replication=1' on the client side as well. That configuration on my own single-node cluster works. But I do not know how to config in the ut.

Looking forward to your input.

@Xuanwo Xuanwo changed the title bugfix(service/hdfs): enable hdfs blocking append fix(service/hdfs): enable hdfs blocking append Nov 16, 2023
@github-actions github-actions Bot added the releases-note/fix The PR fixes a bug or has a title that begins with "fix" label Nov 16, 2023
Xuanwo
Xuanwo reviewed Nov 16, 2023
View reviewed changes
Comment thread core/src/services/hdfs/backend.rs Outdated
Comment thread .github/workflows/service_test_hdfs.yml
Comment thread .github/workflows/service_test_hdfs.yml
@qingwen220 qingwen220 force-pushed the hdfs_blocking_append branch 2 times, most recently from 7d3998b to 4398cca Compare November 17, 2023 06:18
@qingwen220 qingwen220 force-pushed the hdfs_blocking_append branch 2 times, most recently from 833c4bf to 016a58d Compare November 17, 2023 07:33
Xuanwo
Xuanwo reviewed Nov 17, 2023
View reviewed changes
Comment thread core/src/services/hdfs/backend.rs
Xuanwo
Xuanwo reviewed Nov 17, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other LGTM. We just need to make CI happy. Thanks for your contributions!

Comment thread core/src/services/hdfs/backend.rs Outdated
Comment thread core/src/services/hdfs/backend.rs Outdated
@Xuanwo Xuanwo changed the title fix(service/hdfs): enable hdfs blocking append fix(service/hdfs): enable hdfs append support Nov 17, 2023
Xuanwo
Xuanwo approved these changes Nov 17, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Xuanwo Xuanwo merged commit c9fcb80 into apache:main Nov 17, 2023
@suyanhanx
Copy link
Copy Markdown
Member

suyanhanx commented Nov 17, 2023

Thanks!

@G-XD G-XD mentioned this pull request Nov 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@suyanhanx suyanhanx suyanhanx left review comments

@Xuanwo Xuanwo Xuanwo approved these changes

@PsiACE PsiACE Awaiting requested review from PsiACE

Assignees

No one assigned

Labels

releases-note/fix The PR fixes a bug or has a title that begins with "fix"

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support append for hdfs

3 participants

@qingwen220 @suyanhanx @Xuanwo