Skip to content

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Dec 5, 2024
Merged

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45054

morningman merged 2 commits into from
Dec 5, 2024

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Dec 5, 2024 •
edited by morningman
Loading

cherry-pick (#44267)(#44995)

@suxiaogang223
[fix](hudi) upgrade hudi to 0.15.0 (apache#44267)
29f54a7 
### What problem does this PR solve?

1. upgrade hudi to 0.15.0.
2. impl new hudi jni reader based on hudi-hadoop-mr 
3. add session variable `hudi_jni_scanner` to choose which hudi jni
reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old
HudiJniReader, default value is "hadoop"
4. support session variable `force_jni_scanner` for hudi
5. add more cases for hudi p2

### Release note
[opt](hudi) upgrade hudi to 0.15 and support hadoop jni reader
@doris-robot
Copy link

doris-robot commented Dec 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
[fix](hudi) support reading hudi read optimized table with orc format (…
8cad0e4 
…apache#44995)

### What problem does this PR solve?
Problem Summary:
When reading the hudi ro table, it will be pushed back from jni to the
native reader. However, this process will default the file format to
parquet, and does not consider the situation that the hudi table is
stored in orc format.

1. support reading hudi read optimized table with orc format
2. fix explain results of hudiScanNode when force_jni_reader=true
3. add cases about  timestamp with different timezones
@suxiaogang223 suxiaogang223 changed the title branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267) branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) Dec 5, 2024
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Dec 5, 2024

run buildall

1 similar comment
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Dec 5, 2024

run buildall

@morningman morningman merged commit d6482e1 into apache:branch-3.0 Dec 5, 2024
11 of 12 checks passed
@suxiaogang223 suxiaogang223 deleted the fix_hudi branch December 12, 2024 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs

@morningman morningman Awaiting requested review from morningman

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@suxiaogang223 @doris-robot @morningman