Skip to content

branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 4 commits into from
Dec 6, 2024
Merged

branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45041

morningman merged 4 commits into from
Dec 6, 2024

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Dec 5, 2024 •
edited
Loading

  1. upgrade hudi to 0.15.0.
  2. impl new hudi jni reader based on hudi-hadoop-mr
  3. add session variable hudi_jni_scanner to choose which hudi jni reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old HudiJniReader, default value is "hadoop"
  4. support session variable force_jni_scanner for hudi
  5. add more cases for hudi p2

specially for 2.1
not compile HadoopHudiJniReader and set session variable hudi_jni_scanner default as "spark"

cherry-pick pr: #44267 #44995

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@suxiaogang223
[fix](hudi) upgrade hudi to 0.15.0 (apache#44267)
3343260 
1. upgrade hudi to 0.15.0.
2. impl new hudi jni reader based on hudi-hadoop-mr
3. add session variable `hudi_jni_scanner` to choose which hudi jni
reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old
HudiJniReader, default value is "hadoop"
4. support session variable `force_jni_scanner` for hudi
5. add more cases for hudi p2

[opt](hudi) upgrade hudi to 0.15 and support hadoop jni reader
@doris-robot
Copy link

doris-robot commented Dec 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Dec 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 5, 2024

TeamCity be ut coverage result:
Function Coverage: 36.73% (9616/26180)
Line Coverage: 28.12% (79083/281198)
Region Coverage: 26.78% (40585/151568)
Branch Coverage: 23.60% (20651/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3343260a6cdbea78145a243239d113dccf41110f_3343260a6cdbea78145a243239d113dccf41110f/report/index.html

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Dec 5, 2024

run buildall

@suxiaogang223 suxiaogang223 changed the title branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 #44267 branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267) Dec 5, 2024
@suxiaogang223
[fix](hudi) support reading hudi read optimized table with orc format (…
6ee389b 
…apache#44995)

### What problem does this PR solve?
Problem Summary:
When reading the hudi ro table, it will be pushed back from jni to the
native reader. However, this process will default the file format to
parquet, and does not consider the situation that the hudi table is
stored in orc format.

1. support reading hudi read optimized table with orc format
2. fix explain results of hudiScanNode when force_jni_reader=true
3. add cases about  timestamp with different timezones
@suxiaogang223 suxiaogang223 changed the title branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267) branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) Dec 5, 2024
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Dec 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 5, 2024

TeamCity be ut coverage result:
Function Coverage: 36.73% (9616/26180)
Line Coverage: 28.12% (79078/281205)
Region Coverage: 26.77% (40575/151574)
Branch Coverage: 23.60% (20653/87500)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6ee389b914a79d4dc5abdf7c33bcae9e9e4c7b0b_6ee389b914a79d4dc5abdf7c33bcae9e9e4c7b0b/report/index.html

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Dec 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 6, 2024

TeamCity be ut coverage result:
Function Coverage: 36.73% (9616/26181)
Line Coverage: 28.12% (79080/281219)
Region Coverage: 26.77% (40585/151582)
Branch Coverage: 23.60% (20654/87512)
Coverage Report: http://coverage.selectdb-in.cc/coverage/240eb7adccfd455ad521d5032732cdd21252f631_240eb7adccfd455ad521d5032732cdd21252f631/report/index.html

@morningman morningman merged commit 53291bd into apache:branch-2.1 Dec 6, 2024
11 of 12 checks passed
@suxiaogang223 suxiaogang223 deleted the upgrade_hudi_0.15_2.1 branch December 12, 2024 09:26
@suxiaogang223 suxiaogang223 mentioned this pull request Mar 4, 2025
Merged
yiguolei pushed a commit that referenced this pull request Mar 5, 2025
@suxiaogang223
branch-2.1: [fix](hudi) Set Spark Hudi JNI scanner as default (#48602) (
1b33f8c 
#48606)

### What problem does this PR solve?
Related PR: #45041 

Problem Summary:
If we set hudi_jni_scanner to an incorrect value, jni_connector will be
null, causing a core dump.
So we set Spark Hudi JNI scanner as default and the hadoop hudi jni
reader will be supported in future
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@suxiaogang223 @doris-robot @morningman