[refactor](libhdfs) introduce hadoop libhdfs #18204

morningman · 2023-03-29T05:52:43Z

Proposed changes

Issue Number: close #xxx

Problem summary

Introduce hadoop libhdfs
For Linux-X86 platform, use the hadoop libhdfs
For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform

Co-authored-by: adonis0147 adonis0147@gmail.com

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-03-29T05:53:40Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
                                      ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"


In bin/start_be.sh line 262:
        export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
                               ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                  ^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                   ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                      ^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}


In bin/start_be.sh line 263:
        export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
                               ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                  ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}


In bin/start_be.sh line 264:
        export LIBHDFS_OPTS="${JAVA_OPTS}"
                             ^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.


In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
     ^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.


In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
     ^-- SC2028 (info): echo may not expand escape sequences. Use printf.


In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
     ^-- SC2028 (info): echo may not expand escape sequences. Use printf.

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
  https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-29T14:53:00Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/start_be.sh line 101:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
                                      ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"


In bin/start_be.sh line 257:
if [[ -z $JAVA_OPTS ]]; then
         ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ -z ${JAVA_OPTS} ]]; then


In bin/start_be.sh line 260:
    JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
                                   ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                   ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                             ^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    JAVA_OPTS="-Xmx1024m -DlogPath=${DORIS_HOME}/log/jni.log -Xloggc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-29T14:55:01Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/start_be.sh line 101:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
                                      ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"


In bin/start_be.sh line 257:
if [[ -z $JAVA_OPTS ]]; then
         ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ -z ${JAVA_OPTS} ]]; then


In bin/start_be.sh line 260:
    JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
                                   ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                   ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                             ^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    JAVA_OPTS="-Xmx1024m -DlogPath=${DORIS_HOME}/log/jni.log -Xloggc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

github-actions · 2023-03-29T14:56:28Z

be/src/util/jni-util.cpp

+            options = (JavaVMOption*)calloc(no_args, sizeof(JavaVMOption));
+            options[0].optionString = const_cast<char*>(classpath.c_str());
+            java_opts = getenv("JAVA_OPTS");
+            if (java_opts != NULL) {


warning: use nullptr [modernize-use-nullptr]

Suggested change

if (java_opts != NULL) {

if (java_opts != nullptr) {

morningman · 2023-03-29T16:10:36Z

run buildall

morningman · 2023-03-29T23:52:12Z

run buildall

adonis0147 · 2023-03-30T02:44:03Z

be/src/util/jni-util.cpp

+
+        if (java_opts != nullptr) {
+            free(java_opts);
+        }
+        free(options);


It's better to use RAII.

morningman · 2023-03-30T05:43:14Z

run buildall

github-actions · 2023-03-30T05:43:39Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/start_be.sh line 101:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
                                      ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"


In bin/start_be.sh line 259:
if [[ -z $JAVA_OPTS ]]; then
         ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ -z ${JAVA_OPTS} ]]; then


In bin/start_be.sh line 262:
    JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
                                   ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                   ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                             ^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    JAVA_OPTS="-Xmx1024m -DlogPath=${DORIS_HOME}/log/jni.log -Xloggc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

morningman · 2023-03-30T05:51:14Z

run buildall

morningman · 2023-03-30T13:34:38Z

run buildall

github-actions · 2023-03-30T13:37:36Z

clang-tidy review says "All clean, LGTM! 👍"

adonis0147

LGTM

github-actions · 2023-03-30T13:54:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-03-30T13:54:40Z

PR approved by anyone and no changes requested.

adonis0147

LGTM

morningman · 2023-03-30T15:09:32Z

run buildall

github-actions · 2023-03-30T15:10:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-03-30T15:14:18Z

clang-tidy review says "All clean, LGTM! 👍"

1. Introduce hadoop libhdfs 2. For Linux-X86 platform, use the hadoop libhdfs 3. For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform Co-authored-by: adonis0147 <adonis0147@gmail.com>

github-actions bot added the area/vectorization label Mar 29, 2023

morningman mentioned this pull request Mar 29, 2023

[refactor](libhdfs) introduce hadoop libhdfs #17482

Closed

5 tasks

morningman force-pushed the hadoop_hdfs branch from adf343e to 86c0311 Compare March 29, 2023 14:53

github-actions bot reviewed Mar 29, 2023

View reviewed changes

morningman changed the title ~~[Draft](libhdfs) introduce hadoop libhdfs~~ [refactor](libhdfs) introduce hadoop libhdfs Mar 30, 2023

adonis0147 reviewed Mar 30, 2023

View reviewed changes

morningman added 5 commits March 30, 2023 12:29

1. intro libhdfs

a0f4799

fix meta tool

216b1f9

fix jvm

baac148

fix libhdfs3

3c75e4b

fix review

1c51072

morningman force-pushed the hadoop_hdfs branch from 623b6f9 to 1c51072 Compare March 30, 2023 05:42

fix shell

8adec90

fix jni

9d1ffb5

adonis0147 previously approved these changes Mar 30, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

github-actions bot added the reviewed label Mar 30, 2023

1

0477c3b

morningman dismissed adonis0147’s stale review via 0477c3b March 30, 2023 15:08

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

adonis0147 approved these changes Mar 30, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

morningman merged commit 7e61a85 into apache:master Mar 31, 2023

morningman mentioned this pull request Apr 9, 2023

[refactor](libhdfs) introduce hadoop libhdfs #18495

Closed

5 tasks

[refactor](libhdfs) introduce hadoop libhdfs #18204

[refactor](libhdfs) introduce hadoop libhdfs #18204

Uh oh!

Conversation

morningman commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actions bot commented Mar 29, 2023

sh-checker report

Uh oh!

github-actions bot commented Mar 29, 2023

sh-checker report

Uh oh!

github-actions bot commented Mar 29, 2023

sh-checker report

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

morningman commented Mar 29, 2023

Uh oh!

morningman commented Mar 29, 2023

Uh oh!

adonis0147 Mar 30, 2023

Choose a reason for hiding this comment

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

github-actions bot commented Mar 30, 2023

sh-checker report

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

github-actions bot commented Mar 30, 2023

Uh oh!

adonis0147 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 30, 2023

Uh oh!

github-actions bot commented Mar 30, 2023

Uh oh!

adonis0147 left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

github-actions bot commented Mar 30, 2023

Uh oh!

github-actions bot commented Mar 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

morningman commented Mar 29, 2023 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`