Skip to content

fix the bug that 'username' in broker load is invalid #3237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
imay merged 3 commits into from
Apr 6, 2020
Merged

fix the bug that 'username' in broker load is invalid #3237

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b2ae956
make 'username' valid in broker load
Mar 31, 2020
f448aac
delete 'hadoop.job.ugi'
Apr 1, 2020
40d6218
delete redundant code
Apr 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions ...kers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.PrivilegedExceptionAction;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
Expand All @@ -65,7 +66,6 @@ public class FileSystemManager {
private static final String HDFS_SCHEME = "hdfs";
private static final String S3A_SCHEME = "s3a";

private static final String HDFS_UGI_CONF = "hadoop.job.ugi";
private static final String USER_NAME_KEY = "username";
private static final String PASSWORD_KEY = "password";
private static final String AUTHENTICATION_SIMPLE = "simple";
Expand Down Expand Up @@ -246,9 +246,7 @@ public BrokerFileSystem getDistributedFileSystem(String path, Map<String, String
// TODO get this param from properties
// conf.set("dfs.replication", "2");
String tmpFilePath = null;
if (authentication.equals(AUTHENTICATION_SIMPLE)) {
conf.set(HDFS_UGI_CONF, hdfsUgi);
} else if (authentication.equals(AUTHENTICATION_KERBEROS)){
if (authentication.equals(AUTHENTICATION_KERBEROS)){
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
AUTHENTICATION_KERBEROS);

Expand Down Expand Up @@ -287,9 +285,6 @@ public BrokerFileSystem getDistributedFileSystem(String path, Map<String, String
e.getMessage());
}
}
} else {
throw new BrokerException(TBrokerOperationStatusCode.INVALID_ARGUMENT,
"invalid authentication.");
Copy link
Contributor

@HangyuanLiu HangyuanLiu Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to determine whether the authentication method exists

Copy link
Contributor Author

@frwrdt frwrdt Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The authentication has been checked before, so this is redundant here.

}
if (!Strings.isNullOrEmpty(dfsNameServices)) {
// ha hdfs arguments
Expand Down Expand Up @@ -338,7 +333,20 @@ public BrokerFileSystem getDistributedFileSystem(String path, Map<String, String
}

conf.set(FS_HDFS_IMPL_DISABLE_CACHE, "true");
FileSystem dfsFileSystem = FileSystem.get(pathUri.getUri(), conf);
FileSystem dfsFileSystem = null;
Copy link
Contributor

@HangyuanLiu HangyuanLiu Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this method compatible with versions of hdfs?

Copy link
Contributor Author

@frwrdt frwrdt Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version 2.7.3 of package hadoop-common including this method is the same as package hadoop-hdfs.

if (authentication.equals(AUTHENTICATION_SIMPLE) &&
properties.containsKey(USER_NAME_KEY)) {
// Use the specified 'username' as the login name
UserGroupInformation ugi = UserGroupInformation.createRemoteUser(username);
dfsFileSystem = ugi.doAs(new PrivilegedExceptionAction<FileSystem>() {
@Override
public FileSystem run() throws Exception {
return FileSystem.get(pathUri.getUri(), conf);
}
});
} else {
dfsFileSystem = FileSystem.get(pathUri.getUri(), conf);
}
fileSystem.setFileSystem(dfsFileSystem);
}
return fileSystem;
Expand Down