[Bug] when using s3a, java.lang.NoClassDefFoundError（broker） #26938
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
1.2.7
What's Wrong?
when using s3a to export data , the following error occurred
[DEBUG] 2023-11-01 11:55:44,346 method:org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3226)
looking for configuration option fs.s3a.impl
Exception in thread "pool-2-thread-4" java.lang.NoClassDefFoundError: shade/doris/hadoop/com/google/common/util/concurrent/ListeningExecutorService
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2365)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2330)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2426)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3227)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3259)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:472)
at org.apache.doris.broker.hdfs.FileSystemManager.getS3AFileSystem(FileSystemManager.java:474)
at org.apache.doris.broker.hdfs.FileSystemManager.getFileSystem(FileSystemManager.java:216)
at org.apache.doris.broker.hdfs.FileSystemManager.openWriter(FileSystemManager.java:1253)
at org.apache.doris.broker.hdfs.HDFSBrokerServiceImpl.openWriter(HDFSBrokerServiceImpl.java:208)
at org.apache.doris.thrift.TPaloBrokerService$Processor$openWriter.getResult(TPaloBrokerService.java:1015)
at org.apache.doris.thrift.TPaloBrokerService$Processor$openWriter.getResult(TPaloBrokerService.java:995)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: shade.doris.hadoop.com.google.common.util.concurrent.ListeningExecutorService
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
What You Expected?
fix the bug
How to Reproduce?
No response
Anything Else?
I found a possible cause。hadoop-shade-2 relocation and S3AFileSystem refer is not the same。
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct