From b54200b60625943c9c6d7db6424ccacc8cf81021 Mon Sep 17 00:00:00 2001 From: congyi <15605187270@163.com> Date: Thu, 6 Jul 2023 14:25:47 +0800 Subject: [PATCH] add troubleshoot doc for hdfs ha --- core/src/services/hdfs/docs.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/core/src/services/hdfs/docs.md b/core/src/services/hdfs/docs.md index 873c29715261..7c30fd5c4f6b 100644 --- a/core/src/services/hdfs/docs.md +++ b/core/src/services/hdfs/docs.md @@ -67,6 +67,21 @@ To set `CLASSPATH`: export CLASSPATH=$(find $HADOOP_HOME -iname "*.jar" | xargs echo | tr ' ' ':'):${CLASSPATH} ``` +- If HDFS has High Availability (HA) enabled with multiple available NameNodes, some configuration is required: +1. Obtain the entire HDFS config folder (usually located at HADOOP_HOME/etc/hadoop). +2. Set the environment variable HADOOP_CONF_DIR to the path of this folder. +```shell +export HADOOP_CONF_DIR= +``` +3. Append the HADOOP_CONF_DIR to the `CLASSPATH` +```shell +export CLASSPATH=$CLASSPATH:$HADOOP_CONF_DIR +``` +4. Use the `cluster_name` specified in the `core-site.xml` file (located in the HADOOP_CONF_DIR folder) to replace namenode:port. + +```rust +builder.name_node("hdfs://cluster_name"); +``` ## Example ### Via Builder