From b69df00e4d08f736962b57eb649f33ddc999698f Mon Sep 17 00:00:00 2001
From: Wei-Chiu Chuang <weichiu@apache.org>
Date: Thu, 3 Jul 2025 11:37:12 -0700
Subject: [PATCH 1/4] docs: Add production deployment guide

This commit introduces a new documentation page outlining the requirements and best practices for deploying Apache Ozone in a production environment.

The guide covers:
- System, storage, network, and security requirements.
- Recommended configurations for the Linux kernel, local file system, and Ozone itself.

This addresses HDDS-13378.

Change-Id: I907eff74c755b9232400900bce6b8878d41402fe
---
 .../content/start/ProductionDeployment.md     | 66 +++++++++++++++++++
 .../content/start/ProductionDeployment.zh.md  | 66 +++++++++++++++++++
 2 files changed, 132 insertions(+)
 create mode 100644 hadoop-hdds/docs/content/start/ProductionDeployment.md
 create mode 100644 hadoop-hdds/docs/content/start/ProductionDeployment.zh.md

diff --git a/hadoop-hdds/docs/content/start/ProductionDeployment.md b/hadoop-hdds/docs/content/start/ProductionDeployment.md
new file mode 100644
index 000000000000..94746ed0242c
--- /dev/null
+++ b/hadoop-hdds/docs/content/start/ProductionDeployment.md
@@ -0,0 +1,66 @@
+---
+title: Production Deployment
+weight: 6
+---
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+This document provides guidance on the requirements and best practices for a production deployment of Apache Ozone.
+
+## Requirements
+
+### System Requirements
+
+*   **Operating System**: Linux (recommended distributions: Red Hat 8/Rocky 8+, Ubuntu, SUSE; supported architectures: x86/ARM).
+*   **Java Development Kit (JDK)**: Version 8 or higher.
+*   **Time Synchronization**: A time synchronization service such as Chrony or ntpd must be enabled to prevent time drift.
+
+### Storage Requirements
+
+*   **Metadata Storage**: Use SAS SSD or NVMe SSD for metadata (RocksDB and Ratis) to ensure optimal performance.
+*   **DataNode Storage**: Hard disks are acceptable for DataNode data storage.
+*   **Storage Type**: Use direct-attached storage. Do not use Network Attached Storage (NAS) or Storage Area Network (SAN).
+
+### Network Requirements
+
+*   **Network Bandwidth**: A minimum of 25Gbps network card bandwidth is recommended.
+*   **Network Topology**: A leaf-spine network topology with an oversubscription ratio below 3:1 is recommended for predictable performance.
+
+### Security Requirements (Optional but Recommended)
+
+*   **Kerberos**: A Kerberos environment, including a Key Distribution Center (KDC), is recommended for enhanced security.
+
+## Recommended Configurations
+
+### Linux Kernel
+
+*   **CPU Governor**: Set the CPU scaling driver to `performance` mode to maximize performance.
+*   **Transparent Hugepage**: Disable Transparent Hugepage to avoid performance issues.
+*   **SELinux**: Disable SELinux.
+*   **Swappiness**: Set `vm.swappiness=1` to minimize swapping.
+
+### Local File System
+
+*   **LVM**: Disable Logical Volume Manager (LVM) for data drives.
+*   **File System**: Use `ext4` or `xfs` file systems.
+*   **Mount Options**: Mount drives with the `noatime` option to reduce unnecessary disk writes. For SSDs, also add the `discard` option.
+
+### Ozone Configuration
+
+*   **Monitoring**: Install Prometheus and Grafana for monitoring the Ozone cluster.
+*   **Pipeline Limits**: Increase the number of allowed write pipelines to better suit your workload by adjusting `ozone.scm.datanode.pipeline.limit` and `ozone.scm.ec.pipeline.minimum`.
+*   **Heap Sizes**: Configure sufficient heap sizes for Ozone Manager (OM), Storage Container Manager (SCM), Recon, DataNode, S3 Gateway (S3G), and HttpFs services to ensure stability.
diff --git a/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md b/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md
new file mode 100644
index 000000000000..39f27abe29f4
--- /dev/null
+++ b/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md
@@ -0,0 +1,66 @@
+---
+title: 生产环境部署
+weight: 6
+---
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+本文档旨在为 Apache Ozone 的生产环境部署提供需求和最佳实践的指导。
+
+## 需求
+
+### 系统需求
+
+*   **操作系统**: Linux（推荐发行版：Red Hat 8/Rocky 8+、Ubuntu、SUSE；支持架构：x86/ARM）。
+*   **Java 开发工具包 (JDK)**: 版本 8 或更高。
+*   **时间同步**: 必须启用时间同步服务（如 Chrony 或 ntpd）以防止时间漂移。
+
+### 存储需求
+
+*   **元数据存储**: 为确保最佳性能，请使用 SAS SSD 或 NVMe SSD 存储元数据（RocksDB 和 Ratis）。
+*   **DataNode 存储**: DataNode 数据存储可使用硬盘。
+*   **存储类型**: 请使用直接附加存储。不要使用网络附加存储 (NAS) 或存储区域网络 (SAN)。
+
+### 网络需求
+
+*   **网络带宽**: 建议网卡带宽至少为 25Gbps。
+*   **网络拓扑**: 为实现可预测的性能，建议采用超分比例低于 3:1 的叶脊网络拓扑。
+
+### 安全需求 (可选但推荐)
+
+*   **Kerberos**: 为增强安全性，建议使用包括密钥分发中心 (KDC) 在内的 Kerberos 环境。
+
+## 推荐配置
+
+### Linux 内核
+
+*   **CPU 调节器**: 将 CPU 调节驱动设置为 `performance` 模式以最大化性能。
+*   **透明大页**: 禁用透明大页以避免性能问题。
+*   **SELinux**: 禁用 SELinux。
+*   **Swappiness**: 设置 `vm.swappiness=1` 以最小化交换。
+
+### 本地文件系统
+
+*   **LVM**: 禁用数据驱动器的逻辑卷管理器 (LVM)。
+*   **文件系统**: 使用 `ext4` 或 `xfs` 文件系统。
+*   **挂载选项**: 使用 `noatime` 选项挂载驱动器以减少不必要的磁盘写入。对于 SSD，还需添加 `discard` 选项。
+
+### Ozone 配置
+
+*   **监控**: 安装 Prometheus 和 Grafana 以监控 Ozone 集群。
+*   **管道限制**: 通过调整 `ozone.scm.datanode.pipeline.limit` 和 `ozone.scm.ec.pipeline.minimum` 来增加允许的写入管道数量，以更好地适应您的工作负载。
+*   **堆大小**: 为 Ozone Manager (OM)、Storage Container Manager (SCM)、Recon、DataNode、S3 Gateway (S3G) 和 HttpFs 服务配置足够的堆大小，以确保稳定性。

From 1c419516c2abaf51c7d8812207b3e1d09d852d59 Mon Sep 17 00:00:00 2001
From: Wei-Chiu Chuang <weichiu@apache.org>
Date: Thu, 3 Jul 2025 11:43:52 -0700
Subject: [PATCH 2/4] Update Hugo header.

Change-Id: I1e883c996a3aeb4fa34812f35ba80974b6b939c3
---
 hadoop-hdds/docs/content/start/ProductionDeployment.md    | 3 +++
 hadoop-hdds/docs/content/start/ProductionDeployment.zh.md | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/hadoop-hdds/docs/content/start/ProductionDeployment.md b/hadoop-hdds/docs/content/start/ProductionDeployment.md
index 94746ed0242c..d1f8a09496c3 100644
--- a/hadoop-hdds/docs/content/start/ProductionDeployment.md
+++ b/hadoop-hdds/docs/content/start/ProductionDeployment.md
@@ -1,6 +1,9 @@
 ---
 title: Production Deployment
 weight: 6
+menu:
+  main:
+    parent: Getting Started
 ---
 <!--
   Licensed to the Apache Software Foundation (ASF) under one or more
diff --git a/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md b/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md
index 39f27abe29f4..4620ccf31bc8 100644
--- a/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md
+++ b/hadoop-hdds/docs/content/start/ProductionDeployment.zh.md
@@ -1,6 +1,9 @@
 ---
 title: 生产环境部署
 weight: 6
+menu:
+  main:
+    parent: 快速入门
 ---
 <!--
   Licensed to the Apache Software Foundation (ASF) under one or more

From ae74cd7730b5f78d0b73b729dddafea69fff7e1e Mon Sep 17 00:00:00 2001
From: Wei-Chiu Chuang <weichiu@apache.org>
Date: Thu, 10 Jul 2025 10:41:30 -0700
Subject: [PATCH 3/4] docs: Update production deployment guide based on
 feedback

Change-Id: Iac5df570d0f0b963ca4be548c63f2650871e98d3
---
 .../docs/content/start/ProductionDeployment.md  | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/hadoop-hdds/docs/content/start/ProductionDeployment.md b/hadoop-hdds/docs/content/start/ProductionDeployment.md
index d1f8a09496c3..90e29ee60966 100644
--- a/hadoop-hdds/docs/content/start/ProductionDeployment.md
+++ b/hadoop-hdds/docs/content/start/ProductionDeployment.md
@@ -24,6 +24,17 @@ menu:
 
 This document provides guidance on the requirements and best practices for a production deployment of Apache Ozone.
 
+## Ozone Components
+
+A typical production Ozone cluster includes the following services:
+
+*   **Ozone Manager (OM)**: Manages the namespace and metadata of the Ozone cluster. A production cluster requires 3 OM instances for high availability.
+*   **Storage Container Manager (SCM)**: Manages the data nodes and pipelines. A production cluster requires 3 SCM instances for high availability.
+*   **DataNode**: Stores the actual data in containers. A production cluster requires at least 3 DataNodes.
+*   **Recon**: A web-based UI for monitoring and managing the Ozone cluster. A Recon server is strongly recommended, though not required.
+*   **S3 Gateway (S3G)**: An S3-compatible gateway for accessing Ozone. Multiple S3 Gateway instances are strongly recommended to load balance S3 traffic.
+*   **HttpFs**: An HDFS-compatible API for accessing Ozone. This is an optional component.
+
 ## Requirements
 
 ### System Requirements
@@ -34,8 +45,10 @@ This document provides guidance on the requirements and best practices for a pro
 
 ### Storage Requirements
 
-*   **Metadata Storage**: Use SAS SSD or NVMe SSD for metadata (RocksDB and Ratis) to ensure optimal performance.
-*   **DataNode Storage**: Hard disks are acceptable for DataNode data storage.
+*   **Ozone Manager (OM) and Storage Container Manager (SCM) Metadata Storage**: Use SAS SSD or NVMe SSD for metadata (RocksDB and Ratis) to ensure optimal performance.
+*   **DataNode Storage**:
+    *   **Ratis Log**: Use SAS SSD or NVMe SSD for the Ratis log directory for low latency writes.
+    *   **Container Data**: Hard disks are acceptable for container data storage.
 *   **Storage Type**: Use direct-attached storage. Do not use Network Attached Storage (NAS) or Storage Area Network (SAN).
 
 ### Network Requirements

From ed1285ef2ac33cb839c8c4bed9ffa1818f3eeb92 Mon Sep 17 00:00:00 2001
From: Wei-Chiu Chuang <weichiu@apache.org>
Date: Sat, 12 Jul 2025 10:51:13 -0700
Subject: [PATCH 4/4] docs: HDDS-13378. Add production deployment
 recommendations

Change-Id: Icd6c032dfb311ccf117b529e0c8625704d2167c4
---
 .../docs/content/start/ProductionDeployment.md        | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/hadoop-hdds/docs/content/start/ProductionDeployment.md b/hadoop-hdds/docs/content/start/ProductionDeployment.md
index 90e29ee60966..ed24a7b26710 100644
--- a/hadoop-hdds/docs/content/start/ProductionDeployment.md
+++ b/hadoop-hdds/docs/content/start/ProductionDeployment.md
@@ -39,16 +39,23 @@ A typical production Ozone cluster includes the following services:
 
 ### System Requirements
 
+*   **Hardware**: Bare metal machines are recommended for optimal performance. Virtual machines or containers are not recommended for production deployments.
 *   **Operating System**: Linux (recommended distributions: Red Hat 8/Rocky 8+, Ubuntu, SUSE; supported architectures: x86/ARM).
 *   **Java Development Kit (JDK)**: Version 8 or higher.
 *   **Time Synchronization**: A time synchronization service such as Chrony or ntpd must be enabled to prevent time drift.
 
+### Memory Requirements
+
+*   **Ozone Manager (OM), Storage Container Manager (SCM), and Recon**: Recommended heap size in large production clusters is 64GB.
+*   **DataNode, S3 Gateway, and HttpFs**: Recommended heap size is 31GB.
+
 ### Storage Requirements
 
-*   **Ozone Manager (OM) and Storage Container Manager (SCM) Metadata Storage**: Use SAS SSD or NVMe SSD for metadata (RocksDB and Ratis) to ensure optimal performance.
+*   **Ozone Manager (OM), Storage Container Manager (SCM), and Recon Metadata Storage**: Use SAS SSD or NVMe SSD for metadata (RocksDB and Ratis) to ensure optimal performance. It is recommended to use RAID 1 (disk mirroring) for the metadata disks to protect against disk failures.
 *   **DataNode Storage**:
     *   **Ratis Log**: Use SAS SSD or NVMe SSD for the Ratis log directory for low latency writes.
     *   **Container Data**: Hard disks are acceptable for container data storage.
+    *   **Disk Configuration**: It is recommended to use a JBOD (Just a Bunch Of Disks) configuration instead of RAID. Ozone is a replicated distributed storage system and handles data redundancy. Using RAID can decrease performance without providing additional data protection benefits.
 *   **Storage Type**: Use direct-attached storage. Do not use Network Attached Storage (NAS) or Storage Area Network (SAN).
 
 ### Network Requirements
@@ -77,6 +84,6 @@ A typical production Ozone cluster includes the following services:
 
 ### Ozone Configuration
 
-*   **Monitoring**: Install Prometheus and Grafana for monitoring the Ozone cluster.
+*   **Monitoring**: Install Prometheus and Grafana for monitoring the Ozone cluster. For audit logs, consider using a log ingestion framework such as the ELK Stack (Elasticsearch, Logstash, and Kibana) with FileBeat, or other similar frameworks. Alternatively, you can use Apache Ranger to manage audit logs.
 *   **Pipeline Limits**: Increase the number of allowed write pipelines to better suit your workload by adjusting `ozone.scm.datanode.pipeline.limit` and `ozone.scm.ec.pipeline.minimum`.
 *   **Heap Sizes**: Configure sufficient heap sizes for Ozone Manager (OM), Storage Container Manager (SCM), Recon, DataNode, S3 Gateway (S3G), and HttpFs services to ensure stability.