HDDS-12197. Update documentation for all ozone debug tools

hadoop-hdds/docs/content/tools/AuditParser.md → hadoop-hdds/docs/content/tools/debug/AuditParser.md

@@ -1,7 +1,9 @@
 ---
 title: "Audit Parser"
 date: 2018-12-17
-summary: Audit Parser tool can be used for querying the ozone audit logs.
+summary: Tool for querying the ozone audit logs.
+menu: debug
+weight: 6
 ---
 <!---
   Licensed to the Apache Software Foundation (ASF) under one or more

hadoop-hdds/docs/content/tools/ContainerLogParser.md → hadoop-hdds/docs/content/tools/debug/ContainerLogParser.md

@@ -2,7 +2,9 @@
 title: "Container Replica Debugger Tool"
 date: 2025-05-19
 jira: HDDS-12579
-summary: The tool processes container log files from Ozone datanodes to track state transitions. It provides CLI commands for querying containers based on different attributes.
+summary: Processing and querying container log files from Ozone datanodes to track state transitions. 
+menu: debug
+weight: 7
 ---
 <!--
   Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,7 +17,8 @@ summary: The tool processes container log files from Ozone datanodes to track st
   See the License for the specific language governing permissions and
   limitations under the License. See accompanying LICENSE file.
 -->
-# Container Replica Debugger Tool
+
+The tool processes container log files from Ozone datanodes to track state transitions. It provides CLI commands for querying containers based on different attributes.
 
 ## Background
 Containers are the most important part of ozone. Most of the ozone operations happen on them. Containers in ozone go through different states in their lifecycle.

hadoop-hdds/docs/content/tools/debug/DebugDatanode.md

@@ -0,0 +1,39 @@
+---
+title: "Debug Datanode"
+date: 2025-07-28
+summary: Debug commands related to datanode.
+menu: debug
+weight: 3
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+Debug commands related to datanode. Currently, only container replica related commands are available.
+Following is the usage and the subcommands available under the `ozone debug datanode container` command:
+
+```bash
+Usage: ozone debug datanode container [-hV] [--verbose] [COMMAND]
+Container replica specific operations to be executed on datanodes only
+  -h, --help      Show this help message and exit.
+  -V, --version   Print version information and exit.
+      --verbose   More verbose output. Show the stack trace of the errors.
+Commands:
+  list     Show container info of all container replicas on datanode
+  info     Show container info of a container replica on datanode
+  export   Export one container to a tarball
+  inspect  Check the metadata of all container replicas on this datanode.
+```

hadoop-hdds/docs/content/tools/debug/DebugOm.md

@@ -0,0 +1,57 @@
+---
+title: "Debug OM"
+date: 2025-07-28
+summary: Debug commands related to OM.
+menu: debug
+weight: 2
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+Debug commands related to OM.
+It has the following subcommands:
+## generate-compaction-dag 
+Creates a DAG image of the current compaction log of an om.db instance. It is downloaded to the specified location.
+```bash
+Usage: ozone debug om generate-compaction-dag [-hV] [--verbose] --db=<dbPath>
+       -o=<imageLocation>
+Create an image of the current compaction log DAG. This command is an offline
+command. i.e., it can run on any instance of om.db and does not require OM to
+be up.
+      --db=<dbPath>   Path to OM RocksDB
+  -h, --help          Show this help message and exit.
+  -o, --output-file=<imageLocation>
+                      Path to location at which image will be downloaded.
+                        Should include the image file name with ".png"
+                        extension.
+  -V, --version       Print version information and exit.
+      --verbose       More verbose output. Show the stack trace of the errors.
+```
+
+## prefix
+Parses the contents of a prefix.
+```bash
+Usage: ozone debug om prefix [--verbose] --bucket=<bucket> --db=<dbPath>
+                             --path=<filePath> --volume=<volume>
+Parse prefix contents
+      --bucket=<bucket>   bucket name
+      --db=<dbPath>       Path to OM RocksDB
+      --path=<filePath>   prefixFile Path
+      --verbose           More verbose output. Show the stack trace of the
+                            errors.
+      --volume=<volume>   volume name
+```

hadoop-hdds/docs/content/tools/debug/DebugReplicas.md

@@ -0,0 +1,119 @@
+---
+title: "Debug Replicas"
+date: 2025-07-28
+summary: Debug commands for replica-related issues.
+menu: debug
+weight: 4
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+Debug commands for retrieving information and performing various checks on the key replicas in datanodes.
+
+```bash
+Usage: ozone debug replicas [--verbose] [COMMAND]
+Debug commands for replica-related issues, retrieving replica information from
+the OM and performing checks over the network against a running cluster.
+      --verbose   More verbose output. Show the stack trace of the errors.
+Commands:
+  chunk-info  Returns chunk location information about an existing key
+  verify      Run checks to verify data across replicas. By default prints only
+                the keys with failed checks.
+```
+
+## chunk-info
+For a given URI of a key, the command returns all the chunks' location information.
+
+```bash
+Usage: ozone debug replicas chunk-info [-hV] [--verbose] <value>
+Returns chunk location information about an existing key
+      <value>     URI of the key (format: volume/bucket/key).
+                  Ozone URI could either be a full URI or short URI.
+                  Full URI should start with o3://, in case of non-HA
+                  clusters it should be followed by the host name and
+                  optionally the port number. In case of HA clusters
+                  the service id should be used. Service id provides a
+                  logical name for multiple hosts and it is defined
+                  in the property ozone.om.service.ids.
+                  Example of a full URI with host name and port number
+                  for a key:
+                  o3://omhostname:9862/vol1/bucket1/key1
+                  With a service id for a volume:
+                  o3://omserviceid/vol1/
+                  Short URI should start from the volume.
+                  Example of a short URI for a bucket:
+                  vol1/bucket1
+                  Any unspecified information will be identified from
+                  the config files.
+
+  -h, --help      Show this help message and exit.
+  -V, --version   Print version information and exit.
+      --verbose   More verbose output. Show the stack trace of the errors.
+```
+
+## verify
+Verify data across replicas. There are multiple checks available, which can be selected using the command line options:
+
+```bash 
+Usage: ozone debug replicas verify [-hV] [--all-results] [--verbose]
+                                   [--container-cache-size=<containerCacheSize>]
+                                    [-id=<scmServiceId>] [--scm=<scm>]
+                                   ([--checksums] [--block-existence]
+                                   [--container-state]) <uri>
+Run checks to verify data across replicas. By default prints only the keys with
+failed checks.
+      <uri>               Ozone URI could either be a full URI or short URI.
+                          Full URI should start with o3://, in case of non-HA
+                          clusters it should be followed by the host name and
+                          optionally the port number. In case of HA clusters
+                          the service id should be used. Service id provides a
+                          logical name for multiple hosts and it is defined
+                          in the property ozone.om.service.ids.
+                          Example of a full URI with host name and port number
+                          for a key:
+                          o3://omhostname:9862/vol1/bucket1/key1
+                          With a service id for a volume:
+                          o3://omserviceid/vol1/
+                          Short URI should start from the volume.
+                          Example of a short URI for a bucket:
+                          vol1/bucket1
+                          Any unspecified information will be identified from
+                          the config files.
+
+      --all-results       Print results for all passing and failing keys
+      --block-existence   Check for block existence on datanodes.
+      --checksums         Do client side data checksum validation of all
+                            replicas.
+      --container-cache-size=<containerCacheSize>
+                          Size (in number of containers) of the in-memory cache
+                            for container state verification
+                            '--container-state'. Default is 1 million
+                            containers (which takes around 43MB). Value must be
+                            greater than zero, otherwise the default of 1
+                            million is considered. Note: This option is ignored
+                            if '--container-state' option is not used.
+      --container-state   Check the container and replica states. Containers in
+                            [DELETING, DELETED] states, or it's replicas in
+                            [DELETED, UNHEALTHY, INVALID] states fail the check.
+  -h, --help              Show this help message and exit.
+      -id, --service-id=<scmServiceId>
+                          ServiceId of SCM HA Cluster
+      --scm=<scm>         The destination scm (host:port)
+  -V, --version           Print version information and exit.
+      --verbose           More verbose output. Show the stack trace of the
+                            errors.
+```

hadoop-hdds/docs/content/tools/Debug.md → hadoop-hdds/docs/content/tools/debug/Ldb.md

@@ -1,7 +1,9 @@
 ---
-title: "Ozone Debug"
+title: "LDB Tool"
 date: 2024-10-14
-summary: Ozone Debug command can be used for all the debugging related tasks.
+summary: Parsing rocksDB and debugging related issues.
+menu: debug
+weight: 1
 ---
 <!---
   Licensed to the Apache Software Foundation (ASF) under one or more
@@ -20,43 +22,6 @@ summary: Ozone Debug command can be used for all the debugging related tasks.
   limitations under the License.
 -->
 
-Ozone Debug command (`ozone debug`) is a collection of developer tools intended to help in debugging and get more information of various components of ozone.
-
-```bash
-Usage: ozone debug [-hV] [--verbose] [-conf=<configurationPath>]
-                   [-D=<String=String>]... [COMMAND]
-Developer tools for Ozone Debug operations
-      -conf=<configurationPath>
-                  path to the ozone configuration file
-  -D, --set=<String=String>
-                  Map of (configuration_key,configuration_value) for any 
-                    configuration overrides
-  -h, --help      Show this help message and exit.
-  -V, --version   Print version information and exit.
-      --verbose   More verbose output. Show the stack trace of the errors.
-Commands:
-  chunkinfo                  returns chunk location information about an
-                               existing key
-  print-log-dag, pld         Create an image of the current compaction log DAG
-                               in OM.
-  find-missing-padding, fmp  List all keys with any missing padding, optionally
-                               limited to a volume/bucket/key URI.
-  recover                    recover the lease of a specified file. Make sure
-                               to specify file system scheme if ofs:// is not
-                               the default.
-  prefix                     Parse prefix contents
-  ldb                        Parse rocksdb file content
-  read-replicas              Reads every replica for all the blocks associated
-                               with a given key.
-  container                  Container replica specific operations to be
-                               executed on datanodes only
-  ratislogparser             Shell of printing Ratis Log in understandable text
-```
-For more detailed usage see the output of `--help` for each of the subcommands.
-
-
-## ozone debug ldb
-
 Ozone heavily uses RocksDB for storing metadata. This tool helps parse the contents of RocksDB belonging to Ozone Roles.
 Supported DB's : Ozone Manager (om.db) , StorageContainerManager (scm.db),  Datanode/Container (container.db)
 Below is the usage:
@@ -69,6 +34,7 @@ Commands:
   scan                      Parse specified metadataTable
   list_column_families, ls  list all column families in db.
   value-schema              Schema of value in metadataTable
+  checkpoint                Create checkpoint for specified db
 ```
 
 ### list_column_families command
@@ -470,4 +436,12 @@ $ ozone debug ldb --db=/data/metadata/om.db value-schema --cf=keyTable
     "objectID" : "long"
   }
 }
-```
+```
+
+### checkpoint command
+
+`checkpoint` command takes a checkpoint of a rocksdb, at a provided path.
+
+```bash
+$ ozone debug ldb --db=/data/metadata/om.db checkpoint --output=/tmp/om-checkpoint
+```

hadoop-hdds/docs/content/tools/debug/RatisLogParser.md

@@ -0,0 +1,39 @@
+---
+title: "Ratis Log Parser"
+date: 2025-07-28
+summary: Parse ratis logs to be in human understandable form.
+menu: debug
+weight: 5
+---
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+The Ratis log parser tool takes a segment file as input, and give the output in a human-readable format.
+It has the following subcommands, which can be used to parse the Ratis logs of different components:
+
+```bash
+Usage: ozone debug ratislogparser [-hV] [--verbose] [COMMAND]
+Shell of printing Ratis Log in understandable text
+  -h, --help      Show this help message and exit.
+  -V, --version   Print version information and exit.
+      --verbose   More verbose output. Show the stack trace of the errors.
+Commands:
+  datanode  dump datanode segment file
+  generic   dump generic ratis segment file
+  om        dump om ratis segment file
+  scm       dump scm ratis segment file
+```