From 1d6ce20c76290ef378a2860d74afbbdeaf5c20c6 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Fri, 4 Jul 2025 03:55:11 -0700 Subject: [PATCH 1/3] Fixes HDDS-13383. Improve Logs in Ozone documentation. The "Logs in Ozone" page was not very helpful. This patch improves it by: - Documenting which roles produce logs and how to configure them. - Documenting which roles produce audit logs and include a sample. - Explaining how to enable debug logs for both services and CLI tools. Change-Id: I5ffecd895f618278b507688a1a1579d6f5603d30 --- hadoop-hdds/docs/content/tools/LogsInOzone.md | 81 ++++++++++++++++--- 1 file changed, 72 insertions(+), 9 deletions(-) diff --git a/hadoop-hdds/docs/content/tools/LogsInOzone.md b/hadoop-hdds/docs/content/tools/LogsInOzone.md index 5196eb6a244e..aa8e60363fb6 100644 --- a/hadoop-hdds/docs/content/tools/LogsInOzone.md +++ b/hadoop-hdds/docs/content/tools/LogsInOzone.md @@ -1,7 +1,7 @@ --- title: "Logs in Ozone" date: 2023-01-30 -summary: Logs in Ozone. +summary: An overview of logging in Apache Ozone. --- -# AuditLog +Apache Ozone produces different types of logs to help users monitor and troubleshoot the cluster. This document provides an overview of the available logs, their configuration, and how to use them for debugging. -AuditLogs configurations are set in "*-audit-log4j2.properties" files. We -can change the corresponding files to update the audit log policies for -each component. +## Service Logs -## Deletion of AuditLog +Each Ozone service (Ozone Manager, Storage Container Manager, Datanode, S3 Gateway, and Recon) generates its own log file. These logs contain detailed information about the service's operations, including errors and warnings. -The default log appender is Rolling appender, the following configurations -can be added for deletion of out-of-date AuditLogs. +By default, log files are stored in the `$OZONE_LOG_DIR` directory, which is usually set to the `logs` directory under the Ozone installation. The log file names are specific to each service, for example: + +* `ozone-om-....log` for Ozone Manager +* `ozone-scm-....log` for Storage Container Manager +* `ozone-datanode-....log` for Datanode + +The logging behavior for each service is controlled by its `log4j.properties` file, located in the service's `$OZONE_CONF_DIR` directory, usually `etc/hadoop`. You can modify this file to change the log level, appenders, and other logging parameters. + +## Audit Logs + +Audit logs record security-sensitive operations, providing a trail of actions performed on the cluster. The following services produce audit logs: + +* Ozone Manager +* Storage Container Manager +* Datanode +* S3 Gateway + +Audit log configurations are set in `*-audit-log4j2.properties` files. You can change the corresponding files to update the audit log policies for each component. + +### Sample Audit Log Entry + +Here is an example of an audit log entry from the Ozone Manager: + +``` +INFO | OMAudit | ? | user=hdfs | ip=127.0.0.1 | op=CREATE_VOLUME | params={volume=vol1, admin=hdfs, owner=hdfs} | result=SUCCESS +``` + +This entry shows that the user `hdfs` successfully created a volume named `vol1`. + +### Deletion of Audit Logs + +The default log appender is a rolling appender. The following configurations can be added for the deletion of out-of-date AuditLogs. ``` appender.rolling.strategy.type=DefaultRolloverStrategy @@ -42,4 +70,39 @@ appender.rolling.strategy.delete.ifLastModified.type=IfLastModified appender.rolling.strategy.delete.ifLastModified.age=30d ``` -For more details, please check [Log4j2 Delete on Rollover](https://logging.apache.org/log4j/2.x/manual/appenders.html#CustomDeleteOnRollover). \ No newline at end of file +For more details, please check [Log4j2 Delete on Rollover](https://logging.apache.org/log4j/2.x/manual/appenders.html#CustomDeleteOnRollover). + +## Debugging + +You can increase the log verbosity for debugging purposes for both services and CLI tools. + +### Enabling Debug Logs for Services + +To enable debug logging for a service, you need to modify its `log4j.properties` file. Change the log level for the desired logger from `INFO` to `DEBUG`. For example, to enable debug logging for the Ozone Manager, you would edit its `log4j.properties` and change the following line: + +``` +rootLogger.level = info +``` + +to: + +``` +rootLogger.level = debug +``` + +After saving the file, the service will start logging more detailed debug information. + +### Enabling Debug Logs for CLI Tools + +To enable debug logging for Ozone CLI tools (e.g., `ozone sh volume create`), you can set the `OZONE_ROOT_LOGGER` environment variable to `debug`: + +```bash +export OZONE_ROOT_LOGGER=debug,console +ozone sh volume create /vol1 +``` + +Alternatively, you can use the `--loglevel` option with the `ozone` command: + +```bash +ozone --loglevel debug sh volume create /vol1 +``` From a706e0c66b7ccd2f08e514aeab37eb0ed1e34b69 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Mon, 7 Jul 2025 21:37:20 -0700 Subject: [PATCH 2/3] Update hadoop-hdds/docs/content/tools/LogsInOzone.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- hadoop-hdds/docs/content/tools/LogsInOzone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/tools/LogsInOzone.md b/hadoop-hdds/docs/content/tools/LogsInOzone.md index aa8e60363fb6..051237ff75f3 100644 --- a/hadoop-hdds/docs/content/tools/LogsInOzone.md +++ b/hadoop-hdds/docs/content/tools/LogsInOzone.md @@ -97,7 +97,7 @@ After saving the file, the service will start logging more detailed debug inform To enable debug logging for Ozone CLI tools (e.g., `ozone sh volume create`), you can set the `OZONE_ROOT_LOGGER` environment variable to `debug`: ```bash -export OZONE_ROOT_LOGGER=debug,console +export OZONE_ROOT_LOGGER=DEBUG,console ozone sh volume create /vol1 ``` From 6d285e188797813d89831b61bb9ca1edd5c7b46d Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Wed, 9 Jul 2025 10:03:54 -0700 Subject: [PATCH 3/3] Docs: Clarify log level change requires restart The documentation for enabling debug logs for services was misleading. This commit clarifies that a service restart is required for log level changes to take effect. Change-Id: I12acc202e582d498ea4699e65026369b2591f06a --- hadoop-hdds/docs/content/tools/LogsInOzone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/tools/LogsInOzone.md b/hadoop-hdds/docs/content/tools/LogsInOzone.md index 051237ff75f3..552d45d821ff 100644 --- a/hadoop-hdds/docs/content/tools/LogsInOzone.md +++ b/hadoop-hdds/docs/content/tools/LogsInOzone.md @@ -90,7 +90,7 @@ to: rootLogger.level = debug ``` -After saving the file, the service will start logging more detailed debug information. +After saving the file and restarting the service, the service will start logging more detailed debug information. ### Enabling Debug Logs for CLI Tools