From 9366ea37680c6ad4ca673dec2b5d65c308b323fa Mon Sep 17 00:00:00 2001 From: Gargi Jaiswal Date: Thu, 6 Feb 2025 11:19:31 +0530 Subject: [PATCH 1/4] HDDS-10794.Update docker compose user doc --- .../docs/content/start/RunningViaDocker.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/hadoop-hdds/docs/content/start/RunningViaDocker.md b/hadoop-hdds/docs/content/start/RunningViaDocker.md index 9e1e36112253..d7c526ac53f4 100644 --- a/hadoop-hdds/docs/content/start/RunningViaDocker.md +++ b/hadoop-hdds/docs/content/start/RunningViaDocker.md @@ -59,3 +59,94 @@ To shutdown the cluster, please run docker-compose down {{< /highlight >}} +## [Add-Ons](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozone/README.md) + +The cluster can be extended with the following optional services: +* **Monitoring**: Adds Grafana, Jaeger, and Prometheus, and configures Ozone to work with them. This helps you track system performance and diagnose issues. +* **Profiling**: Enables [async-profiler](https://github.com/async-profiler/async-profiler) to sample CPU and memory usage, helping analyze Ozone’s resource consumption. + +You can enable **Monitoring** and **Profiling** add-ons using Docker Compose’s ability to merge multiple configuration files. This can be done by using the [-f option repeatedly](https://docs.docker.com/reference/compose-file/#specifying-multiple-compose-files) +multiple times or by setting the [COMPOSE_FILE environment variable](https://docs.docker.com/compose/how-tos/environment-variables/envvars/#compose_file) for convenience. + +**Enabling Add-ons** + +By default, only the core Ozone cluster runs. To enable additional +features, set the `COMPOSE_FILE` environment variable before starting the cluster: +{{< highlight bash >}} +# no COMPOSE_FILE var => only Ozone + +# => add monitoring +export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml +# => add profiling +export COMPOSE_FILE=docker-compose.yaml:profiling.yaml +# => add both +export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml:profiling.yaml +{{< /highlight >}} + +Once the variable is defined, ozone cluster with add-ons +can be started/scaled/stopped etc. using the same `docker-compose` commands as for the base cluster. + +## Load generator + +Apache Ozone includes a built-in generator called +Freon, which helps to test the system performance +by generating artificial workloads. + +**Running Freon inside a Container** + +You can manually start a `Freon` test by entering a running container, such as the SCM, and executing the following command: +{{< highlight bash >}} +docker-compose exec scm bash +ozone freon ockg -n1000 +{{< /highlight >}} +This runs a test that generates 1,000 objects in Ozone. + +You can also start two flavours of Freon as +separate services, which allows scaling them up. +Once all the data nodes are started, start +Freon by adding its definition to `COMPOSE_FILE` +and re-running the `docker-compose up` or `run.sh` command: +{{< highlight bash >}} +export COMPOSE_FILE="${COMPOSE_FILE}:freon-ockg.yaml" + +docker-compose up -d --no-recreate --scale datanode=3 +# OR +./run.sh -d +{{< /highlight >}} + +## How to use + +**Monitoring** + +You can check the web UI of Prometheus, +Grafana and Jaeger using the following step: + +Do `docker-compose ps` , it will show you list of +services running in the cluster along with the +localhost on which each of them are hosted. +By navigating to the localhost of each services +you can check the `web UI`. + +**Prometheus -** +It follows a pull based approach where +metrics are published on an HTTP endpoint. + +**Grafana -** +It comes with three dashboards for ozone: + +* Ozone-Object metrics +* Ozone-RPC Metrics +* Ozone-Overall Metrics + +**Jaeger -** +It collects distributed tracing information +from Ozone. + +**Profiling** + +Start by hitting the `/prof` endpoint on the +service to be profiled, +eg. http://localhost:9876/prof for SCM. +Detailed instructions +can be found in the Hadoop wiki. + From 77b224a87ec71f08cee7154cdcfb89205b9291e0 Mon Sep 17 00:00:00 2001 From: Gargi Jaiswal Date: Thu, 6 Feb 2025 12:32:29 +0530 Subject: [PATCH 2/4] HDDS-12221.Remove unused ozone.block.deleting.limit.per.task config property --- .../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 +---- .../common/src/main/resources/ozone-default.xml | 13 +------------ .../container/common/TestBlockDeletingService.java | 2 -- .../common/TestSchemaOneBackwardsCompatibility.java | 2 +- .../common/TestSchemaTwoBackwardsCompatibility.java | 3 +-- 5 files changed, 4 insertions(+), 21 deletions(-) diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java index 4c0df91e1a63..d4271487df80 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java @@ -328,10 +328,7 @@ public final class OzoneConfigKeys { public static final int OZONE_KEY_PREALLOCATION_BLOCKS_MAX_DEFAULT = 64; - public static final String OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER = - "ozone.block.deleting.limit.per.task"; - public static final int OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER_DEFAULT - = 1000; + public static final String OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL = "ozone.block.deleting.container.limit.per.interval"; diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml index f7ff9089b926..4e65db7369af 100644 --- a/hadoop-hdds/common/src/main/resources/ozone-default.xml +++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml @@ -437,16 +437,6 @@ threads spawned for block deletions. - - ozone.block.deleting.limit.per.task - 1000 - OZONE, PERFORMANCE, SCM - A maximum number of blocks to be deleted by block deleting - service per - time interval. This property is used to throttle the actual number of - block deletions on a data node per container. - - ozone.block.deleting.service.interval 1m @@ -464,8 +454,7 @@ A timeout value of block deletion service. If this is set greater than 0, the service will stop waiting for the block deleting completion after this - time. If timeout happens to a large proportion of block deletion, this - needs to be increased with ozone.block.deleting.limit.per.task. This + time. This setting supports multiple time unit suffixes as described in dfs.heartbeat.interval. If no suffix is specified, then milliseconds is assumed. diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java index bc56141fb080..cc85a2bc027a 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java @@ -96,7 +96,6 @@ import static java.nio.charset.StandardCharsets.UTF_8; import static org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric; import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL; -import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER; import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_BLOCK_DELETING_SERVICE_INTERVAL; import static org.apache.hadoop.ozone.OzoneConsts.SCHEMA_V1; import static org.apache.hadoop.ozone.OzoneConsts.SCHEMA_V2; @@ -771,7 +770,6 @@ public void testShutdownService(ContainerTestVersionInfo versionInfo) conf.setTimeDuration(OZONE_BLOCK_DELETING_SERVICE_INTERVAL, 500, TimeUnit.MILLISECONDS); conf.setInt(OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL, 10); - conf.setInt(OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER, 10); ContainerSet containerSet = new ContainerSet(1000); // Create 1 container with 100 blocks diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java index 2235b23ce882..17d43cd1cafe 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java @@ -556,7 +556,7 @@ public void testReadDeletedBlocks(String schemaVersion) throws Exception { private void runBlockDeletingService(KeyValueHandler keyValueHandler) throws Exception { conf.setInt(OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL, 10); - conf.setInt(OzoneConfigKeys.OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER, 2); + OzoneContainer container = makeMockOzoneContainer(keyValueHandler); diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java index 0c4612b79fa2..9be77a52ccb8 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java @@ -225,8 +225,7 @@ public void testReadMetadata() throws IOException { public void testDeleteViaTransation() throws IOException, TimeoutException, InterruptedException { conf.setInt(OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL, 10); - conf.setInt(OzoneConfigKeys.OZONE_BLOCK_DELETING_LIMIT_PER_CONTAINER, - BLOCKS_PER_CONTAINER); + // create a container of schema v2 KeyValueContainer container = createTestContainer(); From 52d93b479e9709c7989e436c31c824f7b5369030 Mon Sep 17 00:00:00 2001 From: Gargi Jaiswal Date: Thu, 6 Feb 2025 12:51:18 +0530 Subject: [PATCH 3/4] HDDS-12221.Remove unused ozone.block.deleting.limit.per.task config property --- .../container/common/TestSchemaOneBackwardsCompatibility.java | 1 - .../container/common/TestSchemaTwoBackwardsCompatibility.java | 1 - 2 files changed, 2 deletions(-) diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java index 17d43cd1cafe..8aaed1c9a1ef 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaOneBackwardsCompatibility.java @@ -24,7 +24,6 @@ import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.utils.MetadataKeyFilters.KeyPrefixFilter; import org.apache.hadoop.hdds.utils.db.Table; -import org.apache.hadoop.ozone.OzoneConfigKeys; import org.apache.hadoop.ozone.OzoneConsts; import org.apache.hadoop.ozone.container.common.helpers.BlockData; import org.apache.hadoop.ozone.container.common.helpers.ChunkInfoList; diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java index 9be77a52ccb8..97d564931740 100644 --- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java +++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestSchemaTwoBackwardsCompatibility.java @@ -26,7 +26,6 @@ import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException; import org.apache.hadoop.hdds.utils.db.BatchOperation; import org.apache.hadoop.hdds.utils.db.Table; -import org.apache.hadoop.ozone.OzoneConfigKeys; import org.apache.hadoop.ozone.OzoneConsts; import org.apache.hadoop.ozone.common.ChunkBuffer; import org.apache.hadoop.ozone.container.ContainerTestHelper; From 2b474701cffc8562e473aa561b378bc8e55b5f14 Mon Sep 17 00:00:00 2001 From: Gargi Jaiswal Date: Thu, 6 Feb 2025 15:25:40 +0530 Subject: [PATCH 4/4] HDDS-12221.Remove unused ozone.block.deleting.limit.per.task config property --- .../apache/hadoop/ozone/OzoneConfigKeys.java | 2 - .../docs/content/start/RunningViaDocker.md | 91 ------------------- 2 files changed, 93 deletions(-) diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java index d4271487df80..d5ac13d02c9d 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java @@ -328,8 +328,6 @@ public final class OzoneConfigKeys { public static final int OZONE_KEY_PREALLOCATION_BLOCKS_MAX_DEFAULT = 64; - - public static final String OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL = "ozone.block.deleting.container.limit.per.interval"; public static final int diff --git a/hadoop-hdds/docs/content/start/RunningViaDocker.md b/hadoop-hdds/docs/content/start/RunningViaDocker.md index d7c526ac53f4..9e1e36112253 100644 --- a/hadoop-hdds/docs/content/start/RunningViaDocker.md +++ b/hadoop-hdds/docs/content/start/RunningViaDocker.md @@ -59,94 +59,3 @@ To shutdown the cluster, please run docker-compose down {{< /highlight >}} -## [Add-Ons](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozone/README.md) - -The cluster can be extended with the following optional services: -* **Monitoring**: Adds Grafana, Jaeger, and Prometheus, and configures Ozone to work with them. This helps you track system performance and diagnose issues. -* **Profiling**: Enables [async-profiler](https://github.com/async-profiler/async-profiler) to sample CPU and memory usage, helping analyze Ozone’s resource consumption. - -You can enable **Monitoring** and **Profiling** add-ons using Docker Compose’s ability to merge multiple configuration files. This can be done by using the [-f option repeatedly](https://docs.docker.com/reference/compose-file/#specifying-multiple-compose-files) -multiple times or by setting the [COMPOSE_FILE environment variable](https://docs.docker.com/compose/how-tos/environment-variables/envvars/#compose_file) for convenience. - -**Enabling Add-ons** - -By default, only the core Ozone cluster runs. To enable additional -features, set the `COMPOSE_FILE` environment variable before starting the cluster: -{{< highlight bash >}} -# no COMPOSE_FILE var => only Ozone - -# => add monitoring -export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml -# => add profiling -export COMPOSE_FILE=docker-compose.yaml:profiling.yaml -# => add both -export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml:profiling.yaml -{{< /highlight >}} - -Once the variable is defined, ozone cluster with add-ons -can be started/scaled/stopped etc. using the same `docker-compose` commands as for the base cluster. - -## Load generator - -Apache Ozone includes a built-in generator called -Freon, which helps to test the system performance -by generating artificial workloads. - -**Running Freon inside a Container** - -You can manually start a `Freon` test by entering a running container, such as the SCM, and executing the following command: -{{< highlight bash >}} -docker-compose exec scm bash -ozone freon ockg -n1000 -{{< /highlight >}} -This runs a test that generates 1,000 objects in Ozone. - -You can also start two flavours of Freon as -separate services, which allows scaling them up. -Once all the data nodes are started, start -Freon by adding its definition to `COMPOSE_FILE` -and re-running the `docker-compose up` or `run.sh` command: -{{< highlight bash >}} -export COMPOSE_FILE="${COMPOSE_FILE}:freon-ockg.yaml" - -docker-compose up -d --no-recreate --scale datanode=3 -# OR -./run.sh -d -{{< /highlight >}} - -## How to use - -**Monitoring** - -You can check the web UI of Prometheus, -Grafana and Jaeger using the following step: - -Do `docker-compose ps` , it will show you list of -services running in the cluster along with the -localhost on which each of them are hosted. -By navigating to the localhost of each services -you can check the `web UI`. - -**Prometheus -** -It follows a pull based approach where -metrics are published on an HTTP endpoint. - -**Grafana -** -It comes with three dashboards for ozone: - -* Ozone-Object metrics -* Ozone-RPC Metrics -* Ozone-Overall Metrics - -**Jaeger -** -It collects distributed tracing information -from Ozone. - -**Profiling** - -Start by hitting the `/prof` endpoint on the -service to be profiled, -eg. http://localhost:9876/prof for SCM. -Detailed instructions -can be found in the Hadoop wiki. -