HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier #7726

sodonnel · 2025-01-20T17:39:33Z

What changes were proposed in this pull request?

We have seen some instances where delete container commands are picked from the command queue within the SCM defined deadline. However they run for a very long time in the handler. This cases SCM to think the delete has been dropped or failed, when it is actually still running.

The causes of the slow running command could be:

Something else has a lock on the container for a long time, blocking the delete operation
Slow disk causing the removal of the container files to take a very long time.

To compound this problem, an ICR confirming the delete is not sent until the very last stage of the delete process.

To combat this, two changes are included in this PR:

Introduce a lock timeout of 60 seconds. If it takes longer than this for the lock and pre-checks to complete, the container delete is skipped.
Move the ICR to immediately after the point where the container is removed from the container set. At this stage, there is no way to recover the container without a DN restart and it makes sense to inform SCM that the container is logically removed ASAP.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12114

How was this patch tested?

New unit test added.

…d send ICR earlier

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java

errose28

Thanks @sodonnel overall looks good just some minor comments.

...er-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java

sodonnel · 2025-01-22T22:26:04Z

@errose28 I believe I have addressed the comments. Please have another check.

errose28

Thanks for the quick improvement @sodonnel. Optional if you want to fix the minor whitespace diff before merging

errose28 · 2025-01-23T21:36:37Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java

  public static final String
      OZONE_RECOVERING_CONTAINER_TIMEOUT_DEFAULT = "20m";

-


nit. whitespace diff

…d send ICR earlier (apache#7726) (cherry picked from commit b6cc4af) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java

…d send ICR earlier (apache#7726)

* CDPD-78092. HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier (apache#7726) (cherry picked from commit b6cc4af) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java Change-Id: I62ffb7203f2af5be2901ef923f333de53bbc3656 * CDPD-78149. HDDS-12115. RM selects replicas to delete non-deterministically if nodes are overloaded (apache#7728) (cherry picked from commit efd8adc) Conflicts: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestRatisOverReplicationHandler.java Change-Id: Ia3d54917c7c488a9b706f6ce941e7f466746d3bd * CDPD-78286. HDDS-12135. Set RM default deadline to 12 minutes and datanode offset to 6 minutes (apache#7747) (cherry picked from commit d7616ec) Change-Id: I36f237705f5a94d453bcec72c32056c2be8f38ba * CDPD-78213. HDDS-12127. RM should not expire pending deletes, but retry until delete is confirmed or node is dead (apache#7746) (cherry picked from commit 04f6255) Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ContainerReplicaPendingOps.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestMoveManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestContainerReplicaPendingOps.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestECContainerReplicaCount.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManagerScenarios.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java Change-Id: Ic01591f72706f2473c63dd2e44c3f2a94fb70d43 --------- Co-authored-by: Stephen O'Donnell <stephen.odonnell@gmail.com>

…d send ICR earlier (apache#7726)

…d send ICR earlier (apache#7726) (cherry picked from commit b6cc4af) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java

S O'Donnell added 2 commits January 20, 2025 17:25

HDDS-12114. Prevent delete commands running after a long lock wait an…

d3b9a22

…d send ICR earlier

Move ICR to immediately after the containerSet removal

73ba950

sodonnel requested a review from siddhantsangwan January 20, 2025 17:39

sodonnel changed the title ~~Hdds 12114~~ HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier Jan 20, 2025

Add setting to ozone-default.xml

8a524f4

errose28 reviewed Jan 20, 2025

View reviewed changes

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java Show resolved Hide resolved

ivandika3 requested a review from xichen01 January 21, 2025 06:30

Move ICR to after the container is moved to the tmp dir

73c3a71

xichen01 reviewed Jan 22, 2025

View reviewed changes

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java Show resolved Hide resolved

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java Show resolved Hide resolved

errose28 reviewed Jan 22, 2025

View reviewed changes

S O'Donnell added 3 commits January 22, 2025 21:46

Remove try finally to delete tmp dir

c739b34

Add comments about clock mock

7cdbc3c

Move to type safe config

d6e1c21

errose28 approved these changes Jan 23, 2025

View reviewed changes

remove whitespace change

6f7b4b9

sodonnel merged commit b6cc4af into apache:master Jan 24, 2025
42 checks passed

nandakumar131 pushed a commit to nandakumar131/ozone that referenced this pull request Feb 10, 2025

HDDS-12114. Prevent delete commands running after a long lock wait an…

c76b039

…d send ICR earlier (apache#7726)

Cyrill pushed a commit to Cyrill/ozone that referenced this pull request Nov 10, 2025

HDDS-12114. Prevent delete commands running after a long lock wait an…

706f38c

…d send ICR earlier (apache#7726)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier #7726

HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier #7726

Uh oh!

sodonnel commented Jan 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sodonnel commented Jan 22, 2025

Uh oh!

errose28 left a comment

Uh oh!

errose28 Jan 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		public static final String
		OZONE_RECOVERING_CONTAINER_TIMEOUT_DEFAULT = "20m";

HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier #7726

HDDS-12114. Prevent delete commands running after a long lock wait and send ICR earlier #7726

Uh oh!

Conversation

sodonnel commented Jan 20, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sodonnel commented Jan 22, 2025

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

errose28 Jan 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants