HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed #8349

kostacie · 2025-04-28T12:55:17Z

What changes were proposed in this pull request?

In this PR, a new condition for SCMContainerPlacementRackAware#chooseNode was added, it prevented Timeout exception in the test.

The main problem was that a datanode, which was restarted, didn't become a target for replication and the source datanode was selected instead. The SCM was choosing the same datanode, where the container is located, as a target and it made the test work incorrectly.

The fix just checks whether a current datanode (e.g. the source) is already in the used list. In case if it is, then it'll be added as an exclusion and won't be selected in the future.

What is the link to the Apache JIRA

HDDS-12760

How was this patch tested?

CI: https://github.com/kostacie/ozone/actions/runs/14702987624

adoroszlai · 2025-04-28T16:39:16Z

Thanks @kostacie for the patch. For verifying fix of flaky integration tests, please trigger 10x10 repeated run using flaky-test-check.

…ortedContainerIsClosed

kostacie · 2025-05-19T09:58:40Z

Thank you @adoroszlai for the review.
The results of check: https://github.com/kostacie/ozone/actions/runs/15109038542
Also the description and the PR were updated.

myskov

Thanks @kostacie for the patch

myskov · 2025-05-20T07:33:59Z

...pache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementRackAware.java

+    dnInfos.get(1).setNodeStatus(NodeStatus.inServiceHealthyReadOnly());
+
+    Exception e =
+        assertThrows(Exception.class,


Can be simplified to assertThrows(SCMException.class,

adoroszlai · 2025-05-21T16:17:35Z

Thanks @kostacie for updating the patch, and providing the test run.

https://github.com/kostacie/ozone/actions/runs/15109038542

Can you please also try running the check for ALL test methods from the same class? (There may be interaction among test cases, so running all methods provides more safety wrt. flakiness.)

kostacie · 2025-05-22T07:02:05Z

The test run for the whole class:
https://github.com/kostacie/ozone/actions/runs/15169178726

ashishkumar50 · 2025-05-28T05:14:50Z

...rg/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackAware.java


+      if (usedNodes != null && usedNodes.contains(node)) {
+        excludedNodesForCapacity.add(node.getNetworkFullPath());
+        continue;


excludedNodesForCapacity can be null, needs to initialise if it is null.

myskov · 2025-05-30T10:53:32Z

...pache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementRackAware.java

+            () -> policy.chooseDatanodes(usedNodes, null, null, 1, 0, 0),
+            "No target datanode, this call should fail");
+
+    assertEquals("SCMException", e.getClass().getSimpleName());


this assert is no longer needed, as this is now verified by assertThrows

myskov

@kostacie, should we remove the Flaky test annotation at testImportedContainerIsClosed() now?

ashishkumar50

LGTM

myskov

Thanks @kostacie for updating the patch. Thanks for reviewing @ashishkumar50 @adoroszlai

…ache#8349)

…239-container-reconciliation Commits: 62 da53b5b HDDS-13299. Fix failures related to delete (apache#8665) 8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662) 7329859 HDDS-13023. Container checksum is missing after container import (apache#8459) a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657) f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638) e9c0a45 HDDS-13262. Simplify key name validation (apache#8619) f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647) b574709 HDDS-12924. datanode used space calculation optimization (apache#8365) de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620) 97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639) 9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601) d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648) 6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503) a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645) 9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646) 48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630) 052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559) 90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633) 9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003) ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617) 3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631) 7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575) 5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626) 77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621) ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629) 36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628) d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627) 3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623) 8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625) a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624) 7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600) fd13300 HDDS-10775. Support bucket ownership verification (apache#8558) 3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584) ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554) 38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618) f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590) 0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612) ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616) 5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599) 1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614) 6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606) 4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596) bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607) 6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305) 09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611) 931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605) 3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest 6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573) 752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349) 8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586) ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602) bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571) 0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592) 516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530) 5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598) ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595) e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562) 3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516) f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547) b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532) 203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583) 2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java

kostacie added 3 commits April 24, 2025 11:27

Change default hdds.scm.safemode.min.datanode to 3

96d71a8

HDDS-12760. Change waitForMillis

6234f0a

Removed Flaky

30fe996

kerneltime requested a review from adoroszlai April 28, 2025 16:35

adoroszlai added the test label Apr 28, 2025

kostacie marked this pull request as draft April 29, 2025 11:51

kostacie and others added 3 commits April 29, 2025 18:41

Test flaky

1194d05

HDDS-12760. Added condition & test

9025583

Merge branch 'master' into hdds-12760_intermittent_timeout_in_testImp…

6d6469c

…ortedContainerIsClosed

kostacie marked this pull request as ready for review May 19, 2025 10:10

Cleanup

bdee02f

adoroszlai requested a review from siddhantsangwan May 19, 2025 10:37

Resolve conflicts

ffe1731

myskov reviewed May 20, 2025

View reviewed changes

Refactor test

b02cc4e

adoroszlai requested a review from ashishkumar50 May 27, 2025 10:21

ashishkumar50 reviewed May 28, 2025

View reviewed changes

myskov requested changes May 30, 2025

View reviewed changes

Add list init for excludedNodesForCapacity, change test

841bc96

myskov reviewed Jun 10, 2025

View reviewed changes

Remove Flaky annotation

f979ddb

kostacie requested a review from ashishkumar50 June 10, 2025 11:35

ashishkumar50 approved these changes Jun 11, 2025

View reviewed changes

myskov approved these changes Jun 11, 2025

View reviewed changes

myskov merged commit 752da2b into apache:master Jun 11, 2025
42 checks passed

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request Jun 13, 2025

HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (ap…

2c36806

…ache#8349)

sadanand48 pushed a commit to sadanand48/hadoop-ozone that referenced this pull request Jun 16, 2025

HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (ap…

43b4536

…ache#8349)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed #8349

HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed #8349

Uh oh!

kostacie commented Apr 28, 2025 •

edited

Loading

Uh oh!

adoroszlai commented Apr 28, 2025

Uh oh!

kostacie commented May 19, 2025

Uh oh!

myskov left a comment

Uh oh!

myskov May 20, 2025

Uh oh!

adoroszlai commented May 21, 2025

Uh oh!

kostacie commented May 22, 2025

Uh oh!

ashishkumar50 May 28, 2025

Uh oh!

myskov May 30, 2025

Uh oh!

myskov left a comment •

edited

Loading

Uh oh!

ashishkumar50 left a comment

Uh oh!

myskov left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed #8349

HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed #8349

Uh oh!

Conversation

kostacie commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai commented Apr 28, 2025

Uh oh!

kostacie commented May 19, 2025

Uh oh!

myskov left a comment

Choose a reason for hiding this comment

Uh oh!

myskov May 20, 2025

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented May 21, 2025

Uh oh!

kostacie commented May 22, 2025

Uh oh!

ashishkumar50 May 28, 2025

Choose a reason for hiding this comment

Uh oh!

myskov May 30, 2025

Choose a reason for hiding this comment

Uh oh!

myskov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

myskov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kostacie commented Apr 28, 2025 •

edited

Loading

myskov left a comment •

edited

Loading