Skip to content

Conversation

@errose28
Copy link
Contributor

@errose28 errose28 commented Feb 22, 2025

What changes were proposed in this pull request?

Summary

Currently ozone admin container reconcile is used to trigger reconciliation for a specific container, but the resulting checksums are only shown in ozone admin container info --json which can be verbose and usually requires jq filtering to get the desired information.

This PR adds new functionality to the ozone admin container reconcile command:

  • Multiple container IDs can be passed: ozone admin container reconcile 1 2 3 4
  • IDs can be read from stdin, similar to ozone admin container info: cat <id-file> | ozone admin container reconcile -
  • Either of the above options work with a --status flag that prints replica and checksum information instead of sending reconcile commands: ozone admin container reconcile --status 1 2 3 4
    • In the status output, a replicasMatch field is set to true or false on the client side, which can be used to quickly filter out containers with mismatched replicas.

No proto changes are made. The same ScmClient#getContainer and ScmClient#getContainerReplicas methods used by ozone admin container info are re-used for ozone admin container reconcile --status. This command just provides a more filtered view of the results specific to reconciliation and adds the replicasMatch field.

Examples

Querying status of one container:

$ ozone admin container reconcile --status 1               
[ {
  "containerID" : 1,
  "state" : "CLOSING",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  } ]
} ]

Querying the status of multiple containers (json list output):

sh-4.4$ ozone admin container reconcile --status 1  3             
[ {
  "containerID" : 1,
  "state" : "CLOSED",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  } ]
}, {
  "containerID" : 3,
  "state" : "OPEN",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  } ]
} ]

Given a list of container IDs in a file, output the IDs of all the ones with mismatched replicas:

$ cat ids.txt | ozone admin container reconcile --status - | jq '.[] | select(.replicasMatch = false) | .containerID'
1
3

Same as above, but now send the IDs of mismatched containers back in for reconciliation:

cat ids.txt | ozone admin container reconcile --status - | jq '.[] | select(.replicasMatch = false) | .containerID' | ozone admin container reconcile -
Reconciliation has been triggered for container 1
Reconciliation has been triggered for container 3
Use "ozone admin container reconcile --status" to see the checksums of each container replica

What is the link to the Apache JIRA

HDDS-12078

How was this patch tested?

  • Tested manually in docker
  • Unit tests
  • Update existing acceptance tests to use the new --status flag instead of ozone admin container info -json.

…prove-cli

* HDDS-10239-container-reconciliation: (113 commits)
  HDDS-12361. Mark testGetSnapshotDiffReportJob as flaky
  HDDS-12375. Random object created and used only once (apache#7933)
  HDDS-12335. Fix ozone admin namespace summary to give complete output (apache#7908)
  HDDS-12364. Require Override annotation for overridden methods (apache#7923)
  HDDS-11530. Support listMultipartUploads max uploads and markers (apache#7817)
  HDDS-11867. Remove code paths for non-Ratis SCM. (apache#7911)
  HDDS-12363. Add import options to IntelliJ IDEA style settings (apache#7921)
  HDDS-12215. Mark testContainerStateMachineRestartWithDNChangePipeline as flaky
  HDDS-12331. BlockOutputStream.failedServers is not thread-safe (apache#7885)
  HDDS-12188. Move server-only upgrade classes from hdds-common to hdds-server-framework (apache#7903)
  HDDS-12362. Remove temporary checkstyle suppression file (apache#7920)
  HDDS-12343. Fix spotbugs warnings in Recon (apache#7902)
  HDDS-12286. Fix license headers and imports for ozone-tools (apache#7919)
  HDDS-12284. Fix license headers and imports for ozone-s3-secret-store (apache#7917)
  HDDS-12275. Fix license headers and imports for ozone-integration-test (apache#7904)
  HDDS-12164. Rename and deprecate DFSConfigKeysLegacy config keys (apache#7803)
  HDDS-12283. Fix license headers and imports for ozone-recon-codegen (apache#7916)
  HDDS-12330. Convert Volume, Bucket and Key count to use comma separated numbers (apache#7881)
  HDDS-12281. Fix license headers and imports for ozone-filesystem-hadoop3 (apache#7913)
  HDDS-12285. Fix license headers and imports for ozone-s3gateway (apache#7918)
  ...

Conflicts:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaInfo.java
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/server/JsonUtils.java
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java
@errose28 errose28 changed the title HDDS-12078 improve cli HDDS-12078. Improve container reconciliation CLIs Feb 22, 2025
Copy link
Contributor

@kerneltime kerneltime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer robot tests over integration tests. Overall seems fine, will go over a few more times.

Comment on lines 127 to 128
System.out.println("Use \"ozone admin container reconcile --status " + containerID + "\" to see the checksums of " +
"each container replica");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have a good way to indicate when reconciliation is completed... Maybe we add a reconciliation count as a metric to the container info?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, some sort of feedback that it finished would be good. I was planning to revisit this with the SCM integration. For example, SCM may end up tracking the known in-flight reconciliations and could report the results.

Are you suggesting adding a jmx metric or just a counter in the code? I'm not sure exactly how this work, but I imagine the counter would get persisted on the DN side and SCM would see the updates. Users would then look for the count to increase when they send in a command.

@errose28
Copy link
Contributor Author

I prefer robot tests over integration tests. Overall seems fine, will go over a few more times.

Yes I will add robot tests as well. The current tests are unit tests without a mini ozone cluster. The scm client is mocked to return various combinations of replicas that would otherwise be hard to create in a mini ozone or docker cluster.

@aswinshakil
Copy link
Member

HDDS-10239-container-reconciliation is merged into master. Please merge this change into master and any other future changes into master

errose28 added 9 commits July 24, 2025 16:54
* master: (730 commits)
  HDDS-13083. Handle cases where block deletion generates tree file before scanner (apache#8565)
  HDDS-12982. Reduce log level for snapshot validation failure (apache#8851)
  HDDS-13396. Documentation: Improve the top-level overview page for new users. (apache#8753)
  HDDS-13176. containerIds table value format change to proto from string (apache#8589)
  HDDS-13449. Incorrect Interrupt Handling for DirectoryDeletingService and KeyDeletingService (apache#8817)
  HDDS-2453. Add Freon tests for S3 MPU Keys (apache#8803)
  HDDS-13237. Container data checksum should contain block IDs. (apache#8773)
  HDDS-13489. Fix SCMBlockdeleting unnecessary iteration in corner case. (apache#8847)
  HDDS-13464. Make ozone.snapshot.filtering.service.interval reconfigurable (apache#8825)
  HDDS-13473. Amend validation for OZONE_OM_SNAPSHOT_DB_MAX_OPEN_FILES (apache#8829)
  HDDS-13435. Add an OzoneManagerAuthorizer interface (apache#8840)
  HDDS-8565. Recon memory leak in NSSummary (apache#8823).
  HDDS-12852. Implement a sliding window counter utility (apache#8498)
  HDDS-12000. Add unit test for RatisContainerSafeModeRule and ECContainerSafeModeRule (apache#8801)
  HDDS-13092. Container scanner should trigger volume scan when marking a container unhealthy (apache#8603)
  HDDS-13070. OM Follower changes to create and place sst files from hardlink file. (apache#8761)
  HDDS-13482. Mark testWriteStateMachineDataIdempotencyWithClosedContainer as flaky
  HDDS-13481. Fix success latency metric in SCM panels of deletion grafana dashboard (apache#8835)
  HDDS-13468. Update default value of ozone.scm.ha.dbtransactionbuffer.flush.interval. (apache#8834)
  HDDS-13410. Control block deletion for each DN from SCM. (apache#8767)
  ...

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaInfo.java
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java
hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/hdds/scm/cli/container/TestReconcileSubcommand.java
* HDDS-12078-improve-cli-ai:
  Use common mixin for container ID list
Generated-By: Cursor
Generated-By: Cursor
@errose28 errose28 changed the base branch from HDDS-10239-container-reconciliation to master July 30, 2025 23:24
@errose28
Copy link
Contributor Author

The latest updates should have all functionality and unit tests. Just acceptance tests need to be updated. I've added container ID validation in the common containerIDParameters#getValidatedIDs so we can use the same validation for other commands like ozone admin container info as well, but migrating those can be done in a follow-up change.

@errose28
Copy link
Contributor Author

errose28 commented Aug 7, 2025

@aswinshakil I've updated the acceptance tests for the new CLI, although most of the testing is still happening in the unit test. PTAL

* master: (55 commits)
  HDDS-13525. Rename configuration property to ozone.om.compaction.service.enabled (apache#8928)
  HDDS-13519. Reconciliation should continue if a peer datanode is unreachable (apache#8908)
  HDDS-13566. Fix incorrect authorizer class in ACL documentation (apache#8931)
  HDDS-13084. Trigger on-demand container scan when a container moves from open to unhealthy. (apache#8904)
  HDDS-13432. Accelerating Namespace Usage Calculation in Recon using - Materialised Approach (apache#8797)
  HDDS-13557. Bump jline to 3.30.5 (apache#8920)
  HDDS-13556. Bump assertj-core to 3.27.4 (apache#8919)
  HDDS-13543. [Docs] Design doc for OM bootstrapping process with snapshots. (apache#8900)
  HDDS-13541. Bump sonar-maven-plugin to 5.1.0.4751 (apache#8911)
  HDDS-13101. Remove duplicate information in datanode list output (apache#8523)
  HDDS-13528. Handle null paths when the NSSummary is initializing (apache#8901)
  HDDS-12990. (addendum) Generate tree from metadata when it does not exist during getContainerChecksumInfo call (apache#8881)
  HDDS-13086. Block duplicate reconciliation requests for the same container and datanode within the datanode. (apache#8905)
  HDDS-12990. Generate tree from metadata when it doesn't exist during getContainerChecksumInfo call (apache#8881)
  HDDS-12824. Optimize container checksum read during datanode startup (apache#8604)
  HDDS-13522. Rename axisLabel for No. of delete request received (apache#8879)
  HDDS-12196. Document ozone repair cli (apache#8849)
  HDDS-13514. Intermittent failure in TestNSSummaryMemoryLeak (apache#8889)
  HDDS-13423. Log reason for triggering on-demand container scan (apache#8854)
  HDDS-13466. Disable flaky TestOmSnapshotFsoWithNativeLibWithLinkedBuckets
  ...
@errose28 errose28 requested a review from aswinshakil August 12, 2025 22:46
@aswinshakil aswinshakil marked this pull request as ready for review August 13, 2025 20:56
Copy link
Member

@aswinshakil aswinshakil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @errose28 . The changes look good to me. Let's wait for the CI to pass.

* master:
  HDDS-13553. Recon Staging DB for OM full db reprocess (apache#8917)
  HDDS-13138. [Docs] Update Topology Awareness user doc. (apache#8528)
  HDDS-11944. Usability improvements for Ozone tools. (apache#7597)
  HDDS-12197. Update documentation for all ozone debug tools (apache#8868)
@errose28
Copy link
Contributor Author

I found more issues with the output after trying things manually, which should be fixed now. I also updated the jq queries used in acceptance tests that were previously failing.

@errose28 errose28 requested a review from aswinshakil August 19, 2025 17:52
@aswinshakil
Copy link
Member

Overall, the changes look good to me. I was wondering if we should have a limit on the number list statuses so that we don't overwhelm the system.

@errose28
Copy link
Contributor Author

Each status is a separate call to SCM and the json is streamed as it is generated so it won't create a giant proto to send over the network or fill up client memory. The CLI to query status of multiple containers is a shorthand for running something like for i in {1..1000} do; ozone admin container reconcile --status $i; done so I don't think a limit per CLI invocation has any benefit.

@errose28 errose28 merged commit 0839c24 into apache:master Sep 3, 2025
121 of 122 checks passed
@errose28
Copy link
Contributor Author

errose28 commented Sep 3, 2025

Thanks @aswinshakil for the review.

peterxcli pushed a commit to peterxcli/ozone that referenced this pull request Sep 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants