HDFS-17863: CannotObtainBlockLengthException after DataNode restart

[teamconfx]

Description of PR

This PR fixes HDFS-17863.

The bug accurs where under-construction files become unreadable after DataNode restart, even though the data was successfully flushed with hflush(). This breaks HDFS's visibility guarantee for flushed data.

When a DataNode restarts, under-construction block replicas in the "rbw" (replica being written) directory are loaded as ReplicaWaitingToBeRecovered (RWR state). The getVisibleLength() method in this class unconditionally returned -1:

  // Before (ReplicaWaitingToBeRecovered.java:75-77)
  @Override
  public long getVisibleLength() {
    return -1;  //no bytes are visible
  }

When a client tries to read the file:

1. DFSInputStream calls readBlockLength() to determine the under-construction block length
2. It contacts the DataNode via getReplicaVisibleLength()
3. The DataNode returns -1 (from RWR replica)
4. Client treats this as invalid and throws CannotObtainBlockLengthException

This violates HDFS's hflush() contract which guarantees that flushed data remains visible to readers.

Changes

Changed ReplicaWaitingToBeRecovered.getVisibleLength() to return getNumBytes() instead of -1:

  // After (ReplicaWaitingToBeRecovered.java:75-77)
  @Override
  public long getVisibleLength() {
    return getNumBytes();  // all bytes are visible since validated on load
  }

Why This Fix Is Safe

The fix is safe because the block length returned by getNumBytes() has already been validated against checksums when the replica is loaded from disk.

In BlockPoolSlice.addReplicaToReplicasMap() (lines 693-700), RWR replicas are created with a validated length:

  if (loadRwr) {
    ReplicaBuilder builder = new ReplicaBuilder(ReplicaState.RWR)
        .setBlockId(blockId)
        .setLength(validateIntegrityAndSetLength(file, genStamp))  // <-- Validated!
        .setGenerationStamp(genStamp)
        ...
  }

The validateIntegrityAndSetLength() method (lines 871-920):

1. Reads the checksum from the meta file
2. Validates the last chunk of data against its checksum
3. Returns only the length of data that passes checksum validation
4. Truncates any corrupted trailing data

Therefore, getNumBytes() returns a checksum-verified length that is safe to expose to readers. This is the same validation used for RBW replicas loaded with valid restart metadata.

A New test added:

• TestPersistBlocks#testReadUnderConstructionFileAfterDataNodeRestart - Specifically reproduces the JIRA scenario

I also ran other tests related to this change:
Regression tests - all pass (58 tests):

• TestDatanodeRestart (2 tests)
• TestLeaseRecovery2 (8 tests)
• TestFileLengthOnClusterRestart (1 test)
• TestReadWhileWriting (1 test)
• TestTransferRbw (1 test)
• TestWriteRead (3 tests)
• TestWriteToReplica (6 tests)
• TestBlockRecovery (17 tests)
• TestBlockRecovery2 (5 tests)
• TestBlockListAsLongs (7 tests)
• TestPersistBlocks (7 tests)

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by "
  where is the name of the AI tool used.
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 41s		trunk passed
+1 💚	compile	0m 59s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 55s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 39s		trunk passed
+1 💚	mvnsite	1m 7s		trunk passed
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	2m 8s		trunk passed
+1 💚	shadedclient	16m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 44s		the patch passed
+1 💚	compile	0m 40s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 40s		the patch passed
+1 💚	compile	0m 40s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 40s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	1m 56s		the patch passed
+1 💚	shadedclient	16m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	174m 44s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		248m 31s

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8203/1/artifact/out/Dockerfile
GITHUB PR	#8203
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b04473ab70b6 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 405db4e
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8203/1/testReport/
Max. process+thread count	4830 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8203/1/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.