[hdfs-opendal] Got an `reader got too little data` error on hdfs-opendal

[wForget]

Describe the bug

https://github.com/apache/datafusion-comet/actions/runs/17752053952/job/50448541640?pr=2350

- test native_datafusion scan on fake fs *** FAILED *** (8 seconds, 838 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 6.0 failed 1 times, most recent failure: Lost task 2.0 in stage 6.0 (TID 15) (localhost executor driver): org.apache.spark.SparkException: Encountered error while reading file fake://fake-bucket/Users/runner/work/datafusion-comet/datafusion-comet/spark/target/tmp/comet_fake_3afe6cec-86bd-48ff-97a1-d354fec360327728661446900846429/data/test-file.parquet/part-00001-82d42ceb-6226-4ed6-8b34-d7108fc0b695-c000.snappy.parquet. Details:
	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:864)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:296)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
	at org.apache.spark.sql.comet.CometScanExec$$anon$1.hasNext(CometScanExec.scala:271)
	at org.apache.comet.CometBatchIterator.hasNext(CometBatchIterator.java:60)
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:165)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:164)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:212)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:164)
	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:162)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:213)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.comet.CometBatchIterator.hasNext(CometBatchIterator.java:60)
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:165)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:164)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:212)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:164)
	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:162)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:213)
	at org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriter.write(CometNativeShuffleWriter.scala:106)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.comet.CometNativeException: External: Generic IoError error: Unexpected (permanent) at , context: { expect: 8, actual: 0 } => reader got too little data
	at org.apache.comet.parquet.Native.readNextRecordBatch(Native Method)
	at org.apache.comet.parquet.NativeBatchReader.loadNextBatch(NativeBatchReader.java:812)
	at org.apache.comet.parquet.NativeBatchReader.nextBatch(NativeBatchReader.java:749)
	at org.apache.comet.parquet.NativeBatchReader.nextKeyValue(NativeBatchReader.java:707)
	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:286)
	... 36 more

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[comphead]

@parthchandra cc

[andygrove]

I'm seeing this error quite often. From today:

2026-02-10T19:19:39.9450480Z ParquetReadFromFakeHadoopFsSuite:
2026-02-10T19:19:41.4321960Z - test native_datafusion scan on fake fs *** FAILED *** (1 second, 496 milliseconds)
2026-02-10T19:19:41.4429130Z   org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 2.0 failed 1 times, most recent failure: Lost task 2.0 in stage 2.0 (TID 8) (localhost executor driver): org.apache.comet.CometNativeException: External: Generic IoError error: Unexpected (permanent) at , context: { expect: 8, actual: 0 } => reader got too little data

[andygrove]

@mbutrovich maybe you could take a look when you have a chance?

[comphead]

I would like to take this @mbutrovich I was able to see it in cluster run

org.apache.comet.CometNativeException: External: Generic IoError error: Unexpected (permanent) at , context: { expect: 8, actual: 0 } => reader got too little data
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:148)

[wForget]

I suspect this exception occurs during the reading of parquet footer, since size of footer is 8 bytes:

https://github.com/apache/arrow-rs/blob/2bf6909305091c69edddb0f16c76184edd206141/parquet/src/file/mod.rs#L111

[wForget]

Referring to apache/opendal#4357, I suspect this issue might be caused by:

https://github.com/apache/arrow-rs/blob/2bf6909305091c69edddb0f16c76184edd206141/parquet/src/file/metadata/reader.rs#L560

[wForget]

Referring to apache/opendal#4357, I suspect this issue might be caused by:

https://github.com/apache/arrow-rs/blob/2bf6909305091c69edddb0f16c76184edd206141/parquet/src/file/metadata/reader.rs#L560

@Xuanwo could you please take a look?

[Xuanwo]

Hi, this error is returned by OpenDAL here: https://github.com/apache/opendal/blob/ec16a1e56ca329777805b81285e9928df1faa616/core/core/src/layers/complete.rs#L183-L202

To be more specific, OpenDAL checks the returned data to verify that the fetched data exactly matches the size of the request we sent. In this case, OpenDAL attempted to read 8 bytes, but we received 0 bytes.

If this happens often, we may need to check OpenDAL's HDFS service. One suggestion for the investigation is to try calling OpenDAL directly with a similar read_exact call to see what happens under the hood.

For example:

https://github.com/apache/opendal/blob/ec16a1e56ca329777805b81285e9928df1faa616/core/services/hdfs/src/reader.rs#L64-L70

If we did this in wrong here? Should we just loop read while hdfs returns 0?