Explore iceberg scan empty batches

[comphead]

It doesnt' have to be a bug, but mighty be interesting to explore

Looks like deep check before was able to catch empty schemas, please refer to failing tests https://github.com/apache/datafusion-comet/actions/runs/22410645074/job/64885589948?pr=3536

Leading to errors like

        org.apache.comet.CometNativeException: Batch adaptation failed: Arrow error: Invalid argument error: must either specify a row count or at least one column

when creating a record batch in RecordBatch::try_new(Arc::clone(target_schema), columns).map_err(|e| e.into())
in
adapt_batch_with_expressions

So both columns and row_count were empty

Originally posted by @comphead in #3536 (comment)

[mbutrovich]

Attaching the log so we don't lose it.

issue_3607.txt

[mbutrovich]

Dug into this a bit with Claude and I'm pleased with the investigation:

The zero-column guard in adapt_batch_with_expressions (lines 358-365) is necessary and correct.

Root cause: When Spark plans a query that only needs row counts (e.g., a scalar subquery like SELECT (SELECT max(id) FROM t) ..., or internal sub-plans of MOR delete/update/merge operations), the IcebergScanExec receives a target schema with zero fields. iceberg-rust returns batches with zero columns but non-zero row counts. RecordBatch::try_new fails in this case because with zero columns it can't infer the row count — it needs try_new_with_options with an explicit with_row_count.

Evidence: Instrumentation of the failing TestViews.createViewWithSubqueryExpressionInQueryThatIsRewritten test confirmed:
batch rows=1 cols=0 file_schema_fields=[] target_schema_fields=[]
batch rows=2 cols=0 file_schema_fields=[] target_schema_fields=[]

Both the file schema and target schema have zero fields. The batches carry only row counts and no column data.

Why this appeared in the DF 52 migration: The old DF 51 SchemaMapper::map_batch always used RecordBatch::try_new_with_options with an explicit row count (with the comment "Necessary to handle empty batches"). The DF 52 migration to PhysicalExprAdapter moved from batch-level mapping to expression-level rewriting, which lost that implicit handling. The guard re-adds it.

[comphead]

should we add a link to this ticket to that place in codebase @mbutrovich ?

[mbutrovich]

should we add a link to this ticket to that place in codebase @mbutrovich ?

Yeah we could do that!