Parquet predicate filters fail with "Invalid comparison operation: Utf8View <= Utf8"

[ctsk]

Describe the bug

When running the tpch benchmark with debug logging, some errors are logged - they look like this:

DEBUG datafusion_datasource_parquet::row_group_filter] Error evaluating row group predicate on bloom filter: Arrow error: Invalid argument error: Invalid comparison operation: Utf8View <= Utf8

DEBUG datafusion_datasource_parquet::row_group_filter] Error evaluating row group predicate values Arrow error: Invalid argument error: Invalid comparison operation: Utf8View <= Utf8

DEBUG datafusion_datasource_parquet::page_filter] Error evaluating page index predicate values Arrow error: Invalid argument error: Invalid comparison operation: Utf8View <= Utf8

This does not affect the query result, but I wonder if this there is a cast / conversion missing somewhere.

To Reproduce

If not done yet, download the tpch dataset+convert it to parquet via

./bench.sh data tpch

Then run:

RUST_LOG=debug ./bench.sh run tpch |& grep error

Expected behavior

No errors =)

Additional context

No response

[adriangb]

Thanks for reporting.
We've been making changes to apply predicate pushdown filters against physical file schemas (as opposed the the table schema without partition columns) which I'm guessing causes this bug. It's a shame that it isn't accompanied by a debug_assertion! so it would have failed in tests (maybe).
I'm guessing this is related to / would be fixed by #15780.
Basically we should go over the predicate + file schema and add any necessary casts to the predicate side and remove any redundant casts.

[alamb]

• I think Evaluate filter pushdown against the physical schema for performance and correctness #15780 is ready to go now

I left some suggestions on how to start: #15780 (comment)

[adriangb]

It looks like this was introduced in dccf377.

The issue isn't actually #15780 as I originally thought (and that issue may need some revisiting in general) rather that there is no cast.

Previously we passed the table schemas into the pruning infrastructure.
That eventually hit this cast which converted the array to the table schema's data type.
Since we now pass in the file's phyiscal schema for pruning this cast doesn't do anything and thus this error is hit.

I think the solution should be to add casts for the literals into the file's data type. This avoids casting the data itself, which is especially important for row level predicate pushdown where that would be very expensive.

For cases where the data types match there is never an issue with performance or errors.

[adriangb]

Sadly I don't know that it's possible to make using the file schema completely correct. I'm looking into reverting that bit of the changes we've made, which has unfortunate implications for future optimizations IMO but I want to fix this bug ASAP and we can iterate from there. I'm out of time today but tomorrow I hope to get a PR up, local changes look promising.

[xudong963]

During upgrading DF47, the performance in our prod has obviously regressed, and from the metrics, I saw that there are lots of predicate_evaluation_errors, mainly from parquet part, also we have Utf8View, so I guess it may be related to the issue.