perf: Add COMET_RESPECT_PARQUET_FILTER_PUSHDOWN config

[andygrove]

Which issue does this PR close?

N/A

Rationale for this change

The new native scans perform poorly when Parquet filter pushdown is enabled, which is the default in Spark.

See apache/datafusion#3463 for reasons why filter pushdown is not enabled in DataFusion by default yet.

What changes are included in this PR?

Add a new config that tells Comet whether to respect Spark's filter pushdown config. We need to respect the config when running Spark SQL tests, but want to ignore the config by default for best performance.

How are these changes tested?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.

Project coverage is 33.43%. Comparing base (f09f8af) to head (e7b2a58).
Report is 291 commits behind head on main.

Files with missing lines	Patch %	Lines
.../scala/org/apache/comet/serde/QueryPlanSerde.scala	57.14%	1 Missing and 2 partials ⚠️
.../apache/comet/parquet/CometParquetFileFormat.scala	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #1936       +/-   ##
=============================================
- Coverage     56.12%   33.43%   -22.70%     
+ Complexity      976      804      -172     
=============================================
  Files           119      131       +12     
  Lines         11743    12917     +1174     
  Branches       2251     2402      +151     
=============================================
- Hits           6591     4319     -2272     
- Misses         4012     7660     +3648     
+ Partials       1140      938      -202     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[comphead]

Thanks @andygrove I think its lgtm, although it might be confusing having a parameter that enables another parameter 🤔

[andygrove]

Thanks @andygrove I think its lgtm, although it might be confusing having a parameter that enables another parameter 🤔

Yeah, I know. The alternative is to ask users to disable the Spark config, but I'm assuming that most users won't read the documentation to discover that this is needed for good performance.

[parthchandra]

lgtm

[parthchandra]

This is not necessary right now because this is part of DSV2 support and the new native scan impls do not support DSV2.
Though we might add it for native_iceberg_compat

[andygrove]

Thanks. I reverted this change.

[kazuyukitanimura]

Do we need to add .set("spark.comet.parquet.respectFilterPushdown", "true") at a few more locations?
E.g. TestHive.scala
Could be other locations as well

[andygrove]

All of the Spark SQL tests are passing.

[andygrove]

I checked, and there are no hive tests that reference PARQUET_FILTER_PUSHDOWN_ENABLED.

[andygrove]

Thanks for the reviews @kazuyukitanimura @parthchandra @comphead