Test Spark 4.0.0-preview1

[lfrancke]

Note

Timeboxed: Maximum 2 days

This issue covers trying to get a Docker image including Spark 4.0.0 (preview) built so we can include it as an experimental component.

We already know that Iceberg will probably not work without extra effort: apache/iceberg#10497

The result of this timeboxed issue should be (in this order)

• A docker image with Spark 4.0.0-preview1
• Run our tests against it
• A list of things that do not work with Spark 4 in our stack

Please ensure that you spend a maximum of 2 days on this and make sure that this includes writing down the results.

[sbernauer]

We talked about this in the daily and decided to propably skip it for now (no crazy new features)

[sbernauer]

We decided to not ship this in 24.11, closing this. We can re-visit once 4.0.0 is officially released