Improve / Cleanup DataFusion CI

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The CI system in arrow-datafusion largely dates to the time when it was part of the arrow repository. It is all in a single workflow, which among other things prevents more specific test triggering and also retriggering the entire run if just a single check fails

Describe the solution you'd like
I want to clean up the DataFusion CI system in a as-yet-to-be-determined way to make it faster and more-efficient

Describe alternatives you've considered
leave it alone

Additional context
I recently went through the same exercise in apache/arrow-rs#2149. I am not as

Inspired by #3043

[alamb]

I think this was done enough by @tustvold in #5047 so closing for now