Improve Spark shim layer

[sunchao]

What is the problem the feature request solves?

Currently we have various shim classes which use reflection to work with API changes across different Spark versions. A better approach, perhaps, is to declare different modules correspond to the Spark versions, and use Maven profile to control which module should be loaded.

|---shims
|---|---3.2
|---|---3.3
|---|---3.4

Describe the potential solution

No response

Additional context

No response

[advancedxy]

I like this idea and was planning to propose something similar.