build: Move shim directories

[kazuyukitanimura]

Which issue does this PR close?

Closes #140

Rationale for this change

To add Spark 3.5 and 4.0 shims later.

What changes are included in this PR?

This PR moves the shims to spark version specific directories

How are these changes tested?

Existing tests

[andygrove]

LGTM. Thanks @kazuyukitanimura

[viirya]

I think the original idea in the ticket is to create different shim for different Spark version instead of using reflection API for all Spark versions?

[kazuyukitanimura]

Thanks @andygrove @viirya

I think the original idea in the ticket is to create different shim for different Spark version instead of using reflection API for all Spark versions?

That's right. Currently it is spark-3.x dir only, but I plan to add spark-4.0 for example. If we want to split spark-3.4 dir out of spark-3.x we can do so as follow up.

[viirya]

That's right. Currently it is spark-3.x dir only, but I plan to add spark-4.0 for example. If we want to split spark-3.4 dir out of spark-3.x we can do so as follow up.

I think the original idea is to have spark-3.2 and spark-3.3 shims which use reflection API, and spark-3.4 which uses Spark API as it is available. So for the Spark versions which have the API available, its shim layer doesn't need to use reflection.

That's what I got from the ticket.

I'm not sure if you are going to do that? If so, then it is fine to have this as initial move.

[kazuyukitanimura]

I'm not sure if you are going to do that? If so, then it is fine to have this as initial move.

My focus would be spark-4.0 without reflection API for now. If anyone is up for taking the task of splitting spark-3.x to 3.2, 3.3, and 3.4, they will be welcomed. Yes this as an initial move.

[advancedxy]

LGTM as an initial move

[viirya]

Merged. Thanks.

[kazuyukitanimura]

Thank you all!