Split installation of sdist providers into parallel chunks

[potiuk]

Sdist provider installation takes a lot of time because pip cannot parallelise the sdist package building. But we still want to test all our provider's installation as sdist packages.

This can be achieved by running N parallel installations with only subset of providers being installed in each chunk.

This is what we do in this PR.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[potiuk]

Not stale

[potiuk]

This one speeds up one of the longest CI jobs we have on main / self-hosted-runners from 26 minutes to 6 minutes (whoping 5x faster). The `sdist' packages were only instaleld (just for regression) already - we did not run import verification on them (we run import verification for .whl packages only).

Installing sdist packages has been always super-slow (and recent pip changes made it even slower - because now they always go sdist -> whl - > install and the installation cannot be parallelised to use multiple CPUS.

However - since we only want to see if provider sdist packages can be installed, we can parallelise the installation ourselves - by splitting the list of providers into chunks, and running separate docker container in parallel for each chunk.

This way we can install all sdist packages 5 times faster on seff-hosted machine with 8 CPUS (it's not 8x faster because of some common overhead repeated for each one - we install airflow from sdist package in each of the containers.

This job is only running for self-hosted runners, so it will have no impact on non-committer PRS (but when we will speed up the remaning Helm Unit test that is very slow as well, this will give faster feedback for commiter PRs and faster builds for main "canary" builds (and some 20 minutes build-time cost for those.

[potiuk]

This is how it will look like (https://github.com/apache/airflow/actions/runs/4714228972/jobs/8360483236):

[potiuk]

We also need this one to get feedback improvements (and cc: @hussein-awala - to your doubts from #30672 , this one has only positive impact - it just uses mulitple CPUS of our self-hosted runners much more effectively) we get close to 100% utilisation thanks to splitting sdist installation into chunks and parallelising them in multiple containers. Pretty much same "build time" but with 5x decrease of "elapsed time".

[potiuk]

Would apreciate to get that one merged. Tt will speed up some of the main builds and constraint refreshing, I think sdist and #30672 are the ones that delay (and cancel completion) of quite a number of those complete "canary" builds (because the jobs are cancelled by subsequent merges), and merging those will make the refreshing our cache and constraints more often.

[potiuk]

Last speedup for CI in the current series.

[potiuk]

Conflict after merging #30705 resolved.

[potiuk]

Reaaly nice speedup: from 20 to 6 mins without any other impact (just better use of multiple CPUS). I hope to get a quick review on that one :)

[hussein-awala]

looks good for me, LGTM