[CI] Continuous integration on Apple M1 architecture

[asfimport]

It would be nice if we had some confidence that our next major release runs on Apple Silicon. I am looking at hooking up an M1 Mac Mini to Buildkite so that we are able to run CI jobs on one. If anyone else would like to contribute a machine to the build cluster, please be our guest

Reporter: Wes McKinney / @wesm

Subtasks:

• [CI][C++] Implement CI on Apple M1 for C++
• [CI][R] Implement CI on Apple M1 for R

Related issues:

• [Python][Packaging] Wheel builds for Apple Silicon (is related to)

_{Note: This issue was originally created as ARROW-10657. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
cc @kszucs

[asfimport]

Antoine Pitrou / @pitrou:
Also cc @amol-

[asfimport]

Krisztian Szucs / @kszucs:
We continuous build and test wheels for M1 using crossbow and github actions, so we can have confidence that both libarrow and pyarrow works on apple silicon.

Of course we should configure additional builds for better maintainability.

[asfimport]

Jacob Wujciak / @assignUser:
@kszucs should we add an M1 test for all components?

[asfimport]

Krisztian Szucs / @kszucs:
Eventually yes, but we should probably turn this jira into an umbrella issue with multiple subtasks. Starting with a C++ builds would be great.

[asfimport]

Antoine Pitrou / @pitrou:
Right, C++ is the most important, and then probably Python and R. C/GLib is probably a later option (also cc @kou for that).

[asfimport]

Jacob Wujciak / @assignUser:
I have moved this to 9.0.0 and added the sub task @pitrou I skipped python as the wheel build (see Krisztian's comment) should cover that  right?

[asfimport]

Antoine Pitrou / @pitrou:
Indeed, the wheel build should cover it (though ideally we would have a CI build on PRs as well: there is already a "macOS Python" CI build but that's presumably on x86).

[asfimport]

Krisztian Szucs / @kszucs:
M1 builds on PRs is going to be more complicated since we'd need to configure self-hosted runners for apache/arrow involving apache infra (though certainly doable). Currently the M1 is configured for the crossbow repository.

[raulcd]

Should we close this and maybe add a specific task to add M1 runners for PRs? @assignUser

[assignUser]

Yes, I think we can close #31591 too as we run the source verification on m1 and that builds C++ and the wheels build C++ and the r packages build C++ on m1...

I don't see self-hosted M1 as feasible for PRs as we can not make them ephemeral as of yet.