Support for arm64 simulator builds for iOS

[john-forrest]

I raised this on gitter.im but not sure anyone reads that much now, so excuse the double request. The request is quite simple: has anyone managed to build arm64 binaries for ios simulators, preferable combined ones with x64?

Some of our mobile engineers now have Macs with M1 processors and can't handle the x64 binaries we traditionally send them. From what I can see, I need to build a fat binary where the arm64 binary is built for iphonesimulator, iphoneos. Even creating a polly toolchain where the IPHONEOS_ARCHS is empty does not seem to be supported, but inside hunter there is logic which seems to assume arm architectures mean device and x86 architectures mean simulator. I support a forked hunter anyway, but it feels like I am having to do a lot of change to support this. I am wondering if I have missed a trick or (it feels) nobody has so far tried to do this.

Going forward it seems we need to have two build jobs for iOS: one for device (currently just arm64) and one for simulator (will be both x86 and arm64). (The two arm64 builds are different.) This will be fed into the new xcframework model, where there will be two .a files and not a single one as at present.

[NeroBurner]

I've no experience with IOS or Mac in general. But still have some questions:

• which project(s) are you intending to build?
• does CMake itself support what you need (independently of Hunter)?

[john-forrest]

We pull in quite a few projects but including OpenCV, Boost, protobuf, Jpeg. I could go on. Second question I don't know the answer to but I think the problem is not so much in the cmake packages but those like Boost which are built using other tools.

[drodin]

@john-forrest This issue has nothing to do with Hunter, it's a polly/CMake/Xcode one. AFAIK you won't be able to build combined ios/sim binaries due to outstanding issues in CMake and Xcode. You can build a standalone sim binary with polly:
create a toolchain with empty IPHONEOS_ARCHS and change iphoneos to iphonesimulator on line 10 in polly's os/iphone.cmake
https://github.com/cpp-pm/polly/blob/3a0fd1fc8789f59706108adfae28c1673f8c7335/os/iphone.cmake#L10

You can also build various standalone binaries with another toolchain, like https://github.com/leetal/ios-cmake

[john-forrest]

@drodin it might be wishful thinking on your part that this has nothing to do with hunter but note lines like:

hunter/cmake/projects/Boost/schemes/url_sha1_boost_ios_library.cmake.in

Line 118 in 36588ef

" : <architecture>x86 <target-os>iphone\n"

where simulator is always x86. I found others. As said, the issue in hunter is where the projects are built via autotools or other non-cmake packages. However thanks for the tip in polly. I will try although given the Boost issue I raise, I am slightly sceptical:)

[hjmallon]

@john-forrest : How do you make your iOS builds work? I keep getting the following error when experimenting with really simple projects (i.e. just Hunter, ZLIB and one cpp file). Do you use Polly's ios-nocodesign or other toolchains? Do you have a simple reproducer?

[database is locked Possibly there are two concurrent builds running in the same filesystem location](error: unable to attach DB: error: accessing build database "..snip..": database is locked Possibly there are two concurrent builds running in the same filesystem location.)

@drodin : Are there any CMake upstream issues (perhaps at gitlab.kitware.com) which interested parties should be watching or helping with?

[drodin]

@hjmallon If you want to build for device only, use polly's ios-nocodesign-arm64 (or ios-arm64) toolchain.
The problem is with combined device+simulator build. It depends on arch of your mac and Xcode version. For M1 Air with Xcode 13.4.1, cmake 3.23.3 and polly trunk:

1. if you'll add set(IPHONESIMULATOR_ARCHS "arm64") to ios-nocodesign-arm64.cmake you'll get the following error trying to run your app in simulator: "building for iOS Simulator, but linking in object file built for iOS, for architecture arm64" I have no idea how to fix this this.
2. adding set(IPHONESIMULATOR_ARCHS "x86_64") you'll get "database is locked" error on Xcode > 12.0, because CMake uses Xcode new build system (take a look here https://gitlab.kitware.com/cmake/cmake/-/issues/21282) This can be fixed by switching to legacy build system by adding toolset='buildsystem=1' option to corresponding toolchain in polly's bin/detail/toolchain_table.py, f.e. Toolchain('ios-nocodesign-arm64', 'Xcode', toolset='buildsystem=1', nocodesign=True)
3. if you want to build for arm64 simulator only, set set(IPHONEOS_ARCHS "") and set(IPHONESIMULATOR_ARCHS "arm64") in ios-nocodesign-arm64.cmake and change CMAKE_OSX_SYSROOT from iphoneos to iphonesimulator in polly's os/iphone.cmake on line 10: set(CMAKE_OSX_SYSROOT "iphonesimulator" CACHE STRING "System root for iOS" FORCE)

[hjmallon]

@drodin thanks for getting back to me (I'm interested in getting iOS working for my project but I have little experience with it). I think I have fixed problem 2 here #609

You should be able to test it with the following:

HunterGate(
    URL "https://github.com/hjmallon/hunter/archive/_hm_ios_install_fix.zip"
    SHA1 "dce6d8a2f3f5eedf03116b6fbc90a34329f50d8c"
)

[hjmallon]

It sounds like the wider issue about ARM64 Simulator Combined builds is not yet solved upstream. I personally do not use an Apple Silicon macOS machine, so do not hit the problem (yet...).

My notes:

• Hunter wants the installed libraries to be suitable for all the types of builds possible in the configuration. e.g. if you build for macOS universal 2 it wants all the libraries to be universal 2 so that they can be linked on both ARM64 and x86_64.
• On iOS this has an extra complexity, that people want libraries which are built for not only device archs (arm64) but also simulator archs (x86_64 and now arm64).
• Hunter uses CMAKE_IOS_INSTALL_COMBINED=YES (see https://github.com/cpp-pm/hunter/blob/master/cmake/schemes/url_sha1_cmake.cmake.in#L49) to build libraries where lipo squashes together the device and simulator versions into a fat library (see https://cmake.org/cmake/help/latest/prop_tgt/IOS_INSTALL_COMBINED.html#prop_tgt:IOS_INSTALL_COMBINED)
• With Xcode 'new build system' there is a bug where IOS_INSTALL_COMBINED used nested xcode build commands which it didn't like (database is locked error). CMake fixed this 3.19.5 (see https://cmake.org/cmake/help/latest/release/3.19.html?highlight=3%2019#id5) but you can no longer run cmake --build . --target install, you have to run cmake --install .. Changing to use that in Hunter (see Fix CMAKE_IOS_INSTALL_COMBINED on Xcode 12+ #609) seems to fix this issue.
• However, fat libraries are pretty simple, and are just squashed together by CPU arch. Now that we have two targets iphoneos and iphonesimulator with the same arch arm64 we cannot lipo them together successfully.
• CMake could port everything to use xcframeworks, which are a new type of Apple framework which contains a split by platform as well as fat libraries (so you could have a framework with macOS, iOS, tvOS, simulators side by side). CMake does not supprot creating or linking to these yet (https://gitlab.kitware.com/cmake/cmake/-/issues/21752)
• There may also be another problem with recent Xcode and VALID_ARCHS (https://gitlab.kitware.com/cmake/cmake/-/issues/21636)

[hjmallon]

Sorry I am following this iOS journey myself for a project, so my thoughts might appear slightly scattered, but they are all hopefully relevant. Since we can no longer make successful fat binaries for arm64-device/x86_64-sim/arm64-sim then we need to work out a workaround.

1. Have separate toolchains for iOS device and simulator
2. Maybe we could have separate install dirs for each somehow? (this one is much more work probably)
3. Work upstream in CMake to add static & dynamic xcframework support (which basically makes 'super-fat' structures containing libraries for multiple targets

I don't know which works better for others but I think option 1 seems the cheapest. As you say there are various places in the Hunter codebase which assume things about IPHONEOS_ARCHS / IPHONESIMULATOR_ARCHS (most commonly and incorrectly that arm64 == device). These will have to be fixed.

The easiest way to fix these is probably to try your builds with a toolchain based on ios-nocodesign-arm64.cmake, but with the following patch (to enable arm64/x86_64 simulator builds).

diff --git a/ios-nocodesign-arm64.cmake b/ios-nocodesign-arm64.cmake
index c5a7216..5a2b4df 100644
--- a/ios-nocodesign-arm64.cmake
+++ b/ios-nocodesign-arm64.cmake
@@ -30,9 +30,12 @@ set(CMAKE_MACOSX_BUNDLE YES)
 
 include("${CMAKE_CURRENT_LIST_DIR}/flags/ios_nocodesign.cmake")
 
-set(IPHONEOS_ARCHS arm64)
-set(IPHONESIMULATOR_ARCHS "")
+set(IPHONEOS_ARCHS "")
+set(IPHONESIMULATOR_ARCHS "arm64;x86_64")
 
 include("${CMAKE_CURRENT_LIST_DIR}/compiler/xcode.cmake")
 include("${CMAKE_CURRENT_LIST_DIR}/os/iphone.cmake")
 include("${CMAKE_CURRENT_LIST_DIR}/flags/cxx14.cmake")
+
+set(CMAKE_OSX_SYSROOT "iphonesimulator" CACHE STRING "System root for iOS" FORCE)
+set(CMAKE_XCODE_ATTRIBUTE_ONLY_ACTIVE_ARCH NO CACHE STRING "")

I have found that there is a bug in the default hunter cmake setup with this, patch to follow.

[hjmallon]

PR to fix fat-simulator-only builds for Hunter cmake schemes is here #610. More work will be needed to fix non-CMake projects.

[hjmallon]

@drodin : If you include #610 then most things in Hunter seem to work. For my initial needs this is enough, but I have noticed that all autotools projects and Boost libraries (header only is fine) are broken, at least.

EDIT: I am working on a fix for Boost, do you use any autotools projects?

[NeroBurner]

closing old thread.

Since #780 we generate a osx-15-4-arch-universal2-cxx17 and ios-nocodesign-17-5-arm64-cxx17.

If those don't work or are not enough please send PRs to fix them