[SR-14035] Compiler toolchain build error on M1 mac

[johnno1962]

Previous ID	SR-14035
Radar	rdar://problem/73014167
Original Reporter	@johnno1962
Type	Bug

Additional Detail from JIRA

Votes	0
Component/s	Compiler, Project Infrastructure
Labels	Bug
Assignee	@shahmishal
Priority	Medium

md5: 10a69735b885909bbf12d3e950153532

Issue Description:

HI Apple,

I've tried any number of things but have been unable to build a toolchain using swift/main & Xcode 12.3 and the swift/utils/build-toolchain script. It seems to be something to do with supporting multiple architectures for macOS and simulator w/lipo. This warning may be relevant:

warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: lib/swift/watchsimulator/i386/libswiftCompatibility51.a will be fat and ar(1) will not be able to operate on it

[typesanitizer]

@swift-ci create

[johnno1962]

Still unable to build a toolchain on an M1 using a fresh clone. Error messages include:

FAILED: lib/swift/macosx/libswiftCompatibilityDynamicReplacements.a
cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/CompatibilityDynamicReplacements && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibilityDynamicReplacements.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibilityDynamicReplacements.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibilityDynamicReplacements.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibilityDynamicReplacements.a
fatal error: /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibilityDynamicReplacements.a and /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibilityDynamicReplacements.a have the same architectures (arm64) and can't be in the same fat output file
[22/1588][ 1%][1.247s] cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/Compatibility50 && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibility50.a
FAILED: lib/swift/macosx/libswiftCompatibility50.a
cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/Compatibility50 && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibility50.a
fatal error: /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility50.a and /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility50.a have the same architectures (arm64) and can't be in the same fat output file
[23/1588][ 1%][1.256s] cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/Compatibility51 && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibility51.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility51.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility51.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibility51.a
FAILED: lib/swift/macosx/libswiftCompatibility51.a

[johnno1962]

I've finally been able to build a toolchain for myself on an M1 Mac. It took a good while 🙂. There are about 4 problems that need to be addressed by someone who know more about CMake than I do. The first is the error above where lipo fails when trying to merge the archives for the three architectures x86_64, arm64 and arm64e as they all seem to have been built for arm64. It's odd that this problem should be specific to build on an arm64 machine as I have built "fat" toolchains recently on an Intel machine but I couldn't investigate it any further. In the end I changed SUPPORTED_OSX_ARCHS in the file cmake/modules/DarwinSDKs.cmake to only include arm64 and not attempt a "fat" build (besides it takes a good deal less time!)

The next problem seems to be related to llvm where you get an undefined symbol for ___isOSVersionAtLeast in the source file stdlib/public/runtime/ImageInspectionMachO.cpp. I tried various things but in the end defined my own version of this function returning true in this file.

The third relatively minor problem is with the Swift package manager where I have filed a straightforward PR swiftlang/swift-package-manager#3500 and, with fixes for these problems the toolchain build proceeds most of the way through until you get to another place where you have to provide a symbol ___isOSVersionAtLeast in indexstore-db/lib/LLVMSupport/Support/Path.cpp and you will be rewarded with a toolchain (if not the symbols archive.)

As you can see these are relatively minor problems which someone with knowledge of the CMake files could make easy work of. I'm not sure what the cause of the llvm problem would be. I'm amazed nobody on the compiler team has made the switch to M1 yet as they are just so blazingly fast – or do they know there is something even better on the horizon 🙂.

[johnno1962]

Any ideas @compnerd?

[johnno1962]

I've preformed a new clone and replicated these changes though the following were also necessary:

The missing symbol has changed in the last month from _*isOSVersionAtLeast to *_isPlatformVersionAtLeast

So, in the interim, I've added to swift/stdlib/public/runtime/ImageInspectionMachO.cpp:

extern "C" bool __isPlatformVersionAtLeast();
bool __isPlatformVersionAtLeast()}}{{{}}
{{ return true;}}
{{}}}

You need to copy build/buildbot_osx/lldb-macosx-arm64/bin/lldb-tblgen from an x86_64 build for lldb to build.

The benchmarks fail to run with a undefined swift symbol that doesn't demangle so I've had to:

dyld: Symbol not found: _SayxSicig16_Differentiation14DifferentiableRzlWJrUSpSr

cat >build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Onone

#!/bin/bash

exit 0

^D

cp build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Onone build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_O

cp build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Onone build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Osize

[johnno1962]

The complete diff for the various changes I had to make to get a toolchain were:

diff --git a/benchmark/scripts/build_script_helper.py b/benchmark/scripts/build_script_helper.py  
index 53bf7b19f68..8c9f8a5d995 100755

-   -   -   a/benchmark/scripts/build_script_helper.py  
            +++ b/benchmark/scripts/build_script_helper.py  
            @@ -54,10 +54,11 @@ def main():  
            os.makedirs(bin_dir)

swiftbuild_path = os.path.join(args.toolchain, "usr", "bin", "swift-build")

-   perform_build(args, swiftbuild_path, "debug", "Benchmark_Onone", "-Onone")

-   perform_build(args, swiftbuild_path, "release", "Benchmark_Osize", "-Osize")

-   perform_build(args, swiftbuild_path, "release", "Benchmark_O", "-O")  
    +# perform_build(args, swiftbuild_path, "debug", "Benchmark_Onone", "-Onone")  
    +# perform_build(args, swiftbuild_path, "release", "Benchmark_Osize", "-Osize")  
    +# perform_build(args, swiftbuild_path, "release", "Benchmark_O", "-O")

if **name** == "**main**":  
+ exit(0)  
main()  
diff --git a/benchmark/scripts/test_Benchmark_Driver.py b/benchmark/scripts/test_Benchmark_Driver.py  
index 7690aacf31b..b18f0d1147d 100644

-   -   -   a/benchmark/scripts/test_Benchmark_Driver.py  
            +++ b/benchmark/scripts/test_Benchmark_Driver.py  
            @@ -1071,4 +1071,5 @@ class TestBenchmarkDoctor(unittest.TestCase):

if **name** == "**main**":  
+ exit(0)  
unittest.main()  
diff --git a/cmake/modules/DarwinSDKs.cmake b/cmake/modules/DarwinSDKs.cmake  
index 98b79e1f279..73813d48ca9 100644

-   -   -   a/cmake/modules/DarwinSDKs.cmake  
            +++ b/cmake/modules/DarwinSDKs.cmake  
            @@ -14,7 +14,7 @@ set(SUPPORTED_TVOS_ARCHS "arm64")  
            set(SUPPORTED_TVOS_SIMULATOR_ARCHS "x86_64;arm64")  
            set(SUPPORTED_WATCHOS_ARCHS "armv7k;arm64_32")  
            set(SUPPORTED_WATCHOS_SIMULATOR_ARCHS "i386;x86_64;arm64")  
            -set(SUPPORTED_OSX_ARCHS "x86_64;arm64;arm64e")  
            +set(SUPPORTED_OSX_ARCHS "arm64")

is_sdk_requested(OSX swift_build_osx)  
if(swift_build_osx)  
diff --git a/stdlib/public/runtime/ImageInspectionMachO.cpp b/stdlib/public/runtime/ImageInspectionMachO.cpp  
index b21ca9d0e05..bd953673e30 100644

-   -   -   a/stdlib/public/runtime/ImageInspectionMachO.cpp  
            +++ b/stdlib/public/runtime/ImageInspectionMachO.cpp  
            @@ -172,5 +172,10 @@ int swift::lookupSymbol(const void \*address, SymbolInfo \*info) {  
            return 1;  
            }

+extern "C" bool \_\_isPlatformVersionAtLeast();  
+bool \_\_isPlatformVersionAtLeast() {  
+ return true;  
+}  
+  
\#endif // defined(**APPLE**) && defined(**MACH**) &&  
// !defined(SWIFT_RUNTIME_MACHO_NO_DYLD)

   

[edymtt]

The lipo issue should be addressed by the following PRs

[5.5] #38445
[main] #38415

[johnno1962]

Thanks very much Eric

[johnno1962]

A year on, I've just tried again and basically utils/build-toolchain basically no longer works (on an M1 at least). The issues I mentioned above are still there along with a few new undefined symbols related to half-floats and the bootstrapping process to bring Swift sources into the compiler doesn't seem to work, failing to find a usable version of swiftc. I don't have an intel machine to test if it "just works" on that architecture.

[shahmishal]

@edymtt Looks like you had a fix for this previously, I wonder if something changed recently.

[johnno1962]

edymtt's fix made a big difference but there are many other issues, at least on my machine. My clone is from a week ago.

[johnno1962]

An update on building a toolchain on M1, I've been able to get the build past the previous road blocks I mentioned by performing the whole build under rosetta!

I've not been able to get a toolchain just yet as it trips up at the very last trying to build swift-driver multi-architecture but it does seem to indicate this isn't a holding it wrong issue or specific to my particular clone. The problem is very easy to replicate if you run build-toolchain on an M1. I'm sure someone with the right llvm and CMake knowledge could clear this up in an instant.