CMake: Allow compiling all D modules of a runtime lib at once#2231
CMake: Allow compiling all D modules of a runtime lib at once#2231kinke merged 8 commits intoldc-developers:masterfrom
Conversation
|
@kinke hah, it's nice to see that two separate issues can both be addressed by a simple change! That said, it could be good to split out the object-file-suffix changes into a separate patch, precisely because they address more than just the present issue. What release(s) are you thinking of targeting this towards? |
Yep, I was thinking the same; will happen as part of later PR polishing.
Well separate compilation is not a real issue; If I had a Ryzen CPU with 16 logical cores, I'd most likely still want to compile separately. But for the majority of our CI boxes with 2 CPU cores, all-at-once compilation and some script tweaking could shave off maybe a minute or two and reduce the #lines in the logs. The other motivation behind this approach is that it should be able to hide the template culling issues for Phobos 2.074 (for the normal libs, not the for the unittest libs) and potentially result in higher-performant Phobos due to more redundant template instantiations and so better inline-ability.
I haven't really thought about that. If it works out nicely, the next one (1.4) I guess. |
kinke
force-pushed
the
runtimeCMake
branch
3 times, most recently
from
b265858 to
20efbdf
Compare
That's what I assumed. So, absent some way to tweak the cmake invocation to work around the current build issues, that gives me a target for when I can reintroduce multilib support to the snap package. Thanks! |
|
@joakim-noah: Cross-compiling the testrunner executables is getting in sight. |
kinke
force-pushed
the
runtimeCMake
branch
5 times, most recently
from
00d7919 to
4cb44a8
Compare
|
Oh nice, I'll look at adapting my Android cross-compilation patch to this pull, neither of which use ldc to invoke the linker. As for |
kinke
force-pushed
the
runtimeCMake
branch
2 times, most recently
from
61eb363 to
61fe0e6
Compare
|
There's a new alignment issue on Win64 when linking the testrunners via CMake which I still need to look into. |
|
There's another issue with this PR, Semaphore randomly fails when building Phobos: The randomness is particularly worrying. I quicky looked at |
|
Another annoying issue: the test runner executables aren't relinked if they exist and their tested lib has been updated (due to a changed src module etc.). They are currently linked in manually via linker flags (pull in all objects...) and the build dependency is added via |
|
The Win64 failure was partly due to incremental linking (CMake default) apparently affecting the alignment of globals and this triggering 3 bugs in druntime & Phobos.
|
|
The test runners are now correctly updated. So Semaphore is the last problem, and the failure seems to happen nearly every time now. |
Curious about the timings when building all D modules at once in a single
command-line (like DMD does), I hacked the CMake script and performed a
few tests (Win64, LDC/LLVM with assertions, quad-core CPU without SMT):
All at once, phobos2-ldc: 50.4 secs with -j1, ~1 GB RAM
phobos2-ldc-debug: 14.6 secs with -j1
Separately, phobos2-ldc: 109.5 secs with -j1, 35.1 secs with -j4
phobos2-ldc-debug: 44.8 secs with -j1, 13.1 secs with -j4
So saving all the duplicate parsing work does pay off substantially, at
the cost of much coarser parallelizability and having to recompile all
modules when only a single one changes. Setting the CMake variable
`COMPILE_ALL_D_FILES_AT_ONCE=OFF` restores the previous behavior of
compiling each module separately.
Building druntime (debug+release), Phobos (debug+release) and profile-rt
with `-j4` used to take 60 secs on my box, and now with all-at-once
compilation 50 secs. That meager difference is of course due to the
phobos2-ldc target taking that long; the other CPU cores are idling after
a short while.
I also compiled the Phobos (release) unittests all at once, to check how
much memory that requires. We are talking about 13 GB, so that's not
really an option, so all unittest objects are still compiled separately.
So to take full advantage of this, the `all` target should be built
alongside the unittests, so that all CPU cores have something to do.
---
To get this to work, I had to move from object file suffices (e.g.,
`.../std/math-debug-shared_32.o`) to different base directories, e.g.,
`build-ldc/runtime/objects-debug-shared_32/std/math.o`). This mitigates an
issue due to conflicting object files when building LDC on a 32-bit host
with MULTILIB=ON.
Mostly as preparation for cross-compiling them and to simplify test dependency handling. Also prevents relinking the testrunners on each CTest invokation etc. Instead of adding each unittest lib and each test runner executable as separate test, with correct dependencies between them (even inbetween the unittest libs - shared Phobos depending on static Phobos etc.), I chose to reduce the complexity by only exposing the test runner executables as tests, with no dependencies. This may lead to issues when multiple test runners sharing common build dependencies are built in parallel. But building them this way is not very convenient anyway (e.g., no way to control number of build jobs for each build-testrunner test), so just make sure everything works when running the tests serially via `ctest [-j1]`, and focus more on easily building the test runners externally via a new meta build target `all-test-runners` => `ninja -j4 all all-test-runners && ctest -j4 ...`.
Primarily for cross-compilability. Also add CMake variable `C_SYSTEM_LIBS` to allow the user to specify the names of the C libraries to be linked against shared runtime libs and test runner executables.
Representing a list of * the target's CMAKE_SYSTEM_NAME (Linux, Darwin, Windows, FreeBSD, ...), * and the associated definitions of UNIX, APPLE and MSVC. So the user would use something like `TARGET_SYSTEM=Linux;UNIX` when targeting Linux, `Darwin;APPLE;UNIX` for OSX and `Windows;MSVC` for MSVC targets.
CMake uses incremental linking for MSVC targets by default, with apparent effect on the alignment of globals. A bunch of unittests have been failing on Win64 without these fixes since letting CMake link the test runners. Rainer has fixed the 2 Phobos locations upstream already; the `core.thread` bug is still unfixed, I submitted PR ldc-developers#1888.
std.variant was still failing; I quickly debugged and noted that it uses function addresses as markers and that there's a mismatch there. Turning on identical COMDAT folding for the MS linker, something we do by default when linking via LDC, gets rid of the failure.
add_dependencies() only makes sure the dependencies are up-to-date before running the testrunner's target. So after hacking a source file, the unittest libraries were correctly updated, but the test runners themselves weren't relinked afterwards. target_link_libraries() isn't an option when linking in the library manually via custom linker flags to make sure all objects are pulled in. Fortunately, the LINK_DEPENDS target property allows specifying a file dependency, so that a testrunner is relinked when its library is newer.
|
After merging As both generators failed, I checked the CMake version; Semaphore is using Ubuntu 14.04's 2.8.12.2. Travis is using 3.2.x for Linux. So I switched to latest CMake (currently 3.9) for Semaphore, and it's working now [with ninja, haven't tested with make]. As there's no online documentation for CMake < 3, I'll leave it at that; let's just not use ancient CMake versions for testing. |
|
I've started trying this out: one request I have is that the link.txt for the test runners always puts |
|
I guess the issue is that the related unittest library is linked in manually via linker flags, and that those precede the objects in the CMake linker command line. In that case, the object would need to be specified as first linker flag. So maybe try |
|
That puts the test runner object file first, but then errors out when linking because it's listed twice, clashing symbols. If I remove the test runner object file from |
|
Yeah it's a pain. For POSIX, making the testrunner a custom target with manual linking command |
|
Yeah, that did it, thanks. I'll submit this with my Android PR. |
|
I implemented something in the meantime too, see #2253. |
3 participants
Curious about the timings when building all D modules at once in a single command-line (like DMD does), I hacked the CMake script and performed a few tests (Win64, LDC/LLVM with assertions, quad-core CPU without SMT):
So saving all the duplicate parsing work does pay off substantially, at the cost of much coarser parallelizability and having to recompile all modules when only a single one changes. Setting the CMake variable
COMPILE_ALL_D_FILES_AT_ONCE=OFFrestores the previous behavior of compiling each module separately.Building druntime (debug+release), Phobos (debug+release) and profile-rt with
-j4used to take 60 secs on my box, and now with all-at-once compilation 50 secs. That meager difference is of course due to thephobos2-ldc target taking that long; the other CPU cores are idling after a short while.
I also compiled the Phobos (release) unittests all at once, to check how much memory that requires. We are talking about 13 GB, so that's not really an option, so all unittest objects are still compiled separately.
So to take full advantage of this, the
alltarget should be built alongside the unittests, so that all CPU cores have something to do.To get this to work, I had to move from object file suffices (e.g.,
.../std/math-debug-shared_32.o) to different base directories, e.g.,build-ldc/runtime/objects-debug-shared_32/std/math.o). This mitigates an issue due to conflicting object files when building LDC on a 32-bit host with MULTILIB=ON (@WebDrake).