Add support for precompiled headers by randombit · Pull Request #5298 · randombit/botan

randombit · 2026-02-08T15:40:40Z

Stacked on #5297

Tried at MSVC support but it's a quite different model and I got tired of going back and forth with CI. Left to an interested party, hopefully one with a local Windows machine for faster iterations.

On my laptop, building with Clang 21:

3.10.0 release: 3973.09s user 70.02s system 770% cpu 8:44.74 total
master plus Split part of concepts.h into range_concepts.h #5294 + Various changes to reduce header dependencies in TLS #5296: 3238.27s user 95.83s system 770% cpu 7:12.99 total
above plus PCH: 2070.82s user 58.87s system 772% cpu 4:35.66 total

They options are still, for the moment at least, defined in the relative compiler description files, but they aren't used during the build anymore.

randombit · 2026-02-08T15:44:59Z

I experimented a bit with including our own headers also into the PCH and the improvement was at best minor and often regressed vs just using standard library headers. I also ran into some situations where a PCH had an old version of something from Botan itself and the resulting error messages were very hard to interpret, so I'm inclined to avoid unless it really helps.

randombit · 2026-02-08T15:47:49Z

error: default visibility for functions and variables [-fvisibility] differs in PCH file vs. current file

Seems like older Clang doesn't like this? Clang 21 is fine.

Unfortunate, probably suggests there needs to be 2 PCHs one for the library and the other for tests/cli

randombit · 2026-02-08T16:23:08Z

immintrin.h might not be worth it, only difference between pch_lib.h and pch_exe.h is inclusion of that header and the size difference is crazy:

Clang 21: exe 12 Mb, lib 17 Mb
GCC 15: exe 69 Mb, lib 99 Mb (yikes)

randombit · 2026-02-08T16:50:47Z

@copilot Try adding PCH support to the ninja build

Copilot · 2026-02-08T16:50:54Z

@randombit I've opened a new pull request, #5299, to work on those changes. Once the pull request is ready, I'll request review from you.

randombit · 2026-02-08T16:56:02Z

Weird error right in the middle of a build

  fatal error: file '/home/runner/work/botan/botan/build/obj/pch/pch_exe.h' has been modified since the precompiled header './build/obj/pch/pch_exe.h.pch' was built: mtime changed (was 1770567362, now 1770569364)
  note: please rebuild precompiled header './build/obj/pch/pch_exe.h.pch'
  1 error generated.
  make: *** [Makefile:1864: build/obj/cli/utils.o] Error 1

Not sure what to make of this

Possibly an interaction with ccache. The ccache docs say re PCH: If you use Clang, you must compile with -fno-pch-timestamp.

Co-authored-by: randombit <469092+randombit@users.noreply.github.com>

Add Ninja build support for precompiled headers

coveralls · 2026-02-08T18:47:26Z

coverage: 90.079% (+0.009%) from 90.07%
when pulling ade95ff on jack/precompiled-headers
into f708c36 on master.

randombit · 2026-02-08T21:01:14Z

@copilot OK now try to extend to support Microsoft Visual C++ PCH

IIUC you'll have to modify configure to write some little stub cpp files for MSVC to compile to create the PCH

IIUC the relevant options are /Yc, /Yu, and /FI.

Copilot · 2026-02-08T21:01:24Z

@randombit I've opened a new pull request, #5301, to work on those changes. Once the pull request is ready, I'll request review from you.

randombit · 2026-02-09T01:43:18Z

Needs a squash but should otherwise be good to go

Copilot

Pull request overview

Adds build-system support for precompiled headers (PCH) to reduce overall compile times, along with a few related header/include cleanups for SIMD and CLMUL code paths.

Changes:

Introduce a shared PCH header (src/lib/pch/pch.h) and wire it into Makefile and Ninja build generation (including compiler-specific PCH flags).
Extend configuration and cleanup tooling to create/use/remove PCH build artifacts and adjust compiler/build flags to better support PCH sharing.
Refine SIMD-related includes/ISA annotations and fix missing intrinsic header inclusion needed for CLMUL usage.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/scripts/cleanup.py	Cleans PCH artifacts in the build directory
src/lib/utils/simd/simd_4x64/simd_4x64.h	Adds ISA attribute macro to additional members for consistency
src/lib/utils/simd/simd_4x32/simd_4x32.h	Switches SSSE3 include to a narrower intrinsic header
src/lib/utils/simd/simd_2x64/simd_2x64.h	Adjusts SSSE3 includes and adds ISA attributes to more functions
src/lib/utils/ghash/ghash_cpu/polyval_fn.h	Adds `wmmintrin.h` include for CLMUL intrinsic declarations
src/lib/pch/pch.h	New PCH “umbrella” header of common STL headers
src/lib/misc/zfec/zfec_sse2/zfec_sse2.cpp	Narrows intrinsic include to SSE2-only header
src/configs/repo_config.env	Sets `CCACHE_SLOPPINESS` to support PCH + ccache together
src/build-data/ninja.in	Adds PCH build rules and injects PCH usage into compile rules
src/build-data/makefile.in	Adds PCH targets and makes objects depend on generated PCHs
src/build-data/compile_commands.json.in	Removes ISA flags from compilation database entries
src/build-data/cc/{clang,gcc,xcode}.txt	Defines per-compiler PCH compile/include flags and suffixes
doc/dev_ref/contributing.rst	Documents recommended ccache sloppiness settings for PCH
configure.py	Adds `--disable-pch`, generates PCH variables, and adjusts flags to support PCH

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-09T01:49:30Z

src/scripts/cleanup.py

+            for f in os.listdir(build_config['pch_dir']):
+                for suffix in pch_suffixes:
+                    if f.endswith(suffix):
+                        remove_file(os.path.join(build_config['pch_dir'], f))


os.listdir(build_config['pch_dir']) is unguarded. If the directory was removed manually (or was never created due to an interrupted configure), cleanup.py will raise and make clean/ninja clean will fail. Consider checking os.access(pch_dir, os.X_OK) like other cleanup paths, or catching OSError around os.listdir and treating ENOENT as already-clean.

Suggested change

for f in os.listdir(build_config['pch_dir']):

for suffix in pch_suffixes:

if f.endswith(suffix):

remove_file(os.path.join(build_config['pch_dir'], f))

pch_dir = build_config['pch_dir']

if os.access(pch_dir, os.X_OK):

try:

for f in os.listdir(pch_dir):

for suffix in pch_suffixes:

if f.endswith(suffix):

remove_file(os.path.join(pch_dir, f))

except OSError as e:

if e.errno != errno.ENOENT:

logging.error('Failed accessing PCH directory "%s": %s', pch_dir, e)

else:

logging.debug('PCH directory %s was missing', pch_dir)

Copilot · 2026-02-09T01:49:31Z

configure.py


    link_method = choose_link_method(options)

+    if options.enable_pch:


PCH header copying is guarded by options.enable_pch, but the rest of the build templates are guarded by template_vars['enable_pch'] (which also checks cc.supports_pch()). On unsupported compilers this will still create ${build}/obj/pch and copy pch_*.h, but no build rules will consume them and cleanup.py won't remove them (since pch_dir isn't in build_config). Consider keying this on template_vars['enable_pch'] to keep the filesystem outputs consistent with the generated build rules.

Suggested change

if options.enable_pch:

if template_vars.get('enable_pch'):

Copilot · 2026-02-09T01:49:31Z

src/build-data/ninja.in

 %{for lib_build_info}
-build %{obj}: compile_lib %{src}
-  isa_flags = %{isa_flags}
+build %{obj}: compile_lib %{src} | %{pch_target}
 %{endfor}

 %{for cli_build_info}
-build %{obj}: compile_exe %{src}
+build %{obj}: compile_exe %{src} | %{pch_target}
 %{endfor}

 %{for test_build_info}
-build %{obj}: compile_exe %{src}
+build %{obj}: compile_exe %{src} | %{pch_target}
 %{endfor}


When precompiled headers are disabled/unsupported, pch_target expands to an empty string, leaving a dangling | in lines like build %{obj}: ... | %{pch_target}. This produces an invalid build.ninja and will break e.g. MSVC/clangcl builds where enable_pch becomes false. Emit the order-only dependency only when enable_pch is true (or include the leading | in the variable so it can be empty safely).

Copilot · 2026-02-09T01:49:31Z

src/build-data/ninja.in

  deps = %{ninja_header_deps_style}
 %{endif}
-  command = %{cxx} ${ABI_FLAGS} ${LANG_FLAGS} ${CXXFLAGS} -DBOTAN_IS_BEING_BUILT ${WARN_FLAGS} ${isa_flags} %{public_include_flags} %{internal_include_flags} %{external_include_flags} %{header_deps_flag} %{header_deps_out|concat: $out.d} %{dash_c} $in %{dash_o}$out
+  command = %{cxx} %{pch_include_for_exe} ${ABI_FLAGS} ${LANG_FLAGS} ${CXXFLAGS} -DBOTAN_IS_BEING_BUILT ${WARN_FLAGS} %{public_include_flags} %{internal_include_flags} %{external_include_flags} %{header_deps_flag} %{header_deps_out|concat: $out.d} %{dash_c} $in %{dash_o}$out


compile_exe now injects %{pch_include_for_exe}. However some Ninja build edges using compile_exe (e.g. fuzzers, bogo_shim_object, ct_selftest_object) don't have an order-only dependency on the PCH build. In parallel builds this can lead to nondeterministic PCH usage and, in the worst case, attempting to read a partially-written PCH file. Add the same | pch order-only dependency for all compile_exe invocations when enable_pch is enabled (or ensure these targets don't use the PCH include flag).

Suggested change

command = %{cxx} %{pch_include_for_exe} ${ABI_FLAGS} ${LANG_FLAGS} ${CXXFLAGS} -DBOTAN_IS_BEING_BUILT ${WARN_FLAGS} %{public_include_flags} %{internal_include_flags} %{external_include_flags} %{header_deps_flag} %{header_deps_out|concat: $out.d} %{dash_c} $in %{dash_o}$out

command = %{cxx} ${ABI_FLAGS} ${LANG_FLAGS} ${CXXFLAGS} -DBOTAN_IS_BEING_BUILT ${WARN_FLAGS} %{public_include_flags} %{internal_include_flags} %{external_include_flags} %{header_deps_flag} %{header_deps_out|concat: $out.d} %{dash_c} $in %{dash_o}$out

randombit added 4 commits February 8, 2026 09:56

Don't use ISA flags during compilation anymore

c5aaf01

They options are still, for the moment at least, defined in the relative compiler description files, but they aren't used during the build anymore.

Add missing ISA annotations to SIMD_2x64

d9e5c84

Add missing FN_ISA annotations to SIMD_4x64

aff0275

Add support for precompiled headers with GCC and Clang

5064348

This comment was marked as outdated.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Macro gate immintrin.h

26760ff

This comment was marked as outdated.

Sign in to view

randombit added 3 commits February 8, 2026 10:59

Ignore array-bounds

0033983

Ccache config

c86d8e8

Split PCH for lib vs exe

f48fb76

fix

5d54263

Initial plan

4a8069e

Copilot AI mentioned this pull request Feb 8, 2026

Add Ninja build support for precompiled headers #5299

Merged

clang-cl needs something different

0137ea7

Copilot AI and others added 7 commits February 8, 2026 16:56

Add PCH support to Ninja build system

5e7f02d

Co-authored-by: randombit <469092+randombit@users.noreply.github.com>

Use -fno-pch-timestamp. Disable for emcc pending testing

d10401c

mmintrin

0eb8c8a

Fix makefile rebuilds

24d563e

Merge pull request #5299 from randombit/copilot/sub-pr-5298

4c9547c

Add Ninja build support for precompiled headers

ws

457d5cd

f

32f5c6a

Share the PCH file

8b795d0

Copilot AI mentioned this pull request Feb 8, 2026

Add MSVC precompiled header support #5301

Merged

randombit added 2 commits February 8, 2026 17:04

Clean up pch

21988d7

MSVC attempt

b86bb92

randombit force-pushed the jack/precompiled-headers branch from 1a51c2c to b86bb92 Compare February 8, 2026 22:05

randombit added 15 commits February 8, 2026 17:09

wip

5f44606

fixes

f4ccaff

configure

2165e63

guard

b16983f

write pch cpp

b99c9c7

nina

5602fa2

remove wtf

43a4363

ninja fix

57c3626

shut it bill

d93d667

remove unused phony target

544a5b7

yolo

8c614b1

hack hack hack

3ef3d15

fuck it

35765ab

wtf

b0cb7ac

ah

ade95ff

randombit requested a review from Copilot February 9, 2026 01:42

randombit marked this pull request as ready for review February 9, 2026 01:43

Copilot started reviewing on behalf of randombit February 9, 2026 01:43 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

-            for f in os.listdir(build_config['pch_dir']):
-                for suffix in pch_suffixes:
-                    if f.endswith(suffix):
-                        remove_file(os.path.join(build_config['pch_dir'], f))
+            pch_dir = build_config['pch_dir']
+            if os.access(pch_dir, os.X_OK):
+                try:
+                    for f in os.listdir(pch_dir):
+                        for suffix in pch_suffixes:
+                            if f.endswith(suffix):
+                                remove_file(os.path.join(pch_dir, f))
+                except OSError as e:
+                    if e.errno != errno.ENOENT:
+                        logging.error('Failed accessing PCH directory "%s": %s', pch_dir, e)
+            else:
+                logging.debug('PCH directory %s was missing', pch_dir)


		link_method = choose_link_method(options)

		if options.enable_pch:

	command = %{cxx} %{pch_include_for_exe} ${ABI_FLAGS} ${LANG_FLAGS} ${CXXFLAGS} -DBOTAN_IS_BEING_BUILT ${WARN_FLAGS} %{public_include_flags} %{internal_include_flags} %{external_include_flags} %{header_deps_flag} %{header_deps_out\|concat: $out.d} %{dash_c} $in %{dash_o}$out
	command = %{cxx} ${ABI_FLAGS} ${LANG_FLAGS} ${CXXFLAGS} -DBOTAN_IS_BEING_BUILT ${WARN_FLAGS} %{public_include_flags} %{internal_include_flags} %{external_include_flags} %{header_deps_flag} %{header_deps_out\|concat: $out.d} %{dash_c} $in %{dash_o}$out

Comments

Conversation

randombit commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

randombit commented Feb 8, 2026

Uh oh!

randombit commented Feb 8, 2026

Uh oh!

This comment was marked as outdated.

This comment was marked as off-topic.

This comment was marked as outdated.

randombit commented Feb 8, 2026

Uh oh!

randombit commented Feb 8, 2026

Uh oh!

Copilot AI commented Feb 8, 2026

Uh oh!

randombit commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

randombit commented Feb 8, 2026

Uh oh!

Copilot AI commented Feb 8, 2026

Uh oh!

randombit commented Feb 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

randombit commented Feb 8, 2026 •

edited

Loading

randombit commented Feb 8, 2026 •

edited

Loading

coveralls commented Feb 8, 2026 •

edited

Loading