[4/5] support C++20 Modules, support one phase compilation

[PikachuHyA]

I split the XXL PR #19940 into several small patches.
This is the fourth patch of Support C++20 Modules, which supports one phase compilation.

Changes of build graph

First, files are scanned, resulting in the creation of .ddi files. If the compiler being used is Clang, the tool clang-scan-deps is employed.

Next, the generated .ddi files are aggregated into a .CXXModules.json file using the aggregate-ddi tool.

Following this, the .CXXModules.json file, along with the .ddi files, is used to generate .modmap files through the generate-modmap tool.

Finally, the source files are compiled into object files. If a file is a module interface, a corresponding module file is produced as a byproduct.

The following diagram illustrates the scanning process.

              ┌────────┐                  ┌────────┐                  ┌────────┐
              │foo.cppm│                  │bar.cppm│                  │main.cc │
              └────┬───┘                  └────┬───┘                  └────┬───┘
                   │  c++-module-deps-scanning │                           │
                   │                           │                           │
              ┌────▼───┐                  ┌────▼───┐                  ┌────▼───┐
              │foo.ddi │                  │bar.ddi │                  │main.ddi│
              └────┬───┘                  └────┬───┘                  └────┬───┘
                   │                           │                           │
                   └───────────────────────────┼───────────────────────────┘
                                               │aggregate-ddi
                                      ┌────────▼───────────┐
                                      │demo.CXXModules.json│
                                      └────────────────────┘

The following diagram illustrates the compile process of foo.cppm.

                                                    ┌────────────────────┐
                                                    │demo.CXXModules.json│
                                                    └─────┬──────────────┘
                                                          │
                                                          │
                                        ┌───────┐         │          ┌───────────┐
                                     ┌─►│foo.ddi├─────────┤          │ bar.cppm  │
                                     │  └───────┘         │          └─────┬─────┘
                                     │                    │                │
                                     │                    │                │
                                     │                    │                │
            c++-module-deps-scanning │                    │                │
      ┌────────┐                     │  ┌───────┐   ┌─────▼────┐      ┌────▼─────┐
      │foo.cppm├─────────────────────┴─►│foo.d  │   │foo.modmap│      │ bar.pcm  │
      └───┬────┘                        └──┬────┘   └────┬─────┘      └────┬─────┘
          │                                │             │                 │
          │                                │             │                 │
          │◄───────────────────────────────┴─────────────┴─────────────────┘
          │ c++20-module-compile
 ┌────────▼─────────┐
 │ foo.pcm && foo.o │
 └──────────────────┘

Changes of compile

I modified the CppCompileAction to include the relevant context, inputs, and outputs.

Context: Module files and .CXXModules.json files
Inputs: Module files, .modmap file, and .modmap.input file
Outputs: Module files and .CXXModules.json files
The primary difference in CppCompileAction is the addition of the computeUsedCpp20Modules function.

The restart mechanism is utilized; if the necessary C++20 Modules files are not ready during compilation, the current compilation will exit and wait for an appropriate time to recompile.

[peakschris]

@katre @lberki I'm not sure if you got the notification, but ping... If this PR [4/5] can be merged, we can start using C++ modules with bazel... in eager anticipation :-)

[katre]

Sorry, I didn't notice this because I don't know anything about C++ modules. Re-assigning to @comius, apologies for not noticing this earlier.

[PikachuHyA]

I'm testing this patch. If this patch ready, I will remove the [WIP] prefix.

[PikachuHyA]

@comius @trybka
please review this patch.

[peakschris]

@comius @trybka, friendly ping!

[PikachuHyA]

Synced up with @comius and @pzembrod this morning. I think there will hopefully be a lull in the Java->Starlark migration next week which seems like a good opportunity to try to land this again.

Can you resolve the current conflicts and I will begin an import on Monday/Tuesday?

cc: @fmeum

I will update the PR as soon as possible.

[PikachuHyA]

CI is failing with Error in fail: the ‘generate_modmap’ tool is not defined in the C++ toolchain. This issue is addressed by bazelbuild/rules_cc#496 . We need to move to a rules_cc version that includes bazelbuild/rules_cc#496 to resolve the error.

[fmeum]

@PikachuHyA rules_cc v0.2.9 is out now

[PikachuHyA]

upgraded the rules_cc and sent #27139

The macOS CI failed, so I skiped it.

@trybka @comius @fmeum

[PikachuHyA]

Synced up with @comius and @pzembrod this morning. I think there will hopefully be a lull in the Java->Starlark migration next week which seems like a good opportunity to try to land this again.

Can you resolve the current conflicts and I will begin an import on Monday/Tuesday?

cc: @fmeum

@trybka ping

[trybka]

Hi there,

Began the import earlier this week, just re-pulled based on your commits from yesterday. Currently fixing up some internal tests and then sending out for final review. 🤞

[trybka]

Just a quick status update: as you can see there are now a number of merge conflicts. I'm resolving those in my local branch to ease the back and forth. Had to fix a few more tests and the automatic merge resolution had some fun errors.

[trybka]

I've done my best to resolve merge conflicts but I have a few test failures, @PikachuHyA @fmeum can you take a look?

https://buildkite.com/bazel/google-bazel-presubmit/builds/97272

(should be able to view the current state in https://bazel-review.googlesource.com/c/bazel/+/294090 ? )

[fmeum]

@trybka With this diff

--- a/src/main/java/com/google/devtools/build/lib/rules/cpp/CcStarlarkInternal.java
+++ b/src/main/java/com/google/devtools/build/lib/rules/cpp/CcStarlarkInternal.java
@@ -695,6 +695,9 @@ public class CcStarlarkInternal implements StarlarkValue {
       boolean needsIncludeValidation,
       String toolchainType)
       throws EvalException, TypeException {
+    if (modmapInputFile == null || modmapInputFile == Starlark.NONE) {
+      throw new EvalException("modmap_input_file parameter is required");
+    }
     CoptsFilter coptsFilter =
         createCoptsFilter(
             Starlark.isNullOrNone(coptsFilterObject) ? null : (String) coptsFilterObject);

I get a stack trace in rules_cc. Is it possible that the same patch also has to be applied to and released with rules_cc? I have to admit that I don't fully understand the split between Bazel builtins and rules_cc anymore.

Edit: The stack trace:

ERROR: /private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/workspace/BUILD.bazel:19:10: in cc_binary rule //:main: 
Traceback (most recent call last):
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/rules_impl/cc_binary.bzl", line 813, column 44, in _impl
		binary_info, providers = cc_binary_impl(ctx, [])
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/rules_impl/cc_binary.bzl", line 518, column 67, in cc_binary_impl
		(compilation_context, compilation_outputs) = cc_common.compile(
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/cc_common.bzl", line 537, column 19, in _compile
		return compile(
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/compile/compile.bzl", line 338, column 31, in compile
		_create_cc_compile_actions(
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/compile/compile.bzl", line 601, column 53, in _create_cc_compile_actions
		_create_pic_nopic_compile_source_actions(
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/compile/compile.bzl", line 821, column 53, in _create_pic_nopic_compile_source_actions
		nopic_object = _create_compile_source_action(
	File "/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/sandbox/darwin-sandbox/4337/execroot/_main/_tmp/01b593ee1908b4aeba024c8f2007c393/root/5391ae7b4c966eb32ca7d54a9a74ab30/external/rules_cc+/cc/private/compile/compile.bzl", line 1018, column 42, in _create_compile_source_action
		_cc_internal.create_cc_compile_action(
Error in create_cc_compile_action: modmap_input_file parameter is required

[trybka]

Yes, I think you're right. For now I will disable the test until this rolls out into rules_cc.

[PikachuHyA]

I've done my best to resolve merge conflicts but I have a few test failures, @PikachuHyA @fmeum can you take a look?

https://buildkite.com/bazel/google-bazel-presubmit/builds/97272

(should be able to view the current state in https://bazel-review.googlesource.com/c/bazel/+/294090 ? )

@trybka Thanks for your work.

I have a basic question. I’m not familiar with Gerrit. How can I check out the change at https://bazel-review.googlesource.com/c/bazel/+/294090 on my development machine, including fetching your updated patchsets?

Also, regarding the merge conflicts and failing tests: should I resolve the GitHub PR conflicts locally and push the updates to this PR?

[trybka]

@PikachuHyA

From a bazelbuild/bazel git checkout, I think you can just do this:
git fetch https://bazel.googlesource.com/bazel 77941d369dbc0bbe674514c7dc66f50e3d2caac0 && git checkout FETCH_HEAD

(I found the commit hash on the CI status page)

[trybka]

Also, regarding the merge conflicts and failing tests: should I resolve the GitHub PR conflicts locally and push the updates to this PR?

No, I have a pile of merge conflicts already in my local copy. FWICT the current test failure is just because it's not in rules_cc yet, so we're going to disable it for now. I think that's the only failing test at the moment.

[fmeum]

How can I check out the change at https://bazel-review.googlesource.com/c/bazel/+/294090 on my development machine, including fetching your updated patchsets?

There is a "Download" button at the top of the file list that provides you with with various git commands you can run to get the change.

I think that's the only failing test at the moment.

It's also the only one :-) But we can easily reenable it and if this PR doesn't break any other tests, we can merge now and do that later.

[jwhpryor]

For those of us anxiously following from the side lines, does this mean C++ modules are now supported? I'm unsure where the 5/5 of bug resides, or whether this is actually all that's needed (or if this is being rolled out in a downstream bazel release).

Thank you all for your hard work!

[fmeum]

Yes, C++20 modules are supported with last_green and will be supported by Bazel 9, with the following caveats:

• As of this commit you would still need to use an override to depend on the latest commit of rules_cc, which is not part of a release yet. Edit: You also need to patch in Fix C++20 module compilation rules_cc#511.
• Two-phase compilation (the 5/5 part) is not yet supported. I am positive that this can be cherry-picked into a later 9.x minor release.
• Each cc_library so far only accepts a single module interface. I will try to land a follow-up change that removes this limitation, but that may also have to wait for one of the 9.x releases.

@PikachuHyA please correct me if I missed anything. And thanks again for the contribution!

[PikachuHyA]

For those of us anxiously following from the side lines, does this mean C++ modules are now supported? I'm unsure where the 5/5 of bug resides, or whether this is actually all that's needed (or if this is being rolled out in a downstream bazel release).

Thank you all for your hard work!

@jwhpryor Thanks for following this work.

With this PR merged, Bazel now offers limited support for C++20 Modules. The current limitation is that each cc_binary/cc_library's module_interfaces attribute may contain only a single module interface file (e.g. foo.cppm). This limitation can be removed after #27492 merged (special thanks to @fmeum ).

The work in #22555 will not block anyone who wants to use C++20 Modules with Bazel. I will update that PR later. Currently only Clang supports two-phase compilation; GCC/MSVC do not yet.

Our next priorities include broader module features (e.g. support for import std), and improving GCC/MSVC support. I'll be working on those items.

[lberki]

I have to rain on your parade here, I'm afraid: in my capacity as the person whose job this week is to keep our internal builds running, http://www.github.com/bazelbuid/bazel/pull/22553 breaks a few things. I asked @comius to investigate and I hope it's just some tiny little snag, but for now, that's just a hope and I can't exclude the possibility that a significant amount of work will still need to be done (and I don't know yet where, either)

[peakschris]

@PikachuHyA is there a hello_world example available showing use of C++20 modules and shared libraries?

I found the disabled cc_integration_test and the attribute docstrings but couldn't figure it out from there.

[PikachuHyA]

@PikachuHyA is there a hello_world example available showing use of C++20 modules and shared libraries?

I found the disabled cc_integration_test and the attribute docstrings but couldn't figure it out from there.

@peakschris
sorry for late reply.
I added a repo demonstrating how to use C++20 Modules.
see https://github.com/PikachuHyA/bazel_cxx20_modules_demo/