Add test sharding support for rust_test by brianduff · Pull Request #3774 · bazelbuild/rules_rust

brianduff · 2025-12-12T21:44:20Z

Problem

rust_test targets don't support Bazel's native test sharding. When shard_count is set on a rust_test:

Without --incompatible_check_sharding_support: all tests run N times (once per shard)
With --incompatible_check_sharding_support: the test fails because Rust's libtest harness doesn't read TEST_SHARD_INDEX/TEST_TOTAL_SHARDS environment variables

Solution

This PR adds opt-in sharding support via a new experimental_enable_sharding attribute. When enabled, tests are executed via a wrapper script that:

Detects sharding via TEST_TOTAL_SHARDS environment variable
Touches TEST_SHARD_STATUS_FILE to advertise sharding support to Bazel
Enumerates tests using libtest's --list --format terse flag
Partitions tests by index % TEST_TOTAL_SHARDS == TEST_SHARD_INDEX
Runs only the tests assigned to the current shard using --exact

Usage

rust_test(
    name = "my_test",
    srcs = ["my_test.rs"],
    shard_count = 4,
    experimental_enable_sharding = True,
)

illicitonion

This generally looks good to me (though I don't have enough batch skills to really verify the .bat file). One thought on provider organisation :)

illicitonion · 2025-12-15T21:54:05Z

+
+        # Get the test binary from CrateInfo - it's the output of the compiled test
+        crate_info_provider = None
+        for p in providers:


Let's do something like edit rustc_compile_action to return a dict of provider types to providers, and then do a key lookup, rather than sniffing an ordered list.

much better - done.

…viders

brianduff · 2025-12-16T19:11:04Z

Remaining test breakages appear to be pre-existing issues unrelated to this change.

The windows .bat file was a pain to test, but was able to get it to work eventually. I also have a branch where I use a small rust util to launch the tests - lemme know if we should explore that either here or as a followup.

dzbarsky · 2025-12-24T22:18:14Z

I would love to have the util be a rust wrapper. Seems like that would avoid multiple implementations and a bash dependency

UebelAndre

Once upon a time we had a test wrapper and it ended up being problematic for debuggers. I'm very nervous about introducing this in this way. Is there not a test crate rules_rust could provide that would enable sharding instead?

UebelAndre · 2025-12-25T06:50:06Z

I think for cross-platform consistency, could this be a rust binary?

UebelAndre · 2025-12-25T06:52:14Z

            E.g. `bazel test //src:rust_test --test_arg=foo::test::test_fn`.
        """),
    ),
+    "experimental_enable_sharding": attr.bool(


Should this not be a build flag similar to these?

rules_rust/rust/settings/BUILD.bazel

Lines 85 to 93 in cdaf15f

experimental_link_std_dylib()

experimental_per_crate_rustc_flag()

experimental_use_cc_common_link()

experimental_use_coverage_metadata_files()

experimental_use_global_allocator()

shrey4796 · 2026-03-19T23:27:41Z

Any update on this ? we could really use this feature

bolinfest · 2026-04-02T19:56:50Z

@brianduff @illicitonion good to see you again! We are starting to rely more heavily on Bazel for building https://github.com/openai/codex/ and are very interested in this change.

I would be happy to help shepherd this through if I can!

dzbarsky · 2026-04-02T19:59:28Z

@brianduff @illicitonion good to see you again! We are starting to rely more heavily on Bazel for building https://github.com/openai/codex/ and are very interested in this change.

I would be happy to help shepherd this through if I can!

Our current thinking is the best path forward would be to either patch libtest harness or provide an alternate binary launcher that reuses libtest but adds support for the sharding env vars. (That's basically what Go does as well). Staying in-process should avoid bad interactions with debugger

Port the sharding wrapper feature from bazelbuild#3774 into the hermeticbuild fork. This keeps rustc_compile_action's provider-list API intact and updates rust_test to swap its executable for a generated sharding wrapper when experimental_enable_sharding is set. Co-authored-by: Brian Duff <bduff@linkedin.com> Co-authored-by: Codex <noreply@openai.com>

Port the sharding wrapper feature from bazelbuild#3774 into the hermeticbuild fork. This updates rustc_compile_action to return a dict of named providers so rust_test can replace DefaultInfo without provider-shape sniffing when experimental_enable_sharding is set. Co-authored-by: Brian Duff <bduff@linkedin.com> Co-authored-by: Codex <noreply@openai.com>

Port the sharding wrapper feature from bazelbuild#3774 into the hermeticbuild fork. The implementation wraps rust_test executables when experimental_enable_sharding is set while keeping rustc_compile_action's existing provider-list API for internal and extension callers. rust_test now scans the returned providers to replace DefaultInfo for the wrapper, so extensions such as prost and wasm-bindgen continue to consume rustc_compile_action without API churn. Co-authored-by: Brian Duff <bduff@linkedin.com> Co-authored-by: Codex <noreply@openai.com>

Add test sharding support

94cd8d6

brianduff marked this pull request as draft December 12, 2025 21:46

illicitonion reviewed Dec 15, 2025

View reviewed changes

brianduff force-pushed the push-tusnxzxumrtz branch 2 times, most recently from f6ec6ee to a7f30a5 Compare December 16, 2025 17:47

Refactor rustc_compile_action to return dict of provider types to pro…

2b822a5

…viders

brianduff force-pushed the push-tusnxzxumrtz branch 8 times, most recently from 47f7781 to c23d0dc Compare December 16, 2025 18:56

Fix Windows batch script for test sharding

adb3e34

brianduff force-pushed the push-tusnxzxumrtz branch from c23d0dc to adb3e34 Compare December 16, 2025 19:01

brianduff marked this pull request as ready for review December 16, 2025 19:09

UebelAndre requested changes Dec 25, 2025

View reviewed changes

bolinfest mentioned this pull request Apr 15, 2026

[codex] Add rust_test sharding support hermeticbuild/rules_rust#13

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test sharding support for rust_test#3774

Add test sharding support for rust_test#3774
brianduff wants to merge 3 commits into
bazelbuild:mainfrom
brianduff:push-tusnxzxumrtz

brianduff commented Dec 12, 2025

Uh oh!

illicitonion left a comment

Uh oh!

illicitonion Dec 15, 2025

Uh oh!

brianduff Dec 16, 2025

Uh oh!

brianduff commented Dec 16, 2025

Uh oh!

dzbarsky commented Dec 24, 2025

Uh oh!

UebelAndre left a comment

Uh oh!

UebelAndre Dec 25, 2025

Uh oh!

UebelAndre Dec 25, 2025

Uh oh!

shrey4796 commented Mar 19, 2026

Uh oh!

bolinfest commented Apr 2, 2026

Uh oh!

dzbarsky commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	experimental_link_std_dylib()

	experimental_per_crate_rustc_flag()

	experimental_use_cc_common_link()

	experimental_use_coverage_metadata_files()

	experimental_use_global_allocator()

Conversation

brianduff commented Dec 12, 2025

Problem

Solution

Usage

Uh oh!

illicitonion left a comment

Choose a reason for hiding this comment

Uh oh!

illicitonion Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

brianduff Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

brianduff commented Dec 16, 2025

Uh oh!

dzbarsky commented Dec 24, 2025

Uh oh!

UebelAndre left a comment

Choose a reason for hiding this comment

Uh oh!

UebelAndre Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

UebelAndre Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

shrey4796 commented Mar 19, 2026

Uh oh!

bolinfest commented Apr 2, 2026

Uh oh!

dzbarsky commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants