venv_site_packages broken with the combination: py_binary from dependency bazel module + pip wheel with namespace import path

[shayanhoshyari]

🐞 bug report

Affected Rule

py_binary

Is this a regression?

Yes, was not present in 1.7.0

Description

If you have this combination:

• A dependency module, e.g. rules_zoo which has a py_binary, e.g. @rules_zoo//tool
• @rules_zoo//tool uses a pip dependency that has namespace modules in its import path, e.g. @pip_rules_zoo//google_cloud_storage which has google.cloud.storage as python import and google.cloud is a namespace import.
• Try to use rules_zoo in another root bazel module (e.g. just with bazel run @rules_zoo//tool, or when @rules_zoo//tool is used in an action in a rule that is exported from @rules_zoo//tool).

rules_zoo//tool execution will fail from the root module

Notes:

• Packages without namespace import don't cause this issue.
• Running to tool in rules_zoo works.

🔬 Minimal Reproduction

https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/strange_venv

🔥 Exception or Error

bazelisk run --ui_event_filters=-info --noshow_progress --show_result=0  @rules_dummy//tool
runfiles root is: /private/var/tmp/_bazel_hoshyari/86506b96d48de8ca9131f20dffbb0af5/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/external/rules_dummy+/tool/tool.runfiles
Import "cowsay" was successful
Import "google.cloud.storage" Failed!

🌍 Your Environment

Operating System: MacOS 26

Output of bazel version: 8.5.0

Rules_python version: 1.8.0-rc1

Anything else relevant?

This seems to be related to how we make the symlinks, it seems some of the symlinks are created in the wrong place. This is the runfiles I see for the tool when ran from root module

The symlinks marked with 1 are in the wrong place. Their prefix is _main instead of rules_...+

I used a rule to loop over the runfiles and see what is from what. The ones that are in the wrong place are the ones that are created with runfiles(symlinks=...).

# example of file in right place ("rules_dummy+/...")
File: is_symlink=True short_path=../rules_dummy+/tool/_tool.venv/bin/python3 path=bazel-out/darwin_arm64-fastbuild/bin/external/rules_dummy+/tool/_tool.venv/bin/python3

# example of file in wrong place ("_main/...")
SymlinkEntry: tool/_tool.venv/lib/python3.13/site-packages/google/cloud/common_resources.proto <source file site-packages/google/cloud/common_resources.proto>

Which seems to be Bazel's behavior 😢 https://bazel.build/rules/lib/builtins/ctx#runfiles

Either a SymlinkEntry depset or the map of symlinks to be added to the runfiles. Symlinks are always added under the main workspace's runfiles directory (e.g. <runfiles_root>/_main/<symlink_path>, not the directory corresponding to the current target's repository. See Runfiles symlinks in the rules guide.

It seems that root_symlinks might be a better choice https://bazel.build/extending/rules#runfiles_symlinks

The root_symlinks is a dictionary mapping paths to files, where the paths are relative to the root of the runfiles directory. The symlinks dictionary is the same, but paths are implicitly prefixed with the name of the main workspace (not the name of the repository containing the current target).

[rickeylev]

Ahh, good catch. Yeah, root_symlinks is more appropriate. I think there's a test for this when venv-site-packages is disabled. Thanks for the PR! I'll have a look.

This isn't present in 1.7, but still worked? I swear 1.7 had the logic that would fallback to runfiles symlinks; I don't recall switching from root_symlinks to symlinks.

[rickeylev]

This was pretty subtle. It occurs as an interaction between the py_library logic that creates the optimized VenvSymlinkEntry objects, which the conflict merging logic processes, which then triggers the single-file optimization in the logic that creates the symlinks.

Basically:

# 1. Two different py_library create e.g.
VenvEntry(venv_path="google/foo/__init__.py", ...)
VenvEntry(venv_path="google/bar/__init__.py", ...)

# 2. Conflict merging sees the common prefix and fixes things
link_map["google/{foo,bar}/__init__"] = <respective file>

# 3. final symlinking logic does:
for link_from, link_to in link_map:
  if <link_to is file>:
    <use runfiles.symlinks>

If there was only one VenvEntry, then the link_map[...] = <file object> in step (2) doesn't occur, and then step (3) uses link_to_path

In any case, your fix looks correct! I've pushed the basics of a test to verify it.

[shayanhoshyari]

This isn't present in 1.7, but still worked? I swear 1.7 had the logic that would fallback to runfiles symlinks; I don't recall switching from root_symlinks to symlinks.

1.7 does not have your use of runfiles symlinks yet (to reduce num actions). it uses actions.add_file and actions.add_symlink . That is why it does not have this issue.

https://github.com/bazel-contrib/rules_python/blob/1.7.0/python/private/venv_runfiles.bzl#L51-L66

[shayanhoshyari]

This was pretty subtle. It occurs as an interaction between the py_library logic that creates the optimized VenvSymlinkEntry objects, which the conflict merging logic processes, which then triggers the single-file optimization in the logic that creates the symlinks.

Thanks for the detailed explanation!