Error: 'PyRuntimeInfo' value has no field or method 'pyc_tag' when using platforms

[carpenterjc]

🐞 bug report

Affected Rule

The issue is caused by the rule:
py_llibrary, py_binary and python_register_toolchains

Is this a regression?

Yes: this worked in 0.36.0 and regressed in 0.37.0

Description

Analysis phase failed when you use a platform, it seems to get confused as to which PyRuntimeInfo it expects to see.

🔬 Minimal Reproduction

Run the following command on the example zip provided:
bazel build //... --platforms=//:x86_platform --nobuild

platform_analysis_broken.zip

🔥 Exception or Error

I first saw this:

Traceback (most recent call last):
        File "C:/dev/_bazel/jarbmqax/external/rules_python/python/private/py_library_rule_bazel.bzl", line 39, column 33, in _py_library_impl
                return bazel_py_library_impl(
        File "C:/dev/_bazel/jarbmqax/external/rules_python/python/private/py_library.bzl", line 74, column 51, in py_library_impl
                precompile_result = semantics.maybe_precompile(ctx, direct_sources)
        File "C:/dev/_bazel/jarbmqax/external/rules_python/python/private/common_bazel.bzl", line 107, column 30, in maybe_precompile
                pyc = _precompile(ctx, src, use_pycache = keep_source)
        File "C:/dev/_bazel/jarbmqax/external/rules_python/python/private/common_bazel.bzl", line 168, column 32, in _precompile
                if not target_toolchain.pyc_tag:
Error: 'PyRuntimeInfo' value has no field or method 'pyc_tag'
Available attributes: bootstrap_template, coverage_files, coverage_tool, files, interpreter, interpreter_path, python_version, stub_shebang

🌍 Your Environment

Operating System:

  
windows
  

Output of bazel version:

  
7.4.0
  

Rules_python version:

  
0.37.1
  

Anything else relevant?

We use platforms

[manfredlift]

I have same issue on linux, bazel version 7.4.0 & rules_python 0.37.2. I didn't even use --platform flag like OP.

[rickeylev]

That error indicates that a Bazel builtin PyRuntimeInfo object is ending up in that code path, when a rules_python PyRuntimeInfo object is expected. That usually occurs when the legacy @bazel_tools//tools/python:autodetecting_toolchain gets used, which usually the last one attempted to be used. (The older Bazel builtin PyRuntimeInfo lacks several fields the newer rules_python one does)

Looking at the code in 0.36, this error also looks possible there (the maybe_precompile function doesn't have guards for Bazel builtin PyRuntimeInfo). My guess would be this is because the exec toolchain is now enabled by default (from the changelog):

The exec tools toolchain is enabled by default. It can be disabled by setting --@rules_python//python/config_settings:exec_tools_toolchain=disabled

When disabled, the code paths that would require a rules_python PyRuntimeInfo would be skipped. I bet if you set that in 0.36, you'll get a similar error.

As to why the rules_python toolchain isn't being selected -- that isn't clear. My only guess is because there is no toolchain matching the desired platform. I'd have to see the output of --toolchain_resolution_debug to get a sense of what is happening with toolchains. I noticed in the repro that x86_platfrom is mapping to (I think it was) a 32-bit cpu setting; I'd have to go look at the generated toolchain constraints, but I'm fairly sure everywhere in there is x86_64, and there is no 32-bit toolchain registered.

We could probably add some guards to maybe_precompile to bail out if a Bazel builtin PyRuntimeInfo object ends up there. In the meantime, however a couple possible work arounds:

• Set --@rules_python//python/config_settings:exec_tools_toolchain=disabled (this will make the code paths exit early, as in 0.36)
• Set --extra_toolchains=@rules_python/python/runtime_env_toolchains:all (this will register a rules_python equivalent to the bazel_tools one, but using rules_python objects instead)

[carpenterjc]

The option --extra_toolchains=@rules_python/python/runtime_env_toolchains:all breaks the hermisity of the python process. I have noticed although it provides its copy of the python process in the runfiles the bootstrapper actually launches the system python.

[carpenterjc]

Output showing toolchain resolution:

bazel build   --platforms=//:x86_platform //... --nobuild --toolchain_resolution_debug=@bazel_tools//tools/python:toolchain_type 
INFO: Invocation ID: 61156a41-02a8-4bcc-ab2b-9c2d3671d4e0
WARNING: Build option --platforms has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/python:toolchain_type for target platform //:x86_platform
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-apple-darwin//:python_runtimes; mismatching values: osx, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-unknown-linux-gnu//:python_runtimes; mismatching values: linux, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching values: linux, ppc
      ToolchainResolution:   Rejected toolchain @@python3_11_s390x-unknown-linux-gnu//:python_runtimes; mismatching values: linux, s390x
      ToolchainResolution:   Rejected toolchain @@python3_11_x86_64-apple-darwin//:python_runtimes; mismatching values: osx, x86_64
      ToolchainResolution:   Rejected toolchain @@python3_11_x86_64-pc-windows-msvc//:python_runtimes; mismatching values: windows, x86_64
      ToolchainResolution:   Rejected toolchain @@python3_11_x86_64-unknown-linux-gnu//:python_runtimes; mismatching values: linux, x86_64
      ToolchainResolution:   Toolchain @@bazel_tools//tools/python:_autodetecting_py_runtime_pair is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/python:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/python:toolchain_type toolchains for target platform //:x86_platform:
      ToolchainResolution:   Selected @@bazel_tools//tools/python:_autodetecting_py_runtime_pair to run on execution platform @@platforms//host:host
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/python:toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-apple-darwin//:python_runtimes; mismatching values: osx, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-unknown-linux-gnu//:python_runtimes; mismatching values: linux, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching values: linux, ppc
      ToolchainResolution:   Rejected toolchain @@python3_11_s390x-unknown-linux-gnu//:python_runtimes; mismatching values: linux, s390x
      ToolchainResolution:   Rejected toolchain @@python3_11_x86_64-apple-darwin//:python_runtimes; mismatching values: osx
      ToolchainResolution:   Toolchain @@python3_11_x86_64-pc-windows-msvc//:python_runtimes is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/python:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/python:toolchain_type toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@python3_11_x86_64-pc-windows-msvc//:python_runtimes to run on execution platform @@platforms//host:host

and for the default x64 platform:

bazel build  //... --nobuild --toolchain_resolution_debug=@bazel_tools//tools/python:toolchain_type                              
INFO: Invocation ID: 3f8e0dc4-93d7-456d-9692-23e455a0e4c2
WARNING: Build option --platforms has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/python:toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-apple-darwin//:python_runtimes; mismatching values: osx, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-unknown-linux-gnu//:python_runtimes; mismatching values: linux, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching values: linux, ppc
      ToolchainResolution:   Rejected toolchain @@python3_11_s390x-unknown-linux-gnu//:python_runtimes; mismatching values: linux, s390x
      ToolchainResolution:   Rejected toolchain @@python3_11_x86_64-apple-darwin//:python_runtimes; mismatching values: osx
      ToolchainResolution:   Toolchain @@python3_11_x86_64-pc-windows-msvc//:python_runtimes is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/python:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/python:toolchain_type toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@python3_11_x86_64-pc-windows-msvc//:python_runtimes to run on execution platform @@platforms//host:host
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/python:toolchain_type for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-apple-darwin//:python_runtimes; mismatching values: osx, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_aarch64-unknown-linux-gnu//:python_runtimes; mismatching values: linux, aarch64
      ToolchainResolution:   Rejected toolchain @@python3_11_ppc64le-unknown-linux-gnu//:python_runtimes; mismatching values: linux, ppc
      ToolchainResolution:   Rejected toolchain @@python3_11_s390x-unknown-linux-gnu//:python_runtimes; mismatching values: linux, s390x
      ToolchainResolution:   Rejected toolchain @@python3_11_x86_64-apple-darwin//:python_runtimes; mismatching values: osx
      ToolchainResolution:   Toolchain @@python3_11_x86_64-pc-windows-msvc//:python_runtimes is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@bazel_tools//tools/python:toolchain_type toolchain, stopping
      ToolchainResolution: Recap of selected @@bazel_tools//tools/python:toolchain_type toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@python3_11_x86_64-pc-windows-msvc//:python_runtimes to run on execution platform @@platforms//host:host

[rickeylev]

Thanks, that confirms what I expected. You can see it rejects all the @python... toolchains -- those are the ones created by rules_python. Eventually, it falls back to the @bazel_tools//tools/python:autodetecting one, which matches anything.

The option --extra_toolchains=@rules_python/python/runtime_env_toolchains:all breaks the hermisity of the python process.

Correct. Also, I forgot that --extra_toolchains comes first when searching, so you'd want to do --extra_toolchains=@pythons_hub//:all,@rules_python//python/runtime_env_toolchains:all (remember to use_repo()). This will try to hermetic ones first, and then the non-hermetic one afterwards.

But note that, if you were building for a 32-bit cpu, then you weren't using the hermetic interpreters anyways -- none of them say they're compatible with that. It was silently falling back to the old bazel builtin one.