Is it possible to get fine-grained control over which stdlib extensions are included?

[njsmith]

I was trying to figure out how to include specific stdlib extension modules (e.g. _ssl) but not others (e.g. _asyncio). I found:

• PythonPackagingPolicy.extension_module_filter: this lets you set a few general policies like all or minimal, but doesn't give control on a per-resource level AFAICT
• PythonPackagingPolicy.register_resource_callback: this gives you a ton of flexibility to configure most resources... but for stdlib extension modules, the PythonExtensionModule it receives seems to have add_include=False regardless of the setting of the extension_module_filter.

I think the most intuitive way would be for extension_module_filter to apply first and set the default values for add_include on all the stdlib modules, and then the resource callback would run second and have a chance to override that on a per-module basis. But apparently something else is going on?

Not sure if this is a documentation request or a feature request :-)

[indygreg]

I looked at the source code and the registered resource callback should be getting called for stdlib extension modules and its effects should be honored. So setting .add_include = False inside the callback should work.

However, there is some wonky Starlark code involved in the processing of resource callbacks and I wouldn't be surprised if mutations in the callbacks aren't working correctly. I'm also not sure if we have explicit test coverage of this scenario to catch failures.

I did refactor some Starlark code in the unreleased main branch a few weeks ago to hopefully minimize potential for copies of Starlark values to get out of sync with the original value (some Starlark values were effectively being copied instead of reference counted). This may have fixed this apparent bug. But someone will need to verify.

Thanks for reporting this issue. It definitely feels like a bug to me. We definitely want resource callbacks to influence stdlib resources.

[bbqsrc]

Can confirm the following worked on Windows:

def resource_callback(policy, resource):
    if type(resource) in ("PythonExtensionModule"):
        if resource.name == "_ctypes":
            resource.add_include = False

extension_module_filter does still run first though, and pre-filters everything out, so you can't toggle things back on.

[bbqsrc]

Ahh, seems I spoke to soon. Linking still failed. Will update when I experiment more.

[bbqsrc]

@indygreg I can confirm that the setter is called and the value is set to false, but it doesn't have much of an impact on the actual build behaviour.

[WankkoRee]

The same problem still exists in the 0.24.0. Any solution?