Build package from dynamically created directory via custom build.py

[rragundez]

• Poetry version: 1.5.1

• Python version: 3.10

• OS version and name: macOS 13.5

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I have a custom build section by using the following config in my pyproject.toml

generate-setup-file = false
script = "build.py"

The build.py script dynamically creates my package in a dynamically created directory, let's say my_real_package/ (does not exist at the moment of executing poetry build). In my pyproject.toml I have

[tool.poetry]
packages = [{include = "my_real_package"}]

Then when executing poetry build I am shown the error .../my_real_package does not contain any element at the start, because the my_real_package is created during the build and is not present at the start.

I tested and if I change packages = [{include = "package"}] (this directory exists as a package), run the build so that my_real_package is created and then change back packages = [{include = "my_real_package"}] and run again poetry build, it works.
So this tells me that updating to a warning the error for special cases (like when there is a custom build script) .../my_real_package does not contain any element will make everything work with a single poetry build command where I keep packages = [{include = "my_real_package"}] and my_real_package is dynamically created by my build.py script.

[rragundez]

I am trying to bring poetry and best practices with pyproject.toml into the new upcoming version of the popular Keras package. But there is this last missing piece I need, where the package to build is created dynamically via a script.

rragundez/keras-core#1

[radoering]

I suppose you have to distinguish between building an sdist and a wheel. By default, poetry will build both formats. You can just create one of both by providing the format option.

When building an sdist, the build script is not executed so that it makes sense that the command fails. Maybe, you have to change your include to

[tool.poetry]
packages = [{include = "my_real_package", format = "wheel"}]

When building a wheel, the code looks like the build script will be executed before modules are copied.

Does poetry build -f wheel fails, too?

[rragundez]

Yes, still fails.

Looking into the code I find that the reason is:

The Builder base class instantiation creates a Module which in turn instantiates a PackageInclude which calls check_elements which does checks for the package includes directory (in this case the my_real_package directory).

Not sure how to go about it, or perhaps I am missing some option or way of doing this.

any further ideas @radoering

[radoering]

It seems you are right. So the question is whether we can postpone the creation of the module. At first glance, it looks like the module is only used after executing the build script (at least for the WheelBuilder) so it should be possible.

[rragundez]

Would you like me to look into it and make an initial PR proposal? Or should I wait for some checks from your side first and wait for confirmation if such work would be useful?

[radoering]

Go ahead if you like. It may be as simple as replacing _module with a cached_property. If all existing tests still pass and you are able to add a test that fails without your change (and passes with it), it should be fine.

[radoering]

Currently, _module is initialized in the __init__ (and not set elsewhere). With a cached_property we can postpone the initialization until it is required. So replacing

def __init__(...):
    self._module = Module(
        self._package.name,
        self._path.as_posix(),
        packages=packages,
        includes=includes,
    )

with

@cached_property
def _module:
    return Module(
        self._package.name,
        self._path.as_posix(),
        packages=packages,
        includes=includes,
    )

will initialize _module on first access.

[rragundez]

@radoering please have a look at python-poetry/poetry-core#629