support namespace packing

[mozartilize]

I follow the docs here: https://packaging.python.org/guides/packaging-namespace-packages/
but it does not work, maybe poetry not support yet?

[uSpike]

How would the maintainers feel about an additional optional field in pyproject.toml called module. For example:

[tool.poetry]
name = "abc-foo"
module = "abc.foo"
...

corresponds with a layout like:

pyproject.toml
abc/
    foo/
        __init__.py

[RaptDept]

See #356 (comment)

[dfee]

This isn't an issue for me.

[tool.poetry]
name = "abc.foo"
version = "2018.0.0"
description = "abc foo module"
authors = ["Devin Fee <dev...>"]
packages = [
    { include = "abc" },
]

[tool.poetry.dependencies]
python = "^3.6.5"
"abc.core" = {path = "../abc.core"}  # other namespaced modules work, too.

[moigagoo]

@dfee what if the package name is different and doesn't correlate with the dir structure?

[dfee]

My directory structure is as follows:

repo/
    abc.core/
        pyproject.toml
        abc/
            core/
                __init__.py
    abc.asdf/
        pyproject.toml
        abc/
            asdf/
                __init__.py
    abc.qwerty/
        pyproject.toml
        abc/
            asdf/
                __init__.py

That should explain this line "abc.core" = {path = "../abc.core"} for everyone else.

@moigagoo is your question about something that looks like the following?

repo/  # warning: not my structure
    abc.core/ # same as above
        pyproject.toml
        abc
            core/
                __init__.py
    abc.asdf/ # still just a package root (irrelevant name as I understand it)
        pyproject.toml
        abc.asdf/ # I don't think this complies with PEP420?
            __init__.py
    abc.qwerty/
        pyproject.toml
        my_qwerty/ # I don't think this complies with PEP420?
            __init__.py

Yeah, I just don't know that anything besides my current structure is compliant with PEP420

[sdispater]

Poerty support this use case https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages by explicitly declaring the various namespaced packages with the packages property.

[geryogam]

The name of a distribution is declared in the tool.poetry.name property of the pyproject.toml file and appears:

• in the name of the dist/{distribution_name}-{version}-py3-none-any.whl Wheel distribution file and in the name of the {distribution_name}-{version}.dist-info/ directory of the Wheel distribution file created when the distribution is built;
• in the name of the site-packages/{distribution_name}.egg-link symbolic link created when the distribution is installed with edit mode;
• in the name of the site-packages/{distribution_name}-{version}.dist-info/ directory created when the distribution is installed without edit mode;
• in the output of pip list (under the misleading "Package" column which should be rather named "Distribution") when the distribution is installed.

The name of a root module of a distribution is declared in the file system and appears:

• in the name of the {root_module_name}.py file or {root_module_name}/ directory of the Wheel distribution file created when the distribution is built;
• in the name of the site-packages/{root_module_name} source code directory created when the distribution is installed without edit mode.

By default, Poetry looks up the root module to include in a distribution by the distribution name declared in the tool.poetry.name property of the pyproject.toml file, first in the directory of the pyproject.toml file, then in the src/ subdirectory if it exists.

If your root module cannot be found by this default mechanism (the root module is located elsewhere, the root module has a different name than that of the distribution or there are multiple root modules), you have to declare it in the tool.poetry.packages property of the pyproject.toml file, as described in the documentation: add a dictionary item to the list for each root module, with an include property declaring its name as @sdispater said earlier (e.g., foo.py for a non-package module, bar for a package, baz/qux for a package with only its qux subpackage) and, if necessary, a from property declaring the relative path of its parent directory (e.g., src or lib).

For instance, let us say that you need to build three distributions named application_1, application_2 and library with the following repository layouts:

application_1
├── .git
├── src
│   └── organization
│       └── application
│           └── application_1
│               ├── __init__.py
│               └── __main__.py
└── pyproject.toml

application_2
├── .git
├── src
│   └── organization
│       └── application
│           └── application_2
│               ├── __init__.py
│               └── __main__.py
└── pyproject.toml

library
├── .git
├── src
│   └── organization
│       └── library
│           ├── library_1
│           │   └── __init__.py
│           ├── library_2
│           │   └── __init__.py
│           └── library_3
│               └── __init__.py
└── pyproject.toml

where application_1/src/organization/application/application_1/__main__.py is:

from organization.library.library_1 import f
from organization.library.library_2 import g

print(f())
print(g())

and application_2/src/organization/application/application_2/__main__.py is:

from organization.library.library_3 import h

print(h())

and library/src/organization/library/library_1/__init__.py is:

def f():
    return "foo"

and library/src/organization/library/library_2/__init__.py is:

def g():
    return "bar"

and library/src/organization/library/library_3/__init__.py is:

def h():
    return "baz"

In order to build the distributions, application_1/pyproject.toml should be:

[tool.poetry]
name = "application_1"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and application_2/pyproject.toml should be:

[tool.poetry]
name = "application_2"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and library/pyproject.toml should be:

[tool.poetry]
name = "library"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[wallrite]

This isn't an issue for me.

[tool.poetry]
name = "abc.foo"
version = "2018.0.0"
description = "abc foo module"
authors = ["Devin Fee <dev...>"]
packages = [
    { include = "abc" },
]

[tool.poetry.dependencies]
python = "^3.6.5"
"abc.core" = {path = "../abc.core"}  # other namespaced modules work, too.

@dfee Do you have any problems building your project? I have the same structure and faced problem with poetry build.

I think I've found related issue #2046

[pawamoy]

@maggyero, just to be clear, in

[tool.poetry]
name = "application_1"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

...it is not mandatory to use application_1 as name, right? We could use any name we want, and pip will in any case install the application_1 package in the organization.application namespace'd one.

[geryogam]

@pawamoy No it is not mandatory to use "application_1" as name, you can call your distribution as you want. The binding between the distribution name and the top-level Python module name "application_1" will be recorded in the file top_level.txt of the distribution.

For instance you can see that the setuptools distribution has three top-level Python modules:

$ cat /usr/local/lib/python3.8/site-packages/setuptools-49.2.0-py3.8.egg-ino/top_level.txt
easy_install
pkg_resources
setuptools

Indeed:

$ ls -l /usr/local/lib/python3.8/site-packages/
total 16
drwxr-xr-x   4 maggyero  admin   128 18 jul 13:40 __pycache__
-rw-r--r--   1 maggyero  admin   126 13 jul 14:37 easy_install.py
lrwxr-xr-x   1 maggyero  admin    81 18 jul 13:41 homebrew-protobuf.pth -> ../../../Cellar/protobuf/3.12.3/lib/python3.8/site-packages/homebrew-protobuf.pth
drwxr-xr-x   7 maggyero  admin   224 18 jul 13:40 pip
drwxr-xr-x   8 maggyero  admin   256 18 jul 13:40 pip-20.1.1-py3.8.egg-info
drwxr-xr-x   6 maggyero  admin   192 18 jul 13:40 pkg_resources
drwxr-xr-x  45 maggyero  admin  1440 18 jul 13:40 setuptools
drwxr-xr-x   9 maggyero  admin   288 18 jul 13:40 setuptools-49.2.0-py3.8.egg-info
-rw-r--r--   1 maggyero  admin  2081 18 jul 13:40 sitecustomize.py
drwxr-xr-x  14 maggyero  admin   448 18 jul 13:40 wheel
drwxr-xr-x   9 maggyero  admin   288 18 jul 13:40 wheel-0.34.2-py3.8.egg-info

[Luttik]

Edit I made a dumb comment here that was addressed in #356. tl;dr
If you want to use a nested namespace like organization.domain.tool you can use nested folders (organization/domain/tool) and use this in your package.toml

packages = [
    { include = "organization", from = "src" }  # only use from="src" if that is how you've structured your code.
]

poetry will automatically know how to interpret the folders nested under organization.