Namespace packags and setup.py develop do not place nicely.

[dstufft]

setup.py develop uses an egg-link (cross platform symlink basically) in order to make it work. However this fails miserably when the packages you're trying to develop have a setuptools style namespace package.

[jaraco]

See pip #3 for a related discussion.

[eevee]

fwiw, I've only ever cared about setup.py develop for writing Pylons and Pyramid applications, which rely on entry points because they're run with Paste. If I could just launch them from the project directory like I do everything else then this would be less of a problem.

Entry points themselves are a glorious mess, too, but we don't have anything better.

[ncoghlan]

We do actually have a bettter option: *.pth files in virtual environments are a much better solution to this than dependency links, since they can handling multi-portion namespace packages properly and transparently.

*.pth files are only evil in global installations (since they implicitly alter sys.path for everything run in that environment with site module processing enabled)

[theacodes]

After reading through various stuff on this, I'm still unclear as to what needs to happen for this to be fixed? Does setuptools need to be modified to write a .nspth instead of an .egg-link when running setup.py develop for namespace packages? What are the ramifications of not using .egg-links?

I'm asking these questions because I'm willing to do the work, but I want to make sure I'm carving the right pumpkin.

[ncoghlan]

@jonparrott I think you're unclear because we're all unclear ;)

Something I think may fix this is if setup.py develop changed to work as follows:

• if it sees a venv is active, drop a "-develop.pth" file into the site-packages of the active virtualenv that references the project directory, rather than doing what it does now
• if it doesn't see an active venv, it keeps doing the same thing it does today

We may be trading one problem for another (e.g. anyone using a "shared virtualenv" model for multiple projects that they want to keep separate isn't going to like that change), but we would gain a fair bit in simplicity and predictability (since we'd be using a core import system feature, rather than something specific to setuptools)

[ncoghlan]

Related to the "shared virtualenv" problem, I also just realised that .pth files could be used to implement virtualenv inheritance: putting a .pth in venv B that references the site-packages of venv A will give you a situation where venv B sees all changes made in venv A, but anything new you install (including via setup.py develop changed as suggested above) will go solely into venv B.

[theacodes]

I think you're unclear because we're all unclear ;)

Oh, that makes me feel better.

Something I think may fix this

That sounds reasonable. Could we be even more reasonable and put the new behavior behind a flag (e.g. setup.py develop --use-pth)?

I also just realised that .pth files could be used to implement virtualenv inheritance

That sounds interesting, but not a use case I wanna focus on for the moment.

[ncoghlan]

Starting off with the new behaviour behind a flag sounds like an excellent idea, as that enables two things:

• if the default doesn't work, we can suggest "--use-pth" as a possible fix
• folks for whom the default currently works can readily test the new behaviour to see if it breaks anything

That would suggest it also makes sense to add a "--use-egg-link" option to start setting up for a possible future change to the default behaviour (assuming "--use-pth" actually does turn out to fix the current problems).

[theacodes]

SGTM, I'll start trying to make this materialize.

FYI, I'm familiar with pip's codebase but not setuptools, so the first draft might be slightly more hacky than expected. :)

[theacodes]

(rough) PR sent: pypa/setuptools#789

[merwok]

We now have standards-based namespace packages and develop install. I hope they work well together*, but if not this would be a normal bug to report to the build backends.

A setuptools ticket exists and was closed. I think this ticket can be closed @woodruffw

*(not a fan myself of PEP 420 that makes every data directory an implicit namespace package 😦)