Use `site-packages` folder instead of `app` and `app_packages`

[timrid]

What is the problem or limitation you are having?

While I was working on the integration of the .pth files for a remote debug feature, I dived deep into the different platforms supported by briefcase. I realized how differently all platforms work.

What surprised me the most is why briefcase does not use the official site-packages folder at all, but always works with an app and app_packages folder. As I understand it, this is only to enable the three cases to be executed independently of one another:

• briefcase run --update: Update the application’s source code
• briefcase run --update-requirements: Update application requirements
• briefcase run --update-support: Update application support package

Is this generally correct or am I missing something?

Describe the solution you'd like

If this is the only reason, isn't there a way to use the official site-packages folder and still keep the CLI parameters the same for the users? For example, just change the logic like this:

• briefcase run --update: Create a .whl of the current app code (e.g. helloworld-0.0.1-py3-none-any.whl) and then do a pip install helloworld-0.0.1-py3-none-any.whl --target {site_packages_folder} to update only the app code.
• briefcase run --update-requirements: Updates all other packages specified in pyproject.toml under requires via pip install -r requirements.txt --target {site_packages_folder}. The app code remains the same, because it is not listed under requires.
• briefcase run --update-support: Copies the site-packages folder to a temporary location, then replaces the old with the new support package and then copies the saved site-packages folder back to the original location.

I know that this would be a lot of work, because besides briefcase itself all briefcase templates would have to be adapted. But using the site-packages folder would be closer to the default behavior of Python and from my point of view a significant improvement.

Describe alternatives you've considered

Leave everything as it is. It works, but it's more complicated to understand, as many specific adjustments are required. Furthermore, adding .pth files is more complicated than necessary, as on some platforms they're only found in the site-packages folder and not in app_packages (see here).

Additional context

This topic has already been discussed here and here, but I couldn't find an explicit ticket for this proposal. I also found the comments here and here on this topic, but they don't offer much added value.

[freakboy3742]

What surprised me the most is why briefcase does not use the official site-packages folder at all, but always works with an app and app_packages folder.
...
Is this generally correct or am I missing something?

You're generally correct, but there's one additional requirement that you're missing: The default site-packages location for a Python install is in the lib folder of the Python distribution - and on some platforms, that location can't have files added to it.

On macOS and iOS, the lib folder is in the Python.framework - and that framework is a signed bundle. It's not advisable to change the content inside a signed bundle.

On Linux system packages, the site folder is owned by root - and is shared with other system packages - and adding files to that folder without coordinating with the rest of the platform's packaging ecosystem could be difficult.

It's less of an issue on Android, Windows, and Flatpak - but having the support package independent of the app packages does make the upgrade path marginally easier, as there's a clear separation of concerns. Even if that wasn't the case, maintaining consistency between platforms is helpful to avoid confusion in the long term.

Describe the solution you'd like

If this is the only reason, isn't there a way to use the official site-packages folder and still keep the CLI parameters the same for the users?

In short - not easily, for the reasons I've listed above.

However, there is another option that you haven't considered: Make app_packages a site folder.

The site_packages folder isn't inherently special - it's just on the site list by default because it's a location tied to a the Python interpreter's libdir. The method site.addsitedir() exists to add additional site folders - I'm fairly certain the fix here is to use this method to add app_packages as a site dir, rather than just adding it to PYTHONPATH/sys.path as the stub is currently doing.

I'm less convinced this is the right approach for app - I think it's correct for app to just be a path addition - but I'm willing to be convinced otherwise.

I know that this would be a lot of work, because besides briefcase itself all briefcase templates would have to be adapted.

Even in my proposed solution, all the templates are going to need modifications. Yes, it's a lot of changes, but if it's a bug - the bug needs to be fixed.

The good news is that the stub binary templates are mostly the same (allowing for fairly C dialect differences) - so once we work out the right fix for one of them, the same fix should (broadly speaking) work everywhere.

There may even be some bugs in the existing stubs (for example, you've found the setting on the Linux system stub that is disabling the site package... I'm not 100% sure that's correct. It should be isolated from the user site... but it shouldn't be completely disabling site handling. Unless there's a really compelling reason not to, we should be aiming for consistency between platforms.

But using the site-packages folder would be closer to the default behavior of Python and from my point of view a significant improvement.

Completely agreed that this should be the overall goal. There may be oddities or unusual things about app packaging, but there shouldn't be anything that we can't explain in the broader context of Python as a language.

Additional context

This topic has already been discussed...

I think what you've reported here is the underlying cause of a bunch of esoteric issues in packages that assume that they're going to be installed into a site package. Although your immediate use case is the --debug handling, a fix for this will likely resolve a bunch of other problems.

[timrid]

As I understand it, that was a clear no to this proposal. I have opened a new bug ticket to track the bug that .pth files are not being processed: #2204