[Mac] Debugpy 1.6.5 not installable with poetry

[tom-mcclintock]

Before creating a new issue, please check the FAQ to see if your question is answered there.

Environment data

• OS and version: Mac 12.5.1
• poetry 1.3.2

Actual behavior

Poetry is a python package manager that specifies its virtual envs in a pyproject.toml file. Here is an example file to start with:

[tool.poetry]
name = "example"
version = "0.1.0"
description = "Install me."
authors = []

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Attempting to install the latest debugpy fails:

$ poetry add debugpy
Using version ^1.6.5 for debugpy

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing debugpy (1.6.5): Failed

  CalledProcessError

  Command '['/Users/tom.mcclintock/Repositories/tomsource/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/Users/tom.mcclintock/Repositories/tomsource/.venv', '--no-deps', '/Users/tom.mcclintock/Library/Caches/pypoetry/artifacts/00/1e/5f/034fcde7ad28e523856bdacb87533e7b40228ca678f31ad37f757684bc/debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl']' returned non-zero exit status 1.

  at ~/opt/anaconda3/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/Users/tom.mcclintock/Repositories/tomsource/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/Users/tom.mcclintock/Repositories/tomsource/.venv', '--no-deps', '/Users/tom.mcclintock/Library/Caches/pypoetry/artifacts/00/1e/5f/034fcde7ad28e523856bdacb87533e7b40228ca678f31ad37f757684bc/debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl'] errored with the following return code 1, and output: 
  ERROR: debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│ 
      1542│         return decode(output)
      1543│ 
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install /Users/tom.mcclintock/Library/Caches/pypoetry/artifacts/00/1e/5f/034fcde7ad28e523856bdacb87533e7b40228ca678f31ad37f757684bc/debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/pip.py:58 in pip_install
       54│ 
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│ 

But installing the previous version works: poetry add debugpy=1.6.4.

I believe the crux of the error is: ERROR: debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform

Expected behavior

I'd like to be able to install the latest debugpy :).

[int19h]

Does the error only show up with poetry, but not when pip install is used directly? And if it does show up for the latter, do you get any additional output beyond "not a supported wheel on this platform"?

[tom-mcclintock]

@int19h pip installs debugpy successfully.

poetry install
poetry shell
pip install debugpy

I should clarify though -- the goal is to make debugpy 1.6.5 installable with poetry because it is a dependency of IPyKernel as of a couple days ago. I suspect I'm just the first person to report this issue ;).

[int19h]

As far as platform support goes, we publish a py3-none-any wheel that should serve as a fallback for any platform that is not served by the more specific wheels. But we also publish debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl specifically - and if pip install worked, it sounds like that wheel was fine?

Looking at the log, the wheel that Poetry is trying to install is coming from a path that is its cache. Is that a cached version of the PyPI wheel, or did it try to build its own instead, perhaps in some way that does not involve setup.py? While we do have pyproject.toml, we still rely on setup.py to properly package the binary bits in vendored pydevd, which precluded this last time we tried a more declarative approach.

[tom-mcclintock]

Hmm I think I am following.

RE "is that a cached version of ...?" - how can I help answer that? Are you asking if it's a local cache, or a cache somewhere that poetry looks for wheels?

Thanks again for working with me here!

[int19h]

I was referring to this path:
.../Caches/pypoetry/artifacts/00/1e/5f/034fcde7ad28e523856bdacb87533e7b40228ca678f31ad37f757684bc/debugpy-1.6.5-cp39-cp39-macosx_11_0_x86_64.whl
It would be nice to figure out where exactly this comes from. If it's the same wheel that we publish on PyPI, it should work (or else we have a bigger problem!). My concern is that it might be a wheel produced by Poetry itself from debugpy sources, and that something went wrong in the process of building it. If you can compare that file and its contents to the official wheels on PyPI, the difference might be visible.

BTW, is this actually an x86-64 Mac, or is it ARM? We don't publish wheels for specifically ARM, so the expectation is that the py3-none-any wheel would be used there; but, perhaps, Poetry starts with the assumption that only macOS wheels are relevant, and ends up with the wrong architecture since there's no right one among the candidates it's considering?

[rantav]

I'm seeing the same issue. I'm new to poetry, so I hope I'm not missing something obvious. Anyway to your question @int19h it seems that the whl file is the same as the published one

My local failing file:

$ md5  /Users/rantavory/Library/Caches/pypoetry/artifacts/13/33/e8/f28b845f350c201a4998591d33980c1ce073de4d7d13ad138a8e2d3a94/debugpy-1.6.6-cp39-cp39-macosx_11_0_x86_64.whl
MD5 (/Users/rantavory/Library/Caches/pypoetry/artifacts/13/33/e8/f28b845f350c201a4998591d33980c1ce073de4d7d13ad138a8e2d3a94/debugpy-1.6.6-cp39-cp39-macosx_11_0_x86_64.whl) = 892defd64f3763fcd5401529a6701977

It's: 892defd64f3763fcd5401529a6701977

MD5 Pypi file from https://pypi.org/project/debugpy/1.6.6/#copy-hash-modal-3bdbf641-4208-4348-8b49-64073fcda391
MD5: 892defd64f3763fcd5401529a6701977

[tom-mcclintock]

@int19h sorry for the delayed response. My laptop is x86 (see attached).

I went into the cache and deleted the offending wheel, and attempted to poetry add debugpy again but it still failed. Here is the new error report:

Click me

$ poetry add debugpy
Using version ^1.6.6 for debugpy

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating debugpy (1.6.5 -> 1.6.6): Failed

  CalledProcessError

  Command '['/Users/tom.mcclintock/Repositories/tomsource/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/Users/tom.mcclintock/Repositories/tomsource/.venv', '--upgrade', '--no-deps', '/Users/tom.mcclintock/Library/Caches/pypoetry/artifacts/13/33/e8/f28b845f350c201a4998591d33980c1ce073de4d7d13ad138a8e2d3a94/debugpy-1.6.6-cp39-cp39-macosx_11_0_x86_64.whl']' returned non-zero exit status 1.

  at ~/opt/anaconda3/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/Users/tom.mcclintock/Repositories/tomsource/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/Users/tom.mcclintock/Repositories/tomsource/.venv', '--upgrade', '--no-deps', '/Users/tom.mcclintock/Library/Caches/pypoetry/artifacts/13/33/e8/f28b845f350c201a4998591d33980c1ce073de4d7d13ad138a8e2d3a94/debugpy-1.6.6-cp39-cp39-macosx_11_0_x86_64.whl'] errored with the following return code 1, and output: 
  ERROR: debugpy-1.6.6-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│ 
      1542│         return decode(output)
      1543│ 
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install /Users/tom.mcclintock/Library/Caches/pypoetry/artifacts/13/33/e8/f28b845f350c201a4998591d33980c1ce073de4d7d13ad138a8e2d3a94/debugpy-1.6.6-cp39-cp39-macosx_11_0_x86_64.whl

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/pip.py:58 in pip_install
       54│ 
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│ 

We can see that the wheel is different than last week (now it's downloading debugpy 1.6.6). This means that poetry downloads the wheel, puts it in the cache, and then attempts to install from there, hence the error message pointing to the cache location.

[mpas]

Having the same issue, the versions 1.6.5 and 1.6.6 are not installable with Poetry. The only thing that works is installing versions 1.6.4 (poetry add debugpy@1.6.4). Would be great if we could use the latest versions.

[int19h]

It looks like we got tripped by this change in Big Sur. Basically, the problem is that when the wheel is built, the toolchain assigns the platform tag according to the actual version of macOS used to build, which is macos-11.0. But then when pip running on Python 3.9 is determining the valid platform tags when installing the package, when it asks the OS about its version number, it lies and says that it's macOS 10.16 for backwards compatibility reasons. Consequently, the wheel is deemed not supported.

(The reason why this only happened now is that we were building older versions of Python on macOS 10.x before. But we can't do that anymore because it's been deprecated in Azure Pipelines for some time now and will be completely removed soon.)

The official workaround per Apple is to tell the OS to not lie by setting environment variable SYSTEM_VERSION_COMPAT=0 for the Python process that will be installing packages. A long-term fix would be for Python 3.9 to build with the new macOS SDK, so that it gets correct OS version reported regardless of the compat setting; but if I remember correctly, they don't do major SDK revisions across minor releases within a single major Python version. Thus, I would recommend using Python 3.10 or later if at all possible. If you're just using whatever version comes with macOS out of the box, pyenv is usually the easiest way to upgrade.

[int19h]

Another possibility would be for us to take down the macos-11.0 wheels for Python 3.8 and 3.9, thus forcing pip to install the generic wheel instead. It'll mean that debugging is somewhat slower as it's running as pure Python code, without Cython speedups. On the other hand, it would "just work", while the speedups are still available for those on Python 3.10+.