[Python] Installing pyarrow from source fails with latest setuptools-scm (-> needs "setuptools_scm<8")

[jorisvandenbossche]

Installing released pyarrow packages from source currently fails with the latest setuptools_scm. You will see error messages like this (copied from #37803 (comment)):

error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [38 lines of output]
      <string>:36: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      Traceback (most recent call last):
        File "/home/runner/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/runner/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/runner/.local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 396, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 630, in <module>
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 147, in setup
          _setup_distribution = dist = klass(attrs)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 303, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
          self.finalize_options()
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 680, in finalize_options
          ep(self)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 700, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools_scm/_integration/setuptools.py", line 102, in version_keyword
          _assign_version(dist, config)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools_scm/_integration/setuptools.py", line 54, in _assign_version
          maybe_version = _get_version(config, force_write_version_files=True)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools_scm/_get_version_impl.py", line 94, in _get_version
          write_version_files(config, version=version_string, scm_version=parsed_version)
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools_scm/_get_version_impl.py", line 63, in write_version_files
          dump_version(
        File "/tmp/pip-build-env-x8k25mom/overlay/lib/python3.8/site-packages/setuptools_scm/_integration/dump_version.py", line 36, in dump_version
          assert not write_to.is_absolute(), f"{write_to=}"
      AssertionError: write_to=PosixPath('/tmp/pip-download-2jknfbz1/pyarrow_b07f740b726647018b58ccd586f32386/pyarrow/_generated_version.py')
      [end of output]

See #37803 for some additional reports, and pypa/setuptools-scm#925 for an issue on the setuptools_scm side.

Currently the fix is to ensure an older version of setuptools_scm is installed when installing pyarrow from source (setuptools_scm<8).

Opening this issue to keep track of the problem and to ensure people can more easily find this when encountering the problem (compared to the closed and seemingly unrelated issue at #37803), and we can close this once it is fixed upstream (or here).

[h-vetinari]

It appears that this should be fixed by pypa/setuptools-scm@056584b, which is part of setuptools-scm 8.0.4