Bug: Error installing `PyYAML==5.4.1` as a dependency for `aws-sam-cli`

[realFranco]

Description:

I am trying to install AWS SAM CLI, and I am encountering an error when trying to install the PyYAML dependency. Specifically, when I try to install version 5.4.1 of PyYAML.

Some references:

• Error installing Pyyaml==5.4, Cython_sources yaml/pyyaml#724.

Steps to reproduce:

# Just install the package `aws-sam-cli` on a host that currently has `python3` and `pip3`.

python3 -V
Python 3.10.12

pip3 -V
pip 23.2 from /usr/lib/python3.10/site-packages/pip (python 3.10)

pip3 install --no-cache-dir \
        aws-sam-cli==1.82.0

I pick that specific version becuase of business requirements, but will affect all the versions that has this constraint:

PyYAML>=5.4.1,==5.*

• see: https://github.com/aws/aws-sam-cli/blob/develop/requirements/base.txt#L10

Observed result:

# pip3 install --no-cache-dir aws-sam-cli==1.82.0


Collecting aws-sam-cli==1.82.0
  Downloading aws_sam_cli-1.82.0-py3-none-any.whl (5.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 5.5 MB/s eta 0:00:00
Collecting chevron~=0.12 (from aws-sam-cli==1.82.0)
  Downloading chevron-0.14.0-py3-none-any.whl (11 kB)
Collecting click~=8.0 (from aws-sam-cli==1.82.0)
  Obtaining dependency information for click~=8.0 from https://files.pythonhosted.org/packages/22/b3/1da4ea0efa2e5ae410a347be614162ca08bd24a84059938aa5122d1e751b/click-8.1.5-py3-none-any.whl.metadata
  Downloading click-8.1.5-py3-none-any.whl.metadata (3.0 kB)
Collecting Flask<2.1 (from aws-sam-cli==1.82.0)
  Downloading Flask-2.0.3-py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.6/95.6 kB 4.5 MB/s eta 0:00:00
Requirement already satisfied: boto3==1.*,>=1.19.5 in /usr/lib/python3.10/site-packages (from aws-sam-cli==1.82.0) (1.19.5)
Requirement already satisfied: jmespath~=0.10.0 in /usr/lib/python3.10/site-packages (from aws-sam-cli==1.82.0) (0.10.0)
Collecting ruamel-yaml==0.17.21 (from aws-sam-cli==1.82.0)
  Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 kB 7.9 MB/s eta 0:00:00
Collecting PyYAML==5.*,>=5.4.1 (from aws-sam-cli==1.82.0)
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 3.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      /tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-kbw4u72p/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Expected result:

The expected result is that we can install the package aws-sam-cli with a newer version of PyYAML (>=6.0)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

The host where I am faccing the issue is agnostic because is related with python depedencies.

[realFranco]

• See: Unable to install aws-cli via pip - package dependency broke aws-cli#8036

[realFranco]

The workaround that solve my current problem:

pip3 install \
  wheel==0.40.0 \
&& pip3 install --no-build-isolation \
  Cython==0.29.36 \
  pyyaml==5.4.1 \
&& pip3 install --upgrade --no-cache-dir \
  aws-sam-cli==1.82.0 
      
# Use the `aws-sam-cli` version that you require. For my current case I just need the version "1.82.0".

See: pypa/pip#12158 (comment).

[hawflau]

@realFranco Thanks for raising the issue and providing the workaround. We've also confirmed the workaround works.
We recommend to use our installers whenever possible.

To all - if you have to use pip install to install AWS SAM CLI, please use this workaround (credit to @realFranco):

pip3 install wheel \
&& pip3 install --no-build-isolation "Cython<3" "pyyaml==5.4.1" \
&& pip3 install --upgrade --no-cache-dir aws-sam-cli

[ffMathy]

More information: yaml/pyyaml#724

[sirosen]

I think it's worth noting that the workaround isn't portable to all contexts, and neither are the installers.
We rely pretty heavily on tox to give us tool isolation and (try to... 😬 ) make things consistent between developer machines. e.g. We use aws-sam-cli + aws-sam-cli-local against localstack for some of our testsuites.

It would be great to see this treated as a bug which can be fixed by aws-sam-cli supporting (or simply requiring) pyyaml 6.x

That there's a viable workaround may have bearing on the urgency, of course.

So far, I haven't yet figured out the tox-appropriate way to add a dependency to the build environment (pre-install_command), but I'll post back if I work it out to help any other tox users.

[sirosen]

The only fix I could get working in tox short of doing something very silly was to put the install steps into commands_pre.

I don't know if this has bearing here or not, but awscli updated their pyyaml bounds in part for exactly this issue (aws/aws-cli#8037).

[realFranco]

Hello, I observe a PR that expose the upgrade for the PyYAML package, see: #5549

If that merge were correctly solve the problem as I saw on the project "aws/aws-cli" then this thread could be successfully close.

[realFranco]

The PyYAML package will fail to update on this current scenario.