apache-airflow-providers-amazon added xmlsec as new dependency and pinned to a version that doesn't have wheels for new python versions

[phofl]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon=8.21.0

Apache Airflow version

2.8.3

Operating System

Mac/Ubuntu

Deployment

Other

Deployment details

No response

What happened

The 8.21.0 release added xmlsec as a required dependency to apache-airflow-providers-amazonn and pinned to a version that is 2 years old and doesn't have wheels for

• anything but windows
• python 3.11 and higher

and building the wheel fails.

What you think should happen instead

I found #39103 and it looks like the cause of this was an incompatibility. I can't see why adding it as a dependency is necessary instead of just constraining allowed versions

How to reproduce

mamba create -n airflow_test python=3.11
mamba activate airflow_test
pip install apache-airflow-providers-amazon

Anything else

Every time

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

The 8.21.0 release added xmlsec as a required dependency to apache-airflow-providers-amazonn and pinned to a version that is 2 years old and doesn't have wheels for

It is restricted to the version which didn't breaks current installation. As you might find there is open issue for resolve this, if you know how to fix it, feel free to raise a PR.

BTW, many of the packages do not have a wheel and required to built from the sources and appropriate dev packages.

cc: @vincbeck Do you think we need this is as main dependency for Amazon provider, and could we move it in some extra?

[eladkal]

Preferably we should solve the root issue #39103

[phofl]

Fixing the root cause sounds good, but currently it is not possible to install the amazon provider with any configuration that has Python >=3.11 or a non Windows machine on all Python Versions.

[o-nikolas]

Just had a look into this and I agree with Taragolis's sentiment here:

BTW, many of the packages do not have a wheel and required to built from the sources and appropriate dev packages.

When I did a test install (same as your steps, create a new virtual env based on 3.11 and install the provider package) I had several packages that needed to be built from source, xmlsec among them:

Building wheels for collected packages: xmlsec, methodtools, python-nvd3, unicodecsv, wirerope

the xmlsec build did fail for me, but only because I was missing required dev packages to build the source. After installing libxmlsec1, libxmlsec1-dev and python3.11-dev my installation worked just fine and was able to build a wheel locally for xmlsec.

Perhaps you are missing some development packages as well on your system which are inhibiting the wheel build from completing?

[potiuk]

I think @o-nikolas it's not as straightforward. I described it in #39103 and the problem is that xmlsec python bindings v 1.3.14 expects libxmlsec2 to be installed and used and not libxmlsec1. This might or might not be a problem for security at any point in time and it should be investigated if the dependency can be upgraded (or as @eladkal suggests - turned into an optional dependency and handle lack of xmlsec with optional amazon provider dependency + helpful instructions on what to do). Especially that - as I understand - it's only needed for maybe one integration in Amazon provider.

I am happy to guide anyone who would like to take on that task.

While yes - you can still build the wheel with installing the packages you mentioned, it's a blocker or unnecessary hurdle for people who would like to install amazon provider on anything else than Airflow PROD image - so while we currently have no problems with releasing airflow (because we limitted xmlsec python bindings), it simply makes it more difficult for those who install amazon provider on their own. Mostly up to the Amazon team to decide how to approach it.

[jedcunningham]

I think we just put the pin in the wrong spot to be honest.

• I don't see xmlsec being used in the AWS provider anywhere directly.
• It's in our CI image because of python3-saml.
• python3-saml is an optional extra for Amazon.

We should move the xmlsec pin to that extra instead I believe. PR #39528

[potiuk]

Nope - it's been added specifically on request and by @vincbeck - #35488 to support authentication for AWS

[potiuk]

And yes - it's likely a bit complex but It means that libxmlsec1 (system library) that is used and apparently needed by Amazon provider is clashing with llbxml python bindings that are used by python3-saml - because they need libxmlsec2.

So technically speaking python3-saml should not have this limit (it will work with libxmlsec2 if installed) - but amazon needs libxml1 and it clashes with xmlsec python bindings >= 1.3.14 that need libxmlsec2.