Airflow 2.6.1 Constraints Inconsistency prevents custom install of Sphinx

[jscheffl]

Apache Airflow version

2.6.1

What happened

See also discussion in Slack: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1684331895556709

We use Airflow based on docker image apache/airflow:2.6.1-python3.8 and build custom PIP packages on-top. Of course following the setup guide we source the constraints 3.8.txt alongside to keep a consistent setup.

With the upgrade to Airflow 2.6.1 we are trying to install Sphinx which is pinned to 5.3.0, this sources the dependency of docutils. Constraints pins this to 0.20. Install of Sphinx fails with a dependency error:

ERROR: Cannot install sphinx==5.3.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    sphinx 5.3.0 depends on docutils<0.20 and >=0.14
    The user requested (constraint) docutils==0.20

How

What you think should happen instead

Pip install of Sphinx should be possible with the standard constraints.

How to reproduce

Run:

• docker run -it apache/airflow:2.6.1-python3.8 bash
• pip install sphinx --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.6.1/constraints-3.8.txt"

Operating System

Ubuntu 20.04

Versions of Apache Airflow Providers

Default provider packages

Deployment

Official Apache Airflow Helm Chart

Deployment details

n/a

Anything else

Workaround: Patch docutils down to 0.19 manually in downloaded constraints-3.8.txt
This was the version used in Airflow 2.6.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Thanks - I will have a look at that - that sounds very strange indeed.

[jscheffl]

Currently Sphinx is defined in constraints to 5.3.0, taking a look to changelogs the most recent version 7.0.1 states:

Release 7.0.1 (released May 12, 2023)
Dependencies

• #11411: Support Docutils 0.20. Patch by Adam Turner.

Don't know how, just in setup.py I see sphinx>=5.2.0 - is this the point to upgrade?
Ah, but I see in setup.py there is anot about problems of docutils anyway... but was 0.19 in Airflow 2.6.0?

[potiuk]

The problem is sphinx-rtd-theme which is used to build our docs:

sphinx-rtd-theme 1.2.0 has requirement docutils<0.19, but you have docutils 0.20.1.
sphinx-rtd-theme 1.2.0 has requirement sphinx<7,>=1.6, but you have sphinx 7.0.1.

[potiuk]

This is if I try to manually update sphinx to latest. Seems like a number of projects have the same issue readthedocs/sphinx_rtd_theme#1463

[potiuk]

PR seem to be in review (and green) to fix it. Opened 3 weeks ago readthedocs/sphinx_rtd_theme#1464

[potiuk]

Encouraging it is most welcome

[jscheffl]

Just did a status check, fix in readthedocs/sphinx_rtd_theme#1464 is still not merged :-(
Airflow 2.6.2 though has changed the error message a bit to:

The conflict is caused by:
    sphinx 5.3.0 depends on docutils<0.20 and >=0.14
    The user requested (constraint) docutils==0.20.1

[potiuk]

The fix to upgrade sphinx version to 7 is coming in #33156