Production images on CI are now built from packages

[potiuk]

So far, the production images of Airflow were using sources
when they were built on CI. This PR changes that, to build
airflow + providers packages first and install them
rather than use sources as installation mechanism.

Part of #12261

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

This is the change we discussed some time ago and captured in #12261. The Production image during CI build is now built from wheels rather than directly from sources to reflect a "real" installation case.

This change modifies the "CI" production image build to first:

1. Using 'pip download" + constraints file it downloads all .wheel packages that are needed to install airflow with the chosen "extras" for the production image

2. Buillds all providers packages that are selected for the production

3. Builds airflow .whl package

4. Prepares the production image using those wheel files rather than PyPI.

This way this production image for development tests reflects the exact production "content" - all packages (including airflow) are installed, but at the same time we are using latest sources to build those packages, so the image can also be used in K8S tests because it is build from the current (PR) sources.

After preparing the image I am also checking if all the providers are installed as expected

[potiuk]

@ashb -> Since you were asking for it. The follow -up to this one will be to build the "DockerHub" images, but that will be a separate PR.

Those will be built differently depending on the image:

• The DockerHub "master" images will be built in the same way as this PR (whl packages 'pip downloaded' for external dependencies and .whl packages built from current master sources

• The "official" (non-master) images wil be built from the released PyPI packages rather than wheels,

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

Hey everyone - this looks like is going to be green now. It changes the production image to be build using packages rather than directly Airflow Sources.

This image is now built and verified in the CI and follow up change after this one is merged is to also build image from locally built packages in 2.0 for DockerHub builds. Those images are automatically verified (and #12718 add further checks including nicer way of pip check).

The checks include provider's discovery.

I also described how to use breeze to build and install provider packages very easily in development environment.
It takes three commands:

• ./breeze prepare-provider-packages [PACKAGE ...]
• ./breeze prepare-airflow-packages
• ./breeze --install-airflow-version wheel --install-packages-from-dist --skip-mounting-local-sources

Looking forward to comments.

[potiuk]

I hope this time no Exit 137 and it gets green

[potiuk]

Same here @ashb - it would be great to get one before RC as this way we actually test "production-like" setup.

[ashb]

Few minor comments only -- nothing major

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[github-actions]

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.

[github-actions]

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.