AIP-79 Support Airflow 2.x plugins in fast api

[vincbeck]

To support Airflow 2.x plugins in Airflow 3 the goal is to embed a minimal version of Airflow 2 in fast api using WSGIMiddleware. The Airflow 2.x plugins will run in this application as they are running today in Airflow 2. WSGIMiddleware allows running a WSGI in the fast api application. This Flask application will be accessible under the path /pluginsv2.

The plan is to:

1. (This PR) Copy files from core Airflow to Fab provider to create a minimal version of Airflow 2 in Fab provider. This minimal version of Airflow 2 should only contain core elements needed by the plugins to run. The goal is not to copy the entire Flask application but only a more simplified version of it. This minimal version of Airflow 2 in Fab provider should be self contained in Fab provider, they must not depend on Flask-related objects defined in core Airflow. The goal is, when the Flask application will be deleted from core Airflow, it should be possible because the mini Airflow 2 in Fab provider does not depend on it.
2. (This PR as well) Embed the mini Airflow 2 defined in Fab provider in the fast api application so that Airflow 2.x plugins will be available in Airflow 3
3. When the new Airflow 3 UI becomes the default UI and no fallback to the old UI is possible, remove the Flask application and all related resources (e.g. views) from core Airflow.

The idea is, if you do not have any Airflow 2.x plugin defined in your environment, you should not have Flask as dependency. On the opposite, if you do have some Airflow 2.x plugins in your environment, fab provider will have to be installed and all the code needed to run the Flask application will be in this provider.

The experience for the user will not be ideal because these plugins use the Airflow 2 UI, therefore the UI experience will be inconsistent between the Airflow pages (Airflow 3 UI) and the plugins (Airflow 2 UI). A warning message will be added in the embedded Flask application to let the user knows about this weird experience but also to recommend them to migrate their plugins to Airflow 3 plugins.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vincbeck]

@jedcunningham

[jedcunningham]

Instead of waiting for #2 to happen, let's just copy/trim whatever minimal FAB stuff we need into the FAB provider now - just leave it completely separate from the old UI. Then we aren't blocked at all.

[vincbeck]

Instead of waiting for #2 to happen, let's just copy/trim whatever minimal FAB stuff we need into the FAB provider now - just leave it completely separate from the old UI. Then we aren't blocked at all.

That sounds good to me. That's a good idea indeed, we can save some time by doing this. Should we do it in a separate PR though? This one sets up the mechanism to embed Airflow 2 in Airflow 3 and the other PR(s) would be copying over the Flask app to Fab provider

[jedcunningham]

I'd kinda prefer we don't mount the "whole" AF2 app as-is in AF3, even if we do view it as temporary.

[vincbeck]

I'd kinda prefer we don't mount the "whole" AF2 app as-is in AF3, even if we do view it as temporary.

Alright, I'll do it in this PR then

[vincbeck]

This is a preview of the page when accessing a plugin.

URL is <fast_api_endpoint>/pluginsv2/

[pierrejeambrun]

Amazing work.

Can we also reuse that same application to do the flask auth server ? (When people are using FABAuthManager and we need to reach for the flask webserver) ?

[vincbeck]

Can we also reuse that same application to do the flask auth server ? (When people are use FABAuthManager and we need to reach for the flask webserver) ?

I was wondering the same and I think we should do that

[vincbeck]

Tests are passing, let me know if there are more comments/feedbacks

[jedcunningham]

Overall looks good. Couple things (like the link to the AF3 homepage) that we can sort out later.

[jedcunningham]

Whoo! Nice!