Speed up verification and instalation of providers in CI

[potiuk]

For PRs that are just changing selected providers, there is no need to install and verify all providers, just those affected should be installed (and full verification should only be done when all providers are being built - so generally speaking, when Core PRs are run or when canary builds are run.

This PR distinguishes those two cases:

• installing and verification when all providers are selected
• installing only affected providers when only subset of them are affected

Also a bug was found where core file changes did not trigger provider docs build during implementation. This has been fixed

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

Yet another improvement of CI speed - this time with testing provider packages. For Provider-only changes, a lot of contributors should be able to save a lot of build time for just building and installing the affected provider packages, rather than all of them

[potiuk]

I also found one case where core-changes did not trigger docs build for Providers, which should not be the case, when we change core files, potentilly all providers are affected (for tests, docs building and package preparation).

[hussein-awala]

I think we should test all the providers when we change something in airflow/models like the base sensor or operator. I don't see a test for that. WDYT?

[potiuk]

I think we should test all the providers when we change something in airflow/models like the base sensor or operator. I don't see a test for that. WDYT?

It's covered by the test below. This is actually the issue I mentioned with:

I also found one case where core-changes did not trigger docs build for Providers, which should not be the case, when we change core files, potentilly all providers are affected (for tests, docs building and package preparation).

        pytest.param(
            (
                "airflow/file.py",
                "tests/providers/google/file.py",
            ),
            {
                "affected-providers": "",
                "all-python-versions": "['3.7']",
                "all-python-versions-list-as-string": "3.7",
                "image-build": "true",
                "needs-helm-tests": "false",
                "run-tests": "true",
                "docs-build": "true",
                "docs-filter": "",
                "run-kubernetes-tests": "false",
                "upgrade-to-newer-dependencies": "false",
                "skip-provider-tests": "false",
                "parallel-test-types": "Core Providers[-amazon,google] Other Providers[amazon] WWW "
                "API Always CLI Providers[google]",
            },
            id="All tests should run if core file changed",
        ),

We already had this as a case where we decide that we should run all provider tests where "essential" airflow core chnages. This means if there is a change that is "airflow" sources but:

• not an API
• not a WWW
• not a CLI
• not providers
• not a chart/kubernetes_tests files

We decide to:

• run all Providers tests: "run-tests"= "true", parallel-tests-types has "Providers" (split into three groups now)
• build all Providers docs (fixed in this PR) -> docs_filter = "", docs-build="true"
• (added in this PR) build and install all providers -> afffected_providers = "" (means build all providers to check them)

Yeah, likely we could still refactor it (I think it would have been better to distinguish the values of those two for case where we skip them alltogether but for now it is what it is - for now guarded by the test cases).

[potiuk]

Yeah, likely we could still refactor it (I think it would have been better to distinguish the values of those two for case where we skip them alltogether but for now it is what it is - for now guarded by the test cases).

After the comment -yeah I think I might want to refactor that one .. I also see how convoluted it got and I have some idea how to make it "cleaner" a bit.

[potiuk]

All right. I think this should be much better now:

• I made names of the outputs more consistent ("list-as-string" where we use string to pass lists - and explained why)
• I added optionality for those outputs where empty list could previously mean both None and All.

[potiuk]

Running last tests, and I think in the current version of selective checks and the tests show the intentions much better and the code is cleaner (extracted the important bits):

ALL_DOCS_SELECTED_FOR_BUILD = ""
ALL_PROVIDERS_AFFECTED = ""

        pytest.param(
            ("INTHEWILD.md",),
            {
                "affected-providers-list-as-string": None,
                "run-tests": "false",
                "docs-build": "false",
                "docs-filter-list-as-string": None,
                "skip-provider-tests": "true",
                "parallel-test-types-list-as-string": None,
            },
            id="Nothing should run if only non-important files changed",
        ),
        pytest.param(
            ("tests/system/any_file.py",),
            {
                "affected-providers-list-as-string": None,
                "run-tests": "true",
                "docs-build": "true",
                "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
                "skip-provider-tests": "true",
                "parallel-test-types-list-as-string": "Always",
            },
            id="Only Always and docs build should run if only system tests changed",
        ),
        pytest.param(
            (
                "airflow/cli/test.py",
                "chart/aaaa.txt",
                "tests/providers/google/file.py",
            ),
            {
                "affected-providers-list-as-string": "amazon apache.beam apache.cassandra "
                "cncf.kubernetes common.sql "
                "facebook google hashicorp microsoft.azure microsoft.mssql mysql "
                "oracle postgres presto salesforce sftp ssh trino",
                "run-tests": "true",
                "docs-build": "true",
                "docs-filter-list-as-string": "--package-filter apache-airflow --package-filter helm-chart "
                "--package-filter apache-airflow-providers-amazon "
                "--package-filter apache-airflow-providers-apache-beam "
                "--package-filter apache-airflow-providers-apache-cassandra "
                "--package-filter apache-airflow-providers-cncf-kubernetes "
                "--package-filter apache-airflow-providers-common-sql "
                "--package-filter apache-airflow-providers-facebook "
                "--package-filter apache-airflow-providers-google "
                "--package-filter apache-airflow-providers-hashicorp "
                "--package-filter apache-airflow-providers-microsoft-azure "
                "--package-filter apache-airflow-providers-microsoft-mssql "
                "--package-filter apache-airflow-providers-mysql "
                "--package-filter apache-airflow-providers-oracle "
                "--package-filter apache-airflow-providers-postgres "
                "--package-filter apache-airflow-providers-presto "
                "--package-filter apache-airflow-providers-salesforce "
                "--package-filter apache-airflow-providers-sftp "
                "--package-filter apache-airflow-providers-ssh "
                "--package-filter apache-airflow-providers-trino",
                "skip-provider-tests": "false",
                "parallel-test-types-list-as-string": "Providers[amazon] Always CLI "
                "Providers[apache.beam,apache.cassandra,cncf.kubernetes,common.sql,facebook,"
                "hashicorp,microsoft.azure,microsoft.mssql,mysql,oracle,postgres,presto,"
                "salesforce,sftp,ssh,trino] Providers[google]",
            },
            id="CLI tests and Google-related provider tests should run if cli/chart files changed",
        ),
        pytest.param(
            (
                "airflow/cli/file.py",
                "airflow/www/file.py",
                "airflow/api/file.py",
            ),
            {
                "affected-providers-list-as-string": None,
                "run-tests": "true",
                "docs-build": "true",
                "docs-filter-list-as-string": "--package-filter apache-airflow",
                "skip-provider-tests": "true",
                "parallel-test-types-list-as-string": "WWW API Always CLI",
            },
            id="No providers tests should run if only CLI/API/WWW file changed",
        ),
        pytest.param(
            ("airflow/models/test.py",),
            {
                "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
                "run-tests": "true",
                "docs-build": "true",
                "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
                "run-kubernetes-tests": "false",
                "skip-provider-tests": "false",
                "parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
                "Providers[amazon] WWW "
                "API Always CLI Providers[google]",
            },
            id="Tests for all providers should run if model file changed",
        ),
        pytest.param(
            ("airflow/file.py",),
            {
                "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
                "run-tests": "true",
                "docs-build": "true",
                "docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
                "skip-provider-tests": "false",
                "parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
                "Providers[amazon] WWW "
                "API Always CLI Providers[google]",
            },
            id="Tests for all providers should run if any other than API/WWW/CLI file changed.",
        ),

[potiuk]

cc: @hussein-awala -> I added the "models" specifically in tests - but it's basically the same as "any other" file in airflow (other than www/cli/api which we know are not involved with providers). I did not want to specifically limit models only because there are a number of files providers might depend on (exceptions, decorators, hooks, listeners, you name it) . Once we clarify the "actual" API that Providers can use, we could potentially enumerate those, but for now I assume that "any" change in those packages except API/CLI/WWW might impact providers directly.

It's now explicitly stated in the tests.

[hussein-awala]

looks good, LGTM