Skip to content

Better parallellise Other/Operator tests #35002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Oct 18, 2023
Merged

Better parallellise Other/Operator tests #35002

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions Dockerfile.ci
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1090,8 +1090,8 @@ if [[ ${REMOVE_ARM_PACKAGES:="false"} == "true" ]]; then
python "${IN_CONTAINER_DIR}/remove_arm_packages.py"
fi

declare -a SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
ALL_TESTS ALL_PRESELECTED_TESTS ALL_OTHER_TESTS
declare -a SELECTED_TESTS CLI_TESTS API_TESTS OPERATORS_TESTS ALWAYS_TESTS PROVIDERS_TESTS \
CORE_TESTS WWW_TESTS ALL_TESTS ALL_PRESELECTED_TESTS ALL_OTHER_TESTS

function find_all_other_tests() {
local all_tests_dirs
Expand Down Expand Up @@ -1120,6 +1120,7 @@ else
API_TESTS=("tests/api_experimental" "tests/api_connexion" "tests/api_internal")
PROVIDERS_TESTS=("tests/providers")
ALWAYS_TESTS=("tests/always")
OPERATORS_TESTS=("tests/operators")
CORE_TESTS=(
"tests/core"
"tests/executors"
Expand All @@ -1142,6 +1143,7 @@ else
"${PROVIDERS_TESTS[@]}"
"${CORE_TESTS[@]}"
"${ALWAYS_TESTS[@]}"
"${OPERATORS_TESTS[@]}"
"${WWW_TESTS[@]}"
"${SYSTEM_TESTS[@]}"
)
Expand All @@ -1163,6 +1165,8 @@ else
SELECTED_TESTS=("${CORE_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "Always" ]]; then
SELECTED_TESTS=("${ALWAYS_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "Operators" ]]; then
SELECTED_TESTS=("${OPERATORS_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "WWW" ]]; then
SELECTED_TESTS=("${WWW_TESTS[@]}")
elif [[ ${TEST_TYPE:=""} == "Helm" ]]; then
Expand Down Expand Up @@ -1226,8 +1230,8 @@ else
exit 1
fi
fi
readonly SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
ALL_TESTS ALL_PRESELECTED_TESTS
readonly SELECTED_TESTS CLI_TESTS API_TESTS OPERATORS_TESTS ALWAYS_TESTS PROVIDERS_TESTS \
CORE_TESTS WWW_TESTS ALL_TESTS ALL_PRESELECTED_TESTS

if [[ ${TEST_TYPE:=""} == "Long" ]]; then
EXTRA_PYTEST_ARGS+=(
Expand Down Expand Up @@ -1256,7 +1260,6 @@ echo "Running tests ${SELECTED_TESTS[*]}"
echo

ARGS=("${EXTRA_PYTEST_ARGS[@]}" "${SELECTED_TESTS[@]}")

if [[ ${RUN_SYSTEM_TESTS:="false"} == "true" ]]; then
"${IN_CONTAINER_DIR}/run_system_tests.sh" "${ARGS[@]}"
else
Expand Down
6 changes: 4 additions & 2 deletions dev/breeze/SELECTIVE_CHECKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,13 @@ We have the following unit test types that can be selectively disabled/enabled b
content of the incoming PR:

* Always - those are tests that should be always executed (always folder)
* Operators - tests for the operators (operators folder)
* Core - for the core Airflow functionality (core folder)
* API - Tests for the Airflow API (api and api_connexion folders)
* CLI - Tests for the Airflow CLI (cli folder)
* WWW - Tests for the Airflow webserver (www folder)
* Providers - Tests for all Providers of Airflow (providers folder)
* Other - all other tests remaining after the above tests are selected

We also have several special kinds of tests that are not separated by packages, but they are marked with
pytest markers. They can be found in any of those packages and they can be selected by the appropriate
Expand Down Expand Up @@ -95,8 +97,8 @@ The logic implements the following rules:
* if there are any changes to "common" provider code not belonging to any provider (usually system tests
or tests), then tests for all Providers are run
* The specific unit test type is enabled only if changed files match the expected patterns for each type
(`API`, `CLI`, `WWW`, `Providers`). The `Always` test type is added always if any unit tests are run.
`Providers` tests are removed if current branch is different than `main`
(`API`, `CLI`, `WWW`, `Providers`, `Operators`). The `Always` test type is added always if any unit
tests are run. `Providers` tests are removed if current branch is different than `main`
* If there are no files left in sources after matching the test types and Kubernetes files,
then apparently some Core/Other files have been changed. This automatically adds all test
types to execute. This is done because changes in core might impact all the other test types.
Expand Down
2 changes: 1 addition & 1 deletion dev/breeze/src/airflow_breeze/commands/testing_commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ def docker_compose_tests(


TEST_PROGRESS_REGEXP = r"tests/.*|.*=====.*"
PERCENT_TEST_PROGRESS_REGEXP = r"^tests/.*\[[ \d%]*\].*"
PERCENT_TEST_PROGRESS_REGEXP = r"^tests/.*\[[ \d%]*\].*|^\..*\[[ \d%]*\].*"


def _run_test(
Expand Down
1 change: 1 addition & 0 deletions dev/breeze/src/airflow_breeze/global_constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ class SelectiveUnitTestTypes(Enum):
CLI = "CLI"
CORE = "Core"
OTHER = "Other"
OPERATORS = "Operators"
PROVIDERS = "Providers"
WWW = "WWW"

Expand Down
9 changes: 8 additions & 1 deletion dev/breeze/src/airflow_breeze/utils/selective_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,10 @@ def __hash__(self):
r"^airflow/cli",
r"^tests/cli",
],
SelectiveUnitTestTypes.OPERATORS: [
r"^airflow/operators",
r"^tests/operators",
],
SelectiveUnitTestTypes.PROVIDERS: [
r"^airflow/providers/",
r"^tests/system/providers/",
Expand Down Expand Up @@ -584,6 +588,9 @@ def _get_test_types_to_run(self) -> list[str]:
matched_files.update(
self._select_test_type_if_matching(candidate_test_types, SelectiveUnitTestTypes.CLI)
)
matched_files.update(
self._select_test_type_if_matching(candidate_test_types, SelectiveUnitTestTypes.OPERATORS)
)
matched_files.update(
self._select_test_type_if_matching(candidate_test_types, SelectiveUnitTestTypes.API)
)
Expand Down Expand Up @@ -670,7 +677,7 @@ def parallel_test_types_list_as_string(self) -> str | None:
self._extract_long_provider_tests(current_test_types)

# this should be hard-coded as we want to have very specific sequence of tests
sorting_order = ["Core", "Providers[-amazon,google]", "Other", "Providers[amazon]", "WWW"]
sorting_order = ["Operators", "Core", "Providers[-amazon,google]", "Providers[amazon]", "WWW"]
sort_key = {item: i for i, item in enumerate(sorting_order)}
# Put the test types in the order we want them to run
return " ".join(sorted(current_test_types, key=lambda x: (sort_key.get(x, len(sorting_order)), x)))
Expand Down
85 changes: 53 additions & 32 deletions dev/breeze/tests/test_selective_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,26 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
id="Only API tests and DOCS should run",
)
),
(
pytest.param(
("airflow/operators/file.py",),
{
"affected-providers-list-as-string": None,
"all-python-versions": "['3.8']",
"all-python-versions-list-as-string": "3.8",
"python-versions": "['3.8']",
"python-versions-list-as-string": "3.8",
"image-build": "true",
"needs-helm-tests": "false",
"run-tests": "true",
"run-amazon-tests": "false",
"docs-build": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Operators Always",
},
id="Only Operator tests and DOCS should run",
)
),
(
pytest.param(
(
Expand Down Expand Up @@ -277,9 +297,9 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
"Other Providers[amazon] WWW "
"API Always CLI Providers[google]",
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Other Providers[google]",
},
id="Everything should run - including all providers and upgrading to "
"newer requirements as setup.py changed and all Python versions",
Expand All @@ -300,9 +320,9 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-amazon-tests": "true",
"docs-build": "true",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
"Other Providers[amazon] WWW "
"API Always CLI Providers[google]",
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Other Providers[google]",
},
id="Everything should run and upgrading to newer requirements as dependencies change",
)
Expand Down Expand Up @@ -409,9 +429,9 @@ def test_expected_output_pull_request_main(
"docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
"Other Providers[amazon] WWW "
"API Always CLI Providers[google]",
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Other Providers[google]",
},
id="Everything should run including all providers when full tests are needed",
)
Expand All @@ -436,9 +456,9 @@ def test_expected_output_pull_request_main(
"docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
"Other Providers[amazon] WWW "
"API Always CLI Providers[google]",
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Other Providers[google]",
},
id="Everything should run including full providers when full "
"tests are needed even with different label set as well",
Expand All @@ -461,9 +481,9 @@ def test_expected_output_pull_request_main(
"docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"full-tests-needed": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
"Other Providers[amazon] WWW "
"API Always CLI Providers[google]",
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Other Providers[google]",
},
id="Everything should run including full providers when"
"full tests are needed even if no files are changed",
Expand All @@ -488,7 +508,7 @@ def test_expected_output_pull_request_main(
"full-tests-needed": "true",
"skip-provider-tests": "true",
"upgrade-to-newer-dependencies": "false",
"parallel-test-types-list-as-string": "Core Other WWW API Always CLI",
"parallel-test-types-list-as-string": "Operators Core WWW API Always CLI Other",
},
id="Everything should run except Providers when full tests are needed for non-main branch",
)
Expand Down Expand Up @@ -600,7 +620,7 @@ def test_expected_output_full_tests_needed(
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "Core Other WWW API Always CLI",
"parallel-test-types-list-as-string": "Operators Core WWW API Always CLI Other",
},
id="All tests except Providers should run if core file changed in non-main branch",
),
Expand Down Expand Up @@ -708,6 +728,7 @@ def test_expected_output_pull_request_v2_3(
pytest.param(
(
"airflow/cli/file.py",
"airflow/operators/file.py",
"airflow/www/file.py",
"airflow/api/file.py",
),
Expand All @@ -723,9 +744,9 @@ def test_expected_output_pull_request_v2_3(
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "true",
"parallel-test-types-list-as-string": "WWW API Always CLI",
"parallel-test-types-list-as-string": "Operators WWW API Always CLI",
},
id="No providers tests should run if only CLI/API/WWW file changed",
id="No providers tests should run if only CLI/API/Operators/WWW file changed",
),
pytest.param(
("airflow/models/test.py",),
Expand All @@ -741,9 +762,9 @@ def test_expected_output_pull_request_v2_3(
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Providers[google]",
"API Always CLI Other Providers[google]",
},
id="Tests for all providers should run if model file changed",
),
Expand All @@ -761,11 +782,11 @@ def test_expected_output_pull_request_v2_3(
"run-kubernetes-tests": "false",
"upgrade-to-newer-dependencies": "false",
"skip-provider-tests": "false",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Providers[google]",
"API Always CLI Other Providers[google]",
},
id="Tests for all providers should run if any other than API/WWW/CLI file changed.",
id="Tests for all providers should run if any other than API/WWW/CLI/Operators file changed.",
),
],
)
Expand Down Expand Up @@ -800,9 +821,9 @@ def test_expected_output_pull_request_target(
"docs-build": "true",
"docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Providers[google]",
"API Always CLI Other Providers[google]",
},
id="All tests run on push even if unimportant file changed",
),
Expand All @@ -820,7 +841,7 @@ def test_expected_output_pull_request_target(
"docs-build": "true",
"docs-filter-list-as-string": "--package-filter apache-airflow --package-filter docker-stack",
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": "Core Other WWW API Always CLI",
"parallel-test-types-list-as-string": "Operators Core WWW API Always CLI Other",
},
id="All tests except Providers and Helm run on push"
" even if unimportant file changed in non-main branch",
Expand All @@ -839,9 +860,9 @@ def test_expected_output_pull_request_target(
"docs-build": "true",
"docs-filter-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"upgrade-to-newer-dependencies": "true",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] Other "
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Providers[google]",
"API Always CLI Other Providers[google]",
},
id="All tests run on push if core file changed",
),
Expand Down Expand Up @@ -892,9 +913,9 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
"upgrade-to-newer-dependencies": "true"
if github_event in [GithubEvents.PUSH, GithubEvents.SCHEDULE]
else "false",
"parallel-test-types-list-as-string": "Core Providers[-amazon,google] "
"Other Providers[amazon] WWW "
"API Always CLI Providers[google]",
"parallel-test-types-list-as-string": "Operators Core Providers[-amazon,google] "
"Providers[amazon] WWW "
"API Always CLI Other Providers[google]",
},
str(stderr),
)
Expand Down
4 changes: 2 additions & 2 deletions images/breeze/output-commands-hash.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,5 +68,5 @@ static-checks:19926b8fcea5784b28d4a0d99865363c
testing:docker-compose-tests:fd154a058082fcfda12eb877a9a89338
testing:helm-tests:0669be17b744ba057adbf38681bd8e68
testing:integration-tests:f57fb275b9733a6226601f8095ad4de0
testing:tests:1cf0f4580ecd2a2321b341b1dd411bee
testing:31d3b4f2385aed509adee68201906ed4
testing:tests:ce35463d1c67f3416eba77d3201c90ac
testing:9c9f35945852ff48da2f5aba6447463e
Loading