Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Sep 4, 2024

The "lowest" dependency tests had "hard-coded" exclusion for those providers that were excluded in specific python versions.

This change automates it:

  • selective checks produce dictionary python -> excluded versions
  • the dictionary (as string) is passed to unit tests in actions
  • the dictionary (as string) is passed to breeze container as EXCLUDED_PROVIDERS variable
  • "is_provider_excluded.py" checks if the provider is excluded and exits with success if so

Related: #41555


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk added full tests needed We need to run full set of tests for this PR to merge all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs labels Sep 4, 2024
@potiuk potiuk closed this Sep 4, 2024
@potiuk potiuk reopened this Sep 4, 2024
@potiuk potiuk force-pushed the fix-individual-pytests-when-ignoring-providers branch 2 times, most recently from d49157d to 6d5c9c3 Compare September 4, 2024 11:16
@potiuk
Copy link
Member Author

potiuk commented Sep 4, 2024

That one should also speed up "lowest dependencies" test in main - because we - erroneously excluded papermill from Python 3.12 (where Papermill had been added back a while ago) while we did not exclude apache.beam, so apache.beam tests actually (due to the way how the scripts work) would run ALL tests in a single thread.

@topherinternational
Copy link
Contributor

I think the followup to #41967 I am writing will also address this problem.

@potiuk potiuk added the include success outputs Whether to include success outputs in test results label Sep 4, 2024
@potiuk
Copy link
Member Author

potiuk commented Sep 4, 2024

I think the followup to #41967 I am writing will also address this problem.

We can compare and choose better one :)

@potiuk potiuk force-pushed the fix-individual-pytests-when-ignoring-providers branch 3 times, most recently from 0102e45 to 5ecbaaf Compare September 4, 2024 13:01
@potiuk
Copy link
Member Author

potiuk commented Sep 4, 2024

Actually this one also solves "main" failures of Python 3.12 "Lowest dependencies" tests that I've seen recently!

@topherinternational
Copy link
Contributor

I think the followup to #41967 I am writing will also address this problem.

We can compare and choose better one :)

Or we can have defense in depth :D

@potiuk potiuk force-pushed the fix-individual-pytests-when-ignoring-providers branch 2 times, most recently from 63075e2 to 34917b0 Compare September 4, 2024 16:05
The "lowest" dependency tests had "hard-coded" exclusion for
those providers that were excluded in specific python versions.

This change automates it:

* selective checks produce dictionary python -> excluded versions
* the dictionary (as string) is passed to unit tests in actions
* the dictionary (as string) is passed to breeze container
  as EXCLUDED_PROVIDERS variable
* "is_provider_excluded.py" checks if the provider is excluded
  and exits with success if so

Related: apache#41555
@potiuk potiuk force-pushed the fix-individual-pytests-when-ignoring-providers branch from 34917b0 to 52227f2 Compare September 4, 2024 18:11
@potiuk potiuk merged commit 28e7213 into apache:main Sep 4, 2024
@potiuk potiuk deleted the fix-individual-pytests-when-ignoring-providers branch September 4, 2024 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs area:dev-tools full tests needed We need to run full set of tests for this PR to merge include success outputs Whether to include success outputs in test results

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants