Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented May 27, 2024

When we run all integration tests in CI together on self-hosted runners they are somewhat flaky, however when those tests are run individually on public runners, they are both fast and stable. Also the integration test list has been somewhat duplicated - you had to remember about adding individual integration tests to the separate integration-tests workflow when you added such integration tests.

This PR solves all those problems:

  • we always run integration tests individually
  • they are all run on public runners no matter what build they are on
  • they are run as matrix of jobs and list of tests is extracted via selective checks outputs - no more duplication

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Copy link
Member Author

potiuk commented May 27, 2024

I think that should address the recent flakiness observed in main for integration tests. Looks like when we run all integrations together. even our self-hosted runners do not cope well with having all of them started together. Those tests are generally quite fast (< 2 minutes) - so it should be perfectly fine to run them as matrix jobs on Public runners instead - this way we save some self-hsoted runners in main and we should gain back the stablity.

@potiuk potiuk added the full tests needed We need to run full set of tests for this PR to merge label May 27, 2024
@potiuk potiuk force-pushed the simplify-integration-test-running branch 2 times, most recently from 391954d to 8148bba Compare May 27, 2024 09:46
@potiuk
Copy link
Member Author

potiuk commented May 27, 2024

BTW, This one should address the last "frequent" flakiness observed in main recently.

@potiuk
Copy link
Member Author

potiuk commented May 27, 2024

This is how it will look like now:

image

There will be more jobs, but they also will be rather fast and stable

@potiuk potiuk force-pushed the simplify-integration-test-running branch from 8148bba to 85c460b Compare May 27, 2024 09:56
When we run all integration tests in CI together on self-hosted
runners they are somewhat flaky, however when those tests are run
individually on public runners, they are both fast and stable. Also
the integration test list has been somewhat duplicated - you had to
remember about adding individual integration tests to the separate
integration-tests workflow when you added such integration tests.

This PR solves all those problems:

* we always run integration tests individually
* they are all run on public runners no matter what build they are
  on
* they are run as matrix of jobs and list of tests is extracted
  via selective checks outputs - no more duplication
@potiuk potiuk force-pushed the simplify-integration-test-running branch from 85c460b to c6f04aa Compare May 27, 2024 10:07
@potiuk potiuk merged commit 7cefd0a into apache:main May 27, 2024
@potiuk potiuk deleted the simplify-integration-test-running branch May 27, 2024 11:01
fdemiane pushed a commit to fdemiane/airflow that referenced this pull request Jun 6, 2024
When we run all integration tests in CI together on self-hosted
runners they are somewhat flaky, however when those tests are run
individually on public runners, they are both fast and stable. Also
the integration test list has been somewhat duplicated - you had to
remember about adding individual integration tests to the separate
integration-tests workflow when you added such integration tests.

This PR solves all those problems:

* we always run integration tests individually
* they are all run on public runners no matter what build they are
  on
* they are run as matrix of jobs and list of tests is extracted
  via selective checks outputs - no more duplication
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
When we run all integration tests in CI together on self-hosted
runners they are somewhat flaky, however when those tests are run
individually on public runners, they are both fast and stable. Also
the integration test list has been somewhat duplicated - you had to
remember about adding individual integration tests to the separate
integration-tests workflow when you added such integration tests.

This PR solves all those problems:

* we always run integration tests individually
* they are all run on public runners no matter what build they are
  on
* they are run as matrix of jobs and list of tests is extracted
  via selective checks outputs - no more duplication
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants