Isolate celery tests to separate container #50952
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens. Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action. If we see that we are still hanging despite the isolation, we can later introduce more debug logging for **just** the celery container run.
potiuk
requested review from
amoghrajesh,
ashb,
gopidesupavan and
jedcunningham
as code owners
potiuk
commented
May 22, 2025
potiuk
added
the
backport-to-v3-1-test
amoghrajesh
approved these changes
May 22, 2025
Contributor
amoghrajesh
left a comment
amoghrajesh
left a comment
There was a problem hiding this comment.
Yeah lets isolate and try to fix it
amoghrajesh
reviewed
May 22, 2025
gopidesupavan
approved these changes
May 22, 2025
Member
gopidesupavan
left a comment
gopidesupavan
left a comment
There was a problem hiding this comment.
cool, hope to see celery happy now :)
Member
Author
I am really hopeful here. I've looked trough about 40 failed scheduled runs and I never saw the celery issue when running "lowest deps" tests - it seems to only fail when we run the regular tests together with other providers, so it seems that it is reallly side-effect of some other tests causing it - hopefully when we run it in isolation the problem will be gone (but it might be that the side effect will affect other tests ... who knows.. |
Member
Author
|
Ok. So far - so good first pass was full-green. Which is a good sign. |
Backport failed to create: v3-0-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker fb8c877 v3-0-testThis should apply the commit to the v3-0-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continue |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
May 22, 2025
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens. Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action. If we see that we are still hanging despite the isolation, we can later introduce more debug logging for **just** the celery container run. (cherry picked from commit fb8c877) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
potiuk
added a commit
that referenced
this pull request
May 22, 2025
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens. Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action. If we see that we are still hanging despite the isolation, we can later introduce more debug logging for **just** the celery container run. (cherry picked from commit fb8c877)
dadonnelly316
pushed a commit
to dadonnelly316/airflow
that referenced
this pull request
May 26, 2025
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens. Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action. If we see that we are still hanging despite the isolation, we can later introduce more debug logging for **just** the celery container run.
kaxil
pushed a commit
that referenced
this pull request
Jun 3, 2025
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens. Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action. If we see that we are still hanging despite the isolation, we can later introduce more debug logging for **just** the celery container run. (cherry picked from commit fb8c877)
sanederchik
pushed a commit
to sanederchik/airflow
that referenced
this pull request
Jun 7, 2025
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens. Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action. If we see that we are still hanging despite the isolation, we can later introduce more debug logging for **just** the celery container run.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The celery tests hang intermittently from time to time, and it's rather difficult to pin-point the root cause. This PR attempts to isolate the tests and make them fail faster in case the problem happens.
Currently, after some recent refactoring - none of the tests usually run longer that 18-19 minutes, so we can set much lower timeouts for the test job - 30 minutes "soft" timeout (SIGTERM sent to stop the container and dump logs) and 35 minutes for "hard" failure of GitHub Action.
If we see that we are still hanging despite the isolation, we can later introduce more debug logging for just the celery container run.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.