Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Feb 22, 2024

Our CI sometimes reuses same runner for several jobs (this is until we switch to --ephemeral runners) and this might lead to some side-effects between runners. When the two runners are run for different branches, and airflow is locally installed, it might lead to configuration being written by one version and used by another. This - for example - made
https://github.com/apache/airflow/actions/runs/8011411765/job/21884630103 to fail.

This PR adds extra cleanup to breeze ci free-space command and add extra step in breeze cleanup to remove installed airflow and the whole airflow home.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Our CI sometimes reuses same runner for several jobs (this is until
we switch to --ephemeral runners) and this might lead to some
side-effects between runners. When the two runners are run for
different branches, and airflow is locally installed, it might
lead to configuration being written by one version and used by
another. This - for example - made
https://github.com/apache/airflow/actions/runs/8011411765/job/21884630103
to fail.

This PR adds extra cleanup to `breeze ci free-space` command and
add extra step in `breeze cleanup` to remove installed airflow
and the whole airflow home.
@potiuk potiuk merged commit 2f4dda9 into apache:main Feb 23, 2024
@potiuk potiuk deleted the add-local-airflow-cleanup branch February 23, 2024 07:05
@drajguru
Copy link
Contributor

I used breeze cleanup just now and realized that it deletes everything under the airflow path. Since this also includes the repo .git folder this could be risky for developers if they expect it to clean up docker and build related stuff.

@potiuk
Copy link
Member Author

potiuk commented Feb 26, 2024

It's a very bad idea to checkout airflow in airflow folder in your HOME directory, because this ${HOME}/airflow is where airflow keeps everything during runtime - you should not mix those two. And I also believe breeze warns you very loudly when you do that so that you checkout your directory elsewhere. Don't do that.

@potiuk
Copy link
Member Author

potiuk commented Feb 26, 2024

Or so I thought:

PR adding protection against such checkout is here: #37697

After it's merged, when you checkout your Airlfow in HOME/airflow and use breeze you will see this (and you won't be able to run breeze)

Screenshot 2024-02-26 at 08 02 13 :

@drajguru
Copy link
Contributor

Ok thanks I wasn't aware of this

@potiuk
Copy link
Member Author

potiuk commented Feb 26, 2024

Ok thanks I wasn't aware of this

So now it will be clear to anyone who is not aware :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants