WIP - Fix k8s executor when home path different#26449
WIP - Fix k8s executor when home path different#26449dstandish wants to merge 1 commit intoapache:mainfrom
Conversation
Since apache#21877 we no longer parse the dag as part of --local task run. As a consequence it reads dag file location from the database. But if the dag file is in a different path on the k8s executor worker, then this will be the wrong path. We can fix this by respecting the subdir from the top level command. One problem is, k8s executor, the way it builds the pod args, it sets subdir=DAGS_FOLDER which means that all dags in the folder get parsed, which, we should avoid somehow. So, have to work on this more and see whether we can improve that. It may be easier to just store the relative loc in the DB.
boring-cyborg
bot
added
area:CLI
area:Scheduler
|
@pingzh - maybe you can have better idea how to fix it ? |
@potiuk, since having a DAGS_FOLDER is required full stop, why don't we always store the relative fileloc? Then airflow can always reassemble the paths relative to DAGS_FOLDER and this is a non-issue.... @ashb any concerns with this idea? It would seem possible that we could fix this as part of an upgrade in a patch release... (i.e. update all filelocs to be relative). And yeah... with knowledge that the paths might be different on different services..... storing the full path in fileloc doesn't make a lot of sense. |
Sounds good. Actually I thought we've already fixed those some time ago to be relative :) |
i will work on it. |
|
i'm closing this one in favor of #26509 |
2 participants
Since #21877 we no longer parse the dag as part of --local task run.
As a consequence it reads dag file location from the database. But if the dag file is in a different path on the k8s executor worker, then this will be the wrong path. We can fix this by respecting the subdir from the top level command.
One problem is, at least with k8s executor (and maybe others too?), the way it builds the pod args, it sets subdir=DAGS_FOLDER which means that all dags in the folder get parsed, which, we should avoid somehow. So, have to work on this more and see whether we can improve that.
It may be easier to just store the relative loc in the DB.