AIP - 51 Make CeleryKubernetesExecutor extends BaseExecutor for a common API.

[pierrejeambrun]

Make the CeleryKubernetesExecutor extends the BaseExecutor. This would allow for a common API and not having to handle this particular executor differently.

A solution could be to rework a bit the BaseExecutor via property to allow more control for subclasses without breaking backward comp.

Or at least have some ancestor in common to be able to share some logic.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pierrejeambrun]

@o-nikolas

@dstandish made me notice that. The consequence is that changes we bring to the BaseExecutor are not propagated to the CeleryKubernetesExecutor. (is_local property for instance, which is pretty counter intuitive and prone to error)

[o-nikolas]

@dstandish made me notice that. The consequence is that changes we bring to the BaseExecutor are not propagated to the CeleryKubernetesExecutor. (is_local property for instance, which is pretty counter intuitive and prone to error)

Hey Pierre! Yeah, there is some overlap here. I don't love the way multiple executors are stuffed together. I have plans in the works for an AIP to properly support multiple executors, stay tuned for that :)

[pierrejeambrun]

@o-nikolas,

Even better, can't wait!

Should I leave this open until your proposal come through, just to keep track of this specificity ? I can remove the AIP-51 if this is considered out of scope, let me know :)

[o-nikolas]

Should I leave this open until your proposal come through, just to keep track of this specificity ? Maybe I can remove the AIP-51 if this is considered out of scope ?

We'll still need some short term solution which this ticket can capture? At least to fix what's been released so far (which I think are only the is_local and supports_ad_hoc_ti_run fields) and then whatever we decide here will need to be done for future PRs too.

Two approaches I see:

1. The fields that have been added/modified so far (e.g. is_local and supports_ad_hoc_ti_run) can be added directly to the executors which don't inherit from the base executor (e.g. CeleryKubernetesExecutor, LocalKubernetesExecutor, etc)
   1. Pros: fewer code changes; encapsulates the fix to code that will eventually be deprecated/heavily modified.
   2. Cons: only works for executors within Airflow core. Any custom 3rd party executors which don't inherit from BaseExecutor will also break once this code is released (what level of support and backwards compatibility do we owe these 3rd party executors?)
2. All the locations which use the new fields added to the baseExecutor (e.g. is_local and supports_ad_hoc_ti_run) should first check if the executor object has that attribute set, and if not, use the default from the BaseExceutor class (this is approximately how supports_ad_hoc_ti_run was originally implemented)
   1. Pros: Will be backwards compatible for all executors both in Airflow core and public/3rd party executors
   2. Cons: much uglier code and more brittle to maintain and eventually deprecate

WDYT @pierrejeambrun? (also cc: @potiuk @eladkal)

[pierrejeambrun]

I think you resumed pretty well the options we have.

I naively went for 1) in a quickfix #28277 for is_local. I missed the LocalKubernetesExecutor (follow up pr #28288). For supports_ad_hoc_ti_run, both LocalKubernetesExecutor and CeleryKubernetesExecutor already implement it.

I don't have a lot of context/background on custom executor and how our users leverage them. I would say that for maintainability, having third party executor extends the base executor would be much easier. (and make sense to me). Otherwise we have to keep in mind that 'Any' type can be passed as an executor to airflow core, and this is pretty ugly to handle. (would require changing type and using 'hasattr' everywhere in the codebase where we have an 'Executor' type).

I would lean for 1 with a follow up task to make all core airflow executor extends the base class, and asking custom executor to also inherit this. (but the breaking change is a fair concern)

[o-nikolas]

Yupp, I think that's the way I'm leaning as well. I'll hold off on reviewing those PRs and give some time for others to weigh in since it will change how we fix the inconsistency depending on the discussion here.

Edit: It looks like those are being reviewed/merged despite the open discussion here, so I suppose I'll review the remaining open PR.

[o-nikolas]

Checking in on this one.

I'll leave it open for another day or so in case someone wants to weigh in . But after doing even more thinking I'm even more convinced we should go with option 1. Especially after reviewing and thinking about #28300 from @potiuk, the only part of executor logic that is explicitly public is the BaseExecutor interface. I don't think it's functionally possible for us to support any 3rd party executor which is not a child of this class (i.e. implementing its interface). So while it's possible that someone out there has written an executor which doesn't implement this interface and this project may break that and require them to move over to this public interface, I think it's a fair ask.