Support multiple XCom output in the BaseOperator

[hussein-awala]

closes: #37284

This PR adds a new argument multiple_outputs to BaseOperator to push multiple XCom when it set to True instead of pushing the result to return_value XCom.

[jannisko]

Thanks for jumping on this request! @hussein-awala
I was kind of hoping to contribute myself but I guess I was too slow 😄

[jannisko]

The implementation for the taskflow decorator seems to return the xcoms separately (with different keys) and in its original form with the default key:

I'm not sure if it makes sense to recreate this behavior.

[hussein-awala]

I'm not sure if it makes sense to recreate this behavior.

Yes, we can do that. I wonder if we need to duplicate all the behaviors, for example, inferring multiple_outputs (#20800) when the execute method has a mapping as the returned value type.

[jannisko]

Yes, we can do that. I wonder if we need to duplicate all the behaviors, for example, inferring multiple_outputs (#20800) when the execute method has a mapping as the returned value type.

I didn't actually notice that there was type-hint based inferral until now 😄
I feel like this feature is too niche to warrant introducing a breaking change to all operators though.

[potiuk]

I didn't actually notice that there was type-hint based inferral until now 😄
I feel like this feature is too niche to warrant introducing a breaking change to all operators though.

Yep. Author of the classic operator should make the decision on their own if they want to return multiple_outputs or not.

[JoyceChan-KSO]

Hi, I think this feature been added by airflow 2.0 + version

https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html

is this feature dropped before?

[jannisko]

@JoyceChan-KSO yes this feature was available for the TaskFlow style operators already, but missing for the classic operators inheriting from BaseOperator (see original issue) 🙂