airflow clear not cleaning xcom

[abridgett]

airflow clear is described as "Clear a set of task instance, as if they never ran", it would be great if this included clearing the xcom data from that instance.

[r39132]

Can you provide some more details here? I'm a little unclear on the problem and the symptom.

[abridgett]

sure. Basically I had two tasks - lets call them A and B. A is upstream of B.
A does some work, does an xcom_push (e.g. number_of_coins_flipped=6)
B performs and xcom_pull of this (e.g. notifies slack of number_of_coins_flipped).

I then cleared A (including downstream). This time when A is run it fails - before the xcom_push, B then runs (it wasn't set to check the state of A) and reports that 6 coins were flipped.

I appreciate that checking the state of A would also be a good idea, however it was unexpected behaviour and clearing the old xcom data seems reasonable (but maybe there are problems with that?)

[r39132]

Sounds very reasonable. In general, it's a good practice to have your tasks explicitly fail, so that the state of the tasks is very clear. Is there a reason why you don't want an exception to bubble up in its glory?

[abridgett]

I'm not quite understanding the exception/bubble comment, yes I'd normally want to know about any and all failures (certainly in this case I'd normally want the DAG to halt and not run B.
That still leaves the issues of "should a cleared task clear the xcom_pushes"?

[r39132]

I think the expected behavior here is that when a task starts, the xcom is cleared. That will take care of more cases than airflow clear.

[r39132]

This is fixed. Thank you @jgao54 👍

[abridgett]

Thanks everyone - just looking at the PR I see there was quite a lot of thought and edge cases under the covers but the final result looks great :-) (Not sure if I should close this issue so I'll let it to someone more official!)