Add drain option when canceling Dataflow pipelines

[TobKed]

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

[aaltay]

DataflowJobStatus.JOB_STATE_DRAINED instead of DataflowJobType.JOB_TYPE_STREAMING ?

[TobKed]

Thanks! Fixed.

[aaltay]

Looks great, thank you.

I know little about airflow, 2 clarifying questions:

• Would users generally decide on drain vs cancel when they are starting the operation? (Would they change their minds before onkill?)
• Would they want to wait until drain is completed? Drain may or may not fully complete in 5 minutes timeout window.

[TobKed]

* Would users generally decide on drain vs cancel when they are starting the operation? (Would they change their minds before onkill?)

Users define this parameter in the code which defines the DAG. AFAIK when DAG is running it is not possible to change this parameters afterwards. User can edit DAG and run it again with the new parameters.
@turbaszek could you confirm am I right, please?

* Would they want to wait until drain is completed? Drain may or may not fully complete in 5 minutes timeout window.

I created separate PR to handle timeout, I think it is good idea to allow configure it so users could decide what timeout is suitable for them.
#11501

[TobKed]

Thanks @aaltay
I rebased on the latest master and added tests.
cc @mik-laj @turbaszek @potiuk

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.