Fix MyPy errors in Apache Providers

[potiuk]

Part of #19891

The .py additions are to handle "default_args" passed in
examples. Currently some of the obligatory parameters are
(correctly) passed as default_args. We have no good
mechanism yet to handle it properly for MyPy (it would
require to add a custom MyPy plugin to handle it)

We have no better way to handle it for now.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

cc: @khalidmammadov

[potiuk]

If we agree this is the right way of treating context we could easily automate and apply it to all operators/sensors that have currnetly:

execute(self, context)

or

poke(self, context)

as a single big change (after we complete all other MyPy changes. WDYT?

[potiuk]

[jarek:~/code/airflow] [airflow-master-3.7] fix-mypy-google-cloud-build+ ± grep -r "execute(self, context)" airflow/providers | wc
    362    1700   32196
[jarek:~/code/airflow] [airflow-master-3.7] fix-mypy-google-cloud-build+ ± grep -r "poke(self, context)" airflow/providers | wc
     21      86    1644
[jarek:~/code/airflow] [airflow-master-3.7] fix-mypy-google-cloud-build+ ± 

[potiuk]

And including airlfow built-ins (there we could directly import airflow's context):

[jarek:~/code/airflow] [airflow-master-3.7] fix-mypy-google-cloud-build+ 3s ± grep -r "execute(self, context)" airflow | wc
    373    1744   32907
[jarek:~/code/airflow] [airflow-master-3.7] fix-mypy-google-cloud-build+ ± grep -r "poke(self, context)" airflow | wc          
     28     114    2040

[potiuk]

I am afraid I have no better idea for the "Context" imports from providers though :(

[potiuk]

Need some review here - anyone has better idea what to do instead of:

if TYPE_CHECKING:
    from airflow.utils.context import Context
else:
    Context = MutableMapping[str, Any]

I would like to automatically add it later to all the provider's executes/pokes to really benefit from the latest TypeDict Context improvements :D, but maybe we can find a better way ? The problem with it is we can't rely on the common code in airflow :(

[khalidmammadov]

Hi Jarek, Since typing is not required for runtime, I have amended it to just below. |if TYPE_CHECKING: from airflow.utils.context import Context| Is that OK? - Did the same in #20409

…

On 21/12/2021 13:13, Jarek Potiuk wrote: Need some review here - anyone has better idea what to do instead of: |if TYPE_CHECKING: from airflow.utils.context import Context else: Context = MutableMapping[str, Any] | I would like to automatically add it later to all the provider's executes/pokes to really benefit from the latest TypeDict Context improvements :D, but maybe we can find a better way ? The problem with it is we can't rely on the common code in airflow :( — Reply to this email directly, view it on GitHub <#20422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYJ3NFYVTDOCSBORMHASA3USB4OLANCNFSM5KNYLA4A>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[khalidmammadov]

Ignore me, we still need that part

…

On 21/12/2021 21:34, Khalid Mammadov wrote: Hi Jarek, Since typing is not required for runtime, I have amended it to just below. |if TYPE_CHECKING: from airflow.utils.context import Context| Is that OK? - Did the same in #20409 On 21/12/2021 13:13, Jarek Potiuk wrote: > > Need some review here - anyone has better idea what to do instead of: > > |if TYPE_CHECKING: from airflow.utils.context import Context else: > Context = MutableMapping[str, Any] | > > I would like to automatically add it later to all the provider's > executes/pokes to really benefit from the latest TypeDict Context > improvements :D, but maybe we can find a better way ? The problem > with it is we can't rely on the common code in airflow :( > > — > Reply to this email directly, view it on GitHub > <#20422 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACYJ3NFYVTDOCSBORMHASA3USB4OLANCNFSM5KNYLA4A>. > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[potiuk]

Ignore me, we still need that part

Nice try though :)

[uranusjr]

I don’t think we need the else part. This should work:

if TYPE_CHECKING:
    from airflow.utils.context import Context

def execute(self, context: "Context") -> None:
    ...

[josh-fell]

I don’t think we need the else part. This should work:

if TYPE_CHECKING:
    from airflow.utils.context import Context

def execute(self, context: "Context") -> None:
    ...

I can confirm this works in #20034 for the Docker decorator.

[potiuk]

Ah yeah! good one @uranusjr . I totally forgot that we can quote context :)

[potiuk]

That's why I was waiting with applying that. Now we can actually even make it truly "one-liner - cc: @khalidmammadov :

if TYPE_CHECKING: from airflow.utils.context import Context

(not sure if black will be happy with it though)

[potiuk]

Black is not happy with one-liner :(

[potiuk]

All should be nicely updaated. @uranusjr - good stuff with the "Context"

[potiuk]

Should be good to go and I'd love t refresh stats after that :)

[uranusjr]

It seems I’m missing context on the pyi additions.

[khalidmammadov]

Looking at airflow/utils. Let me know if you as well

[potiuk]

It seems I’m missing context on the pyi additions.

Context added.

[potiuk]

It seems I’m missing context on the pyi additions.

Context added.

[potiuk]

Can I get the approval :) ? MyPy is calling.

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.