Skip to content

Typing in 0.292.0+ version of yandexcloud conflicts with the way types are used in Airflow Yandex Provider #106

@potiuk

Description

@potiuk

The new version of yandex provider added types but the way yandexcloud is used in Airflow in yandex provider is conflicting with it. For now we will temporarily limit yandexcloud as it is not clear how to fix it, but I guess someone from Yandex shoudl take a close look.,

Those MyPy errors are generated when we install the latest version:

airflow/providers/yandex/hooks/yandex.py:141: error: Argument 2 to "SDK" has
incompatible type "**Dict[str, str]"; expected "Optional[Dict[str, str]]" 
[arg-type]
    ...dexcloud.SDK(user_agent=provider_user_agent(), **sdk_config, **self.cr...
                                                        ^~~~~~~~~~
airflow/providers/yandex/hooks/yandex.py:141: note: "Dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
airflow/providers/yandex/hooks/yandex.py:141: note: Consider using "Mapping" instead, which is covariant in the value type
airflow/providers/yandex/hooks/yandex.py:141: error: Argument 2 to "SDK" has
incompatible type "**Dict[str, str]"; expected "Optional[bytes]"  [arg-type]
    ...dexcloud.SDK(user_agent=provider_user_agent(), **sdk_config, **self.cr...
                                                        ^~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:107: error: Missing
positional argument "folder_id" in call to "find_service_account_id" of
"Helpers"  [call-arg]
        service_account_id = service_account_id or hook.sdk.helpers.find_s...
                                                   ^~~~~~~~~~~~~~~~~~~~~~~...
tests/system/providers/yandex/example_yandexcloud.py:[129](https://github.com/apache/airflow/actions/runs/9329024066/job/25681007396?pr=39946#step:6:130): error: Item
"OperationError" of "Union[OperationResult, OperationError]" has no attribute
"response"  [union-attr]
        return operation_result.response.id
               ^~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:129: error: Item "Message"
of "Union[Message, None, Any]" has no attribute "id"  [union-attr]
        return operation_result.response.id
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:129: error: Item "None" of
"Union[Message, None, Any]" has no attribute "id"  [union-attr]
        return operation_result.response.id
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:152: error: Item
"OperationError" of "Union[OperationResult, OperationError]" has no attribute
"response"  [union-attr]
        return MessageToDict(operation_result.response)
                             ^~~~~~~~~~~~~~~~~~~~~~~~~
tests/system/providers/yandex/example_yandexcloud.py:152: error: Argument 1 to
"MessageToDict" has incompatible type "Union[Message, None, Any]"; expected
"Message"  [arg-type]
        return MessageToDict(operation_result.response)
                             ^~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/hooks/dataproc.py:32: error: Incompatible types in
assignment (expression has type "Dataproc", variable has type
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]")  [assignment]
            self.client = self.sdk.wrappers.Dataproc(
                          ^
airflow/providers/yandex/operators/dataproc.py:188: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_cluster"  [attr-defined]
            operation_result = self.hook.client.create_cluster(
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:225: error: List comprehension
has incompatible type List[yandexcloud._wrappers.dataproc.InitializationAction];
expected List[airflow.providers.yandex.operators.dataproc.InitializationAction] 
[misc]
                    self.hook.sdk.wrappers.InitializationAction(
                    ^
airflow/providers/yandex/operators/dataproc.py:227: error: Argument "args" has
incompatible type "Sequence[str]"; expected "List[str]"  [arg-type]
                        args=init_action.args,
                             ^~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:290: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "delete_cluster"  [attr-defined]
            hook.client.delete_cluster(self.cluster_id)
            ^~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:330: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_hive_job"  [attr-defined]
            hook.client.create_hive_job(
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:385: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_mapreduce_job"  [attr-defined]
            hook.client.create_mapreduce_job(
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:452: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_spark_job"  [attr-defined]
            hook.client.create_spark_job(
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
airflow/providers/yandex/operators/dataproc.py:522: error:
"Callable[[Type[Any], Optional[Any], Optional[str], bool], Any]" has no
attribute "create_pyspark_job"  [attr-defined]
            hook.client.create_pyspark_job(
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 17 errors in 4 files (checked 3104 source files)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions