Skip to content

Conversation

@Taragolis
Copy link
Contributor

Fallback to the deprecated parameters broken for a very long time in GCSToLocalFilesystemOperator and CloudDataTransferServiceListOperationsOperator

docker run --platform=linux/amd64 -it --rm apache/airflow:2.0.2 bash

airflow@873858e9d559:/opt/airflow$ python
Python 3.6.15 (default, Sep  8 2021, 02:32:33) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from airflow.providers.google.cloud.transfers.gcs_to_local import GCSToLocalFilesystemOperator
>>> from airflow.providers.google.cloud.operators.cloud_storage_transfer_service import CloudDataTransferServiceListOperationsOperator

>>> GCSToLocalFilesystemOperator(task_id="t1", bucket="foo", object="bar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/transfers/gcs_to_local.py", line 120, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to GCSToLocalFilesystemOperator (task_id: t1). Invalid arguments were:
**kwargs: {'object': 'bar'}

>>> GCSToLocalFilesystemOperator(task_id="t2", bucket="foo", object_name="bar", object="spam")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/transfers/gcs_to_local.py", line 120, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to GCSToLocalFilesystemOperator (task_id: t2). Invalid arguments were:
**kwargs: {'object': 'spam'}

>>> CloudDataTransferServiceListOperationsOperator(task_id="t3", filter="foo")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py", line 532, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to CloudDataTransferServiceListOperationsOperator (task_id: t3). Invalid arguments were:
**kwargs: {'filter': 'foo'}

>>> CloudDataTransferServiceListOperationsOperator(task_id="t4", request_filter="foo", filter="bar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py", line 532, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to CloudDataTransferServiceListOperationsOperator (task_id: t4). Invalid arguments were:
**kwargs: {'filter': 'bar'}

There is no reason to fix it after 10 major version of Google Provider


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis requested review from eladkal and potiuk February 27, 2024 08:22
@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Feb 27, 2024
@potiuk potiuk merged commit 8e42a2e into apache:main Feb 27, 2024
@Taragolis Taragolis deleted the this-is-one-day-for-gcp branch February 27, 2024 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants