Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove broken deprecated fallback into the Google provider operators #37740

Merged
merged 1 commit into from
Feb 27, 2024

Conversation

Taragolis
Copy link
Contributor

Fallback to the deprecated parameters broken for a very long time in GCSToLocalFilesystemOperator and CloudDataTransferServiceListOperationsOperator

docker run --platform=linux/amd64 -it --rm apache/airflow:2.0.2 bash

airflow@873858e9d559:/opt/airflow$ python
Python 3.6.15 (default, Sep  8 2021, 02:32:33) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from airflow.providers.google.cloud.transfers.gcs_to_local import GCSToLocalFilesystemOperator
>>> from airflow.providers.google.cloud.operators.cloud_storage_transfer_service import CloudDataTransferServiceListOperationsOperator

>>> GCSToLocalFilesystemOperator(task_id="t1", bucket="foo", object="bar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/transfers/gcs_to_local.py", line 120, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to GCSToLocalFilesystemOperator (task_id: t1). Invalid arguments were:
**kwargs: {'object': 'bar'}

>>> GCSToLocalFilesystemOperator(task_id="t2", bucket="foo", object_name="bar", object="spam")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/transfers/gcs_to_local.py", line 120, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to GCSToLocalFilesystemOperator (task_id: t2). Invalid arguments were:
**kwargs: {'object': 'spam'}

>>> CloudDataTransferServiceListOperationsOperator(task_id="t3", filter="foo")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py", line 532, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to CloudDataTransferServiceListOperationsOperator (task_id: t3). Invalid arguments were:
**kwargs: {'filter': 'foo'}

>>> CloudDataTransferServiceListOperationsOperator(task_id="t4", request_filter="foo", filter="bar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 89, in __call__
    obj: BaseOperator = type.__call__(cls, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py", line 532, in __init__
    super().__init__(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 94, in wrapper
    result = func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 414, in __init__
    "arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to CloudDataTransferServiceListOperationsOperator (task_id: t4). Invalid arguments were:
**kwargs: {'filter': 'bar'}

There is no reason to fix it after 10 major version of Google Provider


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis requested review from potiuk and eladkal February 27, 2024 08:22
@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Feb 27, 2024
@potiuk potiuk merged commit 8e42a2e into apache:main Feb 27, 2024
58 checks passed
@Taragolis Taragolis deleted the this-is-one-day-for-gcp branch February 27, 2024 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants