Google Display & Video 360 operators fail on POST requests

[echom]

Apache Airflow version: airflow-1.10.6

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.18.6

Environment: Google Cloud Composer v1.10.0

• Others:
  httplib2 >= 0.18
  apache-airflow-backport-providers-google >= 2020.6.24

What happened:

airflow.providers.google.marketing_platform.operators.display_video.GoogleDisplayVideo360RunReportOperator as well as airflow.providers.google.marketing_platform.operators.display_video.GoogleDisplayVideo360CreateReportOperator fail to complete when setting the api_version parameter to the current version v1.1 instead of the default (and now deprecated) v1. In both cases the POST request return a redirect HTTP code (303) without a Location header. Full error message log (note that httplib2 debug output is enabled): log.txt

What you expected to happen: Both operators should execute successfully.

What do you think went wrong?: I believe that there different interpretations of the HTTP 303 code between the DV360 API and httplib2. Httplib2 tries to follow the 303 redirect but fails due to the missing location header. I believe the implementation of httplib2 is correct, however, the DV360 operators need to handle the arguably invalid response and interpret it as a successful operation with no result. I was able to work around it by temporarily disabling follow_redirectson the service object inside the GoogleDisplayVideo360Hook while the POST requests where executed.

How to reproduce it:
Run either of the operators, e.g.:

run = GoogleDisplayVideo360RunReportOperator(
      task_id="test_run",
      dag=dag,
      api_version="v1.1",
      report_id=report_id)

Anything else we need to know:

How often does this problem occur? Once? Every time etc?: Every time

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[turbaszek]

Thanks @echom for the issue! I will take a look at it :)

[echom]

Apologies, I posted the wrong log file from an old Orchestra implementation. Just updated to the correct log file, the error is the same.

[turbaszek]

@echom if you know how to fix the issue would you like to do this?

[turbaszek]

Hm, interesting because I run this operator from out example DAG:

    create_report = GoogleDisplayVideo360CreateReportOperator(
        body=REPORT, task_id="create_report", api_version="v1.1"
    )

and it seems to be working

[turbaszek]

Which version of google libraries do you use?

google-api-core==1.21.0
google-api-python-client==1.9.3

[echom]

I am running on Google Cloud Composer 1.10.6. According to their versioning:

google-api-core==1.16.0
google-api-python-client==1.7.8

Let me try to explicitly install your versions.

[turbaszek]

I think we were similar issue but I cannot find it. @mik-laj do you remember something more?
#9194

[echom]

Ok, after a bit of a struggle with version issues I now arrived at the following:

httplib2==0.18
apache-airflow-backport-providers-google==2020.6.24
google-api-core==1.21.0
google-api-python-client==1.9.3
google-auth=1.20.0
grpcio=1.30.0

Still the issue persists.

@turbaszek Can you do me a favor and check if the API responds with a 303 for you? You should be able to get the response logged when httplib2.debuglevel = 1

I have a line in my log that looks like this:

{logging_mixin.py:112} INFO - reply: 'HTTP/1.1 303 See Other\r\n'

[turbaszek]

No 303 on my side. However, this seems to be related:
#8810

[rafalbiegacz]

#8810 is closed.

Should this item be closed as "no repro"?

[eladkal]

Closing as can't reproduce. If issue still relevant please open a new issue.