Missing example DAGs/system tests for Google services #8280

mik-laj · 2020-04-13T20:45:31Z

Description

Hello,

We have a rule that every GCP operators should have an example DAG and system test. This is true in many cases, but there are minor exceptions.
https://github.com/apache/airflow/blob/master/tests/always/test_project_structure.py#L155-L162

We also lack examples for individual operators.
https://github.com/apache/airflow/blob/master/tests/always/test_project_structure.py#L164-L235

If you decide to finish this ticket you don't have to do all the work yourself. One PR can only deal with a single operator and it's ok.

These example DAGs are key to ensuring high-quality integration.

If used in system tests, they prevent regression and facilitate testing.
If used in the documentation, they allow us to learn about operators in a real example. Users can easily do CTRL + C, CTRL + V, which makes it easier to write new DAGs.

If you haven't used the GCP yet, after creating the account you will get $300, which will allow you to get to know these services better.

The implementation of this task will allow a better understanding of GCP services, as well as learn methods of testing that is required by the community. If anyone is interested in this task, I am willing to provide all the necessary tips and information.

Are you wondering how to start contributing to this project? Start by reading our contributor guide

Related Issues

N/A

The text was updated successfully, but these errors were encountered:

SlowSnowFox · 2020-04-13T21:29:05Z

I would be happy to give it a try. I've only worked with airflow and AWS so far but I'm sure I should be able to come up with at least some examples.

mik-laj · 2020-04-22T01:51:00Z

@SlowSnowFox I'm glad you want to work on it. Have you encountered any difficulties? Can I help you?

SlowSnowFox · 2020-04-23T23:13:21Z

Hey, Sorry for the late reply. I'll be able to make a pull request for some of them on the weekend. Is it ok if I just package all examples I've made in a pull request and list the specific tests in the commit message?

mik-laj · 2020-04-23T23:36:51Z

Hey.
That sounds good. We'll see how we work. In case of problems, I will show you how to quickly divide it into many PR.
Best regards,

joppevos · 2020-04-24T09:56:48Z

Happy to try the gcs_to_bigquery and bigquery_to_gcs examples if there is not being worked on yet.

mik-laj · 2020-04-24T10:06:09Z

Go ahead. Get started. I look forward to your example. I would be happy if you also added a system test, because it will allow us to check the example more easily.

joppevos · 2020-04-24T10:16:49Z

@mik-laj great! correct me if I'm wrong, but it seems there is already a gcs_to_bigquery example over here. https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_gcs_to_bq.py

mik-laj · 2020-04-24T10:23:58Z

Fantastic. It is now enough to change the file name and add a system test.
We want operator and sample file names to match.

airflow/providers/google/cloud/example_dags/example_gcs_to_bq.py
airflow/providers/google/cloud/operators/gcs_to_bigquery.py

This is the main requirement for now.

ephraimbuddy · 2020-05-07T05:28:03Z

There's an example DAG and system tests for text_to_speech.py, speech_to_text.py and translate_speech.py.
The examples are all in https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_speech.py
and the system tests are also all in https://github.com/apache/airflow/blob/master/tests/providers/google/cloud/operators/test_speech_system.py

mik-laj · 2020-05-07T08:17:00Z

@ephraimbuddy Would you like to split this file? Each module should have a separate sample DAG and a separate system test. This is important because of code maintenance. When everything adheres to one principle, it is easier to make changes automatically.

ephraimbuddy · 2020-05-07T08:37:35Z

Ok. I'll be separating them into different files. Thanks for explanation

irvifa · 2020-07-06T18:03:34Z

I saw there's many PR that have been merged. Is this issue already closed then?

mik-laj · 2020-07-06T18:23:58Z

@irvifa Some examples are still missing. I updated the first post.
https://github.com/apache/airflow/blob/master/tests/test_project_structure.py#L125

irvifa · 2020-07-11T04:12:16Z

@mik-laj can you please review my PR: #9760
I was wondering on how I can make sure e2e test works when I can't create an instance or db in the example.

Previously there's already example of how to run export from CloudSQL to GCS described in https://airflow.readthedocs.io/en/stable/_modules/airflow/contrib/example_dags/example_gcp_sql.html. However, based on apache#8280 the test itself is not available yet.

mik-laj · 2020-11-11T18:35:57Z

We have a lot of DAG examples, but sometimes these DAGs don't contain all operators. More information in the first post.

rachael-ds · 2021-01-27T09:33:53Z

Happy to work on the documentation/example dags for the Data Loss Prevention operator :)

mik-laj · 2021-01-27T09:35:31Z

@rachael-ds I assigned you to this ticket 🐈

Bowrna · 2021-11-23T11:23:03Z

@mik-laj I am a newbie here and I am trying to add mssql_to_gcs example dag. GCP is also new to me and I have created an account and enabled service account. I have to add/edit the connection details in GCP connection to check if my example dag is working. I have created an example dag with mssql already. When configuring connections for GCP, i had to add keyfile path, keyfile secret name, keyfile json. But i am not sure how to get these values. Is there any docs that helps me to understand this part? Thanks.

potiuk · 2021-11-23T12:58:17Z

Here are the relevant GCP instructions @Bowrna https://cloud.google.com/docs/authentication/production

Bowrna · 2021-11-23T17:39:42Z

@potiuk I have followed the instructions and enabled the service account. I have the JSON key after enabling the service account. But I am not sure what values I have to give in the keyfile path, keyfile secret name, keyfile json in the Google Cloud connections form. I only have the JSON generated out by enabling service account.

potiuk · 2021-11-24T19:15:08Z

In Breeze you can put the files in "files" dir and it will be visible inside as "/files/*" and then in the connection you should specify path to that file :). I think you can specify either Json orh "Keyfile + Secret" - you do not have to specify all three. I think this page has good explanation of what is in the key. You can also - as exercise look at the unit tests of GcpBaseHook - it should have tests for all the different authentication options and should show you which combinations are valid.

Bowrna · 2021-11-25T05:54:47Z

In Breeze you can put the files in "files" dir and it will be visible inside as "/files/*" and then in the connection you should specify path to that file :). I think you can specify either Json orh "Keyfile + Secret" - you do not have to specify all three. I think this page has good explanation of what is in the key. You can also - as exercise look at the unit tests of GcpBaseHook - it should have tests for all the different authentication options and should show you which combinations are valid.

thanks @potiuk. Checking the unit test would be great way to understand this configuration.

eladkal · 2022-04-21T19:27:22Z

Closed in favor of AIP 47
https://github.com/apache/airflow/projects/15
We have a dedicated issue per each example dag

mik-laj added provider:google Google (including GCP) related issues kind:feature Feature Requests good first issue labels Apr 13, 2020

mik-laj added kind:bug This is a clearly a bug and removed kind:feature Feature Requests labels Apr 13, 2020

joppevos mentioned this issue Apr 25, 2020

Add system test for gcs_to_bigquery #8556

Merged

6 tasks

ephraimbuddy mentioned this issue May 8, 2020

Add separate example DAGs and system tests for google cloud speech #8778

Merged

6 tasks

This was referenced May 28, 2020

Add example dag and system test for LocalFilesystemToGCSOperator #9043

Merged

add example dag and system test for GoogleSheetsToGCSOperator #9056

Merged

add separate example dags and system tests for GCSToGoogleSheetsOperator #9066

Merged

irvifa mentioned this issue Jul 11, 2020

test: Add mysql_to_gcs example for Google Cloud Platform provider #9760

Closed

6 tasks

ephraimbuddy mentioned this issue Sep 6, 2020

Extract missing gcs_to_local example DAG from gcs example #10767

Merged

This was referenced Sep 15, 2020

Add example dag and system test for S3ToGCSOperator #10951

Merged

Add example dag and system test for MySQLToGCSOperator #10990

Merged

potiuk added the Hacktoberfest label Sep 25, 2020

mik-laj mentioned this issue Nov 11, 2020

Detect partial examples DAGs for Google provider #12277

Merged

kaxil removed the Hacktoberfest label Nov 12, 2020

This was referenced Nov 20, 2020

gcs_to_bq: Support schema in different bucket #12520

Closed

[AIRFLOW-12520] feat(gcs_to_bq): support schema in different bucket #12521

Closed

mik-laj mentioned this issue Dec 21, 2020

Add more operators to example DAG for Cloud Tasks #13235

Merged

mik-laj mentioned this issue Jan 7, 2021

Enforce system tests for Google operators #8390

Closed

5 tasks

mik-laj assigned rachael-ds Jan 27, 2021

mik-laj mentioned this issue Jun 27, 2021

Implemented Basic EKS Integration #16571

Merged

mik-laj mentioned this issue Jul 22, 2021

Enforce system tests for Google operators #17165

Closed

potiuk added the hacktoberfest label Oct 6, 2021

Bowrna mentioned this issue Nov 29, 2021

Added example DAG for MSSQL to Google Cloud Storage (GCS) #19873

Merged

eladkal removed the hacktoberfest label Feb 1, 2022

eladkal unassigned rachael-ds Mar 13, 2022

eladkal closed this as completed Apr 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing example DAGs/system tests for Google services #8280

Missing example DAGs/system tests for Google services #8280

mik-laj commented Apr 13, 2020 •

edited

Loading

SlowSnowFox commented Apr 13, 2020

mik-laj commented Apr 22, 2020

SlowSnowFox commented Apr 23, 2020

mik-laj commented Apr 23, 2020

joppevos commented Apr 24, 2020

mik-laj commented Apr 24, 2020

joppevos commented Apr 24, 2020 •

edited

Loading

mik-laj commented Apr 24, 2020

ephraimbuddy commented May 7, 2020

mik-laj commented May 7, 2020 •

edited

Loading

ephraimbuddy commented May 7, 2020

irvifa commented Jul 6, 2020

mik-laj commented Jul 6, 2020 •

edited

Loading

irvifa commented Jul 11, 2020

mik-laj commented Nov 11, 2020

rachael-ds commented Jan 27, 2021

mik-laj commented Jan 27, 2021

Bowrna commented Nov 23, 2021 •

edited

Loading

potiuk commented Nov 23, 2021

Bowrna commented Nov 23, 2021

potiuk commented Nov 24, 2021

Bowrna commented Nov 25, 2021

eladkal commented Apr 21, 2022

Missing example DAGs/system tests for Google services #8280

Missing example DAGs/system tests for Google services #8280

Comments

mik-laj commented Apr 13, 2020 • edited Loading

SlowSnowFox commented Apr 13, 2020

mik-laj commented Apr 22, 2020

SlowSnowFox commented Apr 23, 2020

mik-laj commented Apr 23, 2020

joppevos commented Apr 24, 2020

mik-laj commented Apr 24, 2020

joppevos commented Apr 24, 2020 • edited Loading

mik-laj commented Apr 24, 2020

ephraimbuddy commented May 7, 2020

mik-laj commented May 7, 2020 • edited Loading

ephraimbuddy commented May 7, 2020

irvifa commented Jul 6, 2020

mik-laj commented Jul 6, 2020 • edited Loading

irvifa commented Jul 11, 2020

mik-laj commented Nov 11, 2020

rachael-ds commented Jan 27, 2021

mik-laj commented Jan 27, 2021

Bowrna commented Nov 23, 2021 • edited Loading

potiuk commented Nov 23, 2021

Bowrna commented Nov 23, 2021

potiuk commented Nov 24, 2021

Bowrna commented Nov 25, 2021

eladkal commented Apr 21, 2022

mik-laj commented Apr 13, 2020 •

edited

Loading

joppevos commented Apr 24, 2020 •

edited

Loading

mik-laj commented May 7, 2020 •

edited

Loading

mik-laj commented Jul 6, 2020 •

edited

Loading

Bowrna commented Nov 23, 2021 •

edited

Loading