Add Tabular provider #23704

Fokko · 2022-05-13T21:40:30Z

First version of the Tabular hook :)

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

airflow/providers/tabular/hooks/tabular.py

kbendick

Thanks @Fokko! Left some comments with questions about various things.

airflow/providers/tabular/hooks/tabular.py

airflow/utils/db.py

docs/apache-airflow-providers-tabular/connections.rst

turbaszek · 2022-05-16T19:40:28Z

.github/CODEOWNERS

@@ -59,6 +59,7 @@
 /airflow/providers/google/ @turbaszek
 /airflow/providers/snowflake/ @turbaszek @potiuk @mik-laj
 /airflow/providers/cncf/kubernetes @jedcunningham
+/airflow/providers/tabular/ @Fokko


airflow/providers/tabular/example_dags/example_tabular.py

tests/providers/tabular/hooks/test_tabular.py

eladkal

There is an ongoing discussion in mailing list about if we should add more providers
https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

though I would say this is a unique case as a PMC is the codeowner :)

airflow/models/dag.py

Fokko · 2022-05-17T14:05:10Z

Thanks for pointing me to the thread @eladkal

I actually started on a 3rd party provider, but setting it up within Airflow itself makes it so much easier because of Breeze :) @kbendick and I will maintain this one.

airflow/providers/tabular/provider.yaml

kaxil

This will probably be in the same boat as #22659 where we are waiting for what comes out of https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

Co-authored-by: Kyle Bendickson <[email protected]>

kaxil · 2022-05-17T20:21:53Z

This will probably be in the same boat as #22659 where we are waiting for what comes out of https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

Welp, looks like @eladkal had already commented about it :)

airflow/providers/tabular/example_dags/example_tabular.py

Fokko · 2022-06-14T14:41:32Z

@potiuk thanks for sharing the command. I just ran it and cleanup up the excess in the command hash file.

Would be great to get this in at some point, looking forward to the summary

potiuk · 2022-06-14T20:12:12Z

Would be great to get this in at some point, looking forward to the summary

Coming :)

fd-provider-tabular

Fokko · 2022-06-16T16:29:28Z

@potiuk I saw your mail: https://lists.apache.org/thread/6ngq79df7op541gfwntspdtsvzlv1cr6

It makes sense and I see the current issues. That being said, would it be possible to get this in? Both @kbendick and I will make sure that the operator will be maintained. One thing on the operator, the Tabular one will be quite small. I can also see that we release Apache Iceberg operators in the future as well, but that will probably also be just a wrapper around the Python pyiceberg library.

potiuk · 2022-06-16T18:04:41Z

@potiuk I saw your mail: https://lists.apache.org/thread/6ngq79df7op541gfwntspdtsvzlv1cr6

It makes sense and I see the current issues. That being said, would it be possible to get this in? Both @kbendick and I will make sure that the operator will be maintained. One thing on the operator, the Tabular one will be quite small. I can also see that we release Apache Iceberg operators in the future as well, but that will probably also be just a wrapper around the Python pyiceberg library.

I believe so :). I wanted just to get some community feedback, but I think having just a commitment from someone who has an interest (be it direct stakeholder or say a company that implements a lot of integrations with the product) - even verbally and confirmed in an issue should be more than enough. Feel free to comment in the discussion I started too :).

potiuk · 2022-06-27T15:18:52Z

Hey @Fokko, It took a bit long but I think we got to a consensus regarding new providers, and we also propose some kind of mix-governance aproach, where (among the others) the stakeholders for the future providers (which we are going to technically split to separate repositories soon) should take a bit more responsibility for maintenance:

#24680

If that does not scare you away, and you still want to add the provider to Airflow community providers, feel free to rebase the PR. I will also ask you (hopefully it will be merged soon) to rebase it after we merge #24672 - we are going to change the way how we keep depdencies for providers in order to prepare them to separate to different repository.

Let us know what you want to do, either close the PR or rebase it and lead it to completion after #24672 is merged.

…ular

Fokko · 2022-06-29T21:13:27Z

Hey @potiuk, of course I'm still interested in getting this in! I'm working on rebasing the PR, but I'm also seeing some issues with the recent changes. Let me dig into it and get back to you

…ular

airflow/providers/tabular/CHANGELOG.rst

eladkal · 2022-06-30T12:21:09Z

airflow/providers/tabular/example_dags/example_tabular.py

@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one


new example dags should be written according to AIP-47
the example dag should be in system/providers/tabular path
see #24641 for reference

All green now :)

airflow/providers/tabular/provider.yaml

…ular

josh-fell

Just some small things but LGTM.

josh-fell · 2022-07-06T13:40:04Z

docs/apache-airflow-providers-tabular/index.rst

+`Tabular <https://tabular.io/>`__
+
+
+Release: 0.0.1


Suggested change

Release: 0.0.1

Release: 1.0.0

Ah.. was to quick :).. @Fokko - mind for a small follow-up?

Sure thing!

josh-fell · 2022-07-06T13:40:44Z

docs/apache-airflow-providers-tabular/index.rst

+Installation
+------------
+
+You can install this package on top of an existing Airflow 2.1+ installation via


Suggested change

You can install this package on top of an existing Airflow 2.1+ installation via

You can install this package on top of an existing Airflow 2.2+ installation via

The minimum version for providers was bumped to 2.2.0 recently 🙂

Missed that one, I'll do a follow-up! 👍🏻

josh-fell · 2022-07-06T13:43:26Z

tests/providers/tabular/hooks/test_tabular.py

+from airflow.providers.tabular.hooks.tabular import TabularHook
+
+
+class TestTabularHook(unittest.TestCase):


For net-new unit tests we should prefer pytest over unittest.

I actually prefer pytest as well. Let me follow up on this one

josh-fell · 2022-07-06T13:45:24Z

docs/apache-airflow-providers-tabular/connections.rst

@@ -0,0 +1,39 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one


I think we typically want to have a URI example for connection in the doc. Is that right @mik-laj?

potiuk

Looks cool!

kbendick · 2022-07-06T20:08:48Z

Thank you @Fokko!

Fokko requested review from kaxil, potiuk and mik-laj as code owners May 13, 2022 21:40

boring-cyborg bot added area:dev-tools area:providers kind:documentation labels May 13, 2022

kbendick reviewed May 16, 2022

View reviewed changes

airflow/providers/tabular/hooks/tabular.py Outdated Show resolved Hide resolved

kbendick reviewed May 16, 2022

View reviewed changes

airflow/providers/tabular/hooks/tabular.py Outdated Show resolved Hide resolved

airflow/utils/db.py Outdated Show resolved Hide resolved

docs/apache-airflow-providers-tabular/connections.rst Outdated Show resolved Hide resolved

turbaszek reviewed May 16, 2022

View reviewed changes

airflow/providers/tabular/example_dags/example_tabular.py Outdated Show resolved Hide resolved

turbaszek reviewed May 16, 2022

View reviewed changes

airflow/providers/tabular/example_dags/example_tabular.py Outdated Show resolved Hide resolved

turbaszek reviewed May 16, 2022

View reviewed changes

tests/providers/tabular/hooks/test_tabular.py Outdated Show resolved Hide resolved

Fokko force-pushed the fd-provider-tabular branch 2 times, most recently from eb6028a to ddfe525 Compare May 16, 2022 19:58

Fokko requested review from XD-DENG and ashb as code owners May 17, 2022 13:28

eladkal reviewed May 17, 2022

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

Fokko force-pushed the fd-provider-tabular branch from a682089 to 7c03bf4 Compare May 17, 2022 14:01

Fokko force-pushed the fd-provider-tabular branch 2 times, most recently from b75b5a3 to 13c3d7a Compare May 17, 2022 14:56

eladkal reviewed May 17, 2022

View reviewed changes

airflow/providers/tabular/provider.yaml Outdated Show resolved Hide resolved

kaxil reviewed May 17, 2022

View reviewed changes

Fokko force-pushed the fd-provider-tabular branch 3 times, most recently from fa180e7 to 38afb69 Compare May 17, 2022 18:20

Add Tabular provider

29e6511

Co-authored-by: Kyle Bendickson <[email protected]>

Fokko force-pushed the fd-provider-tabular branch from 38afb69 to 29e6511 Compare May 17, 2022 18:52

kbendick reviewed May 19, 2022

View reviewed changes

airflow/providers/tabular/example_dags/example_tabular.py Outdated Show resolved Hide resolved

Cleanup command hash file

55346f7

Fokko added 2 commits June 16, 2022 18:12

Merge branch 'main' of github.com:apache/airflow into

1b9fc18

fd-provider-tabular

Regen the hash file

c3f515c

Merge branch 'main' into fd-provider-tabular

8e783e1

Fokko added 2 commits June 29, 2022 22:19

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…

032d765

…ular

Update providers

2bc51e7

Fokko added 3 commits June 29, 2022 23:13

Move to the new setup

5111a5d

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…

bd196be

…ular

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…

b529a36

…ular

eladkal reviewed Jun 30, 2022

View reviewed changes

Fokko added 7 commits July 5, 2022 16:20

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…

f6e8f07

…ular

Move example dag

287bf8c

Update docs

af93366

Add link

b963d20

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…

5c1fa56

…ular

Fix the doc

7075b91

Merge branch 'main' of github.com:apache/airflow into fd-provider-tab…

28a7fa1

…ular

josh-fell approved these changes Jul 6, 2022

View reviewed changes

potiuk approved these changes Jul 6, 2022

View reviewed changes

potiuk merged commit 10273e3 into apache:main Jul 6, 2022

Fokko deleted the fd-provider-tabular branch July 6, 2022 14:03

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Tabular provider #23704

Add Tabular provider #23704

Fokko commented May 13, 2022 •

edited

Loading

kbendick left a comment

turbaszek May 16, 2022

eladkal left a comment

Fokko commented May 17, 2022

kaxil left a comment

kaxil commented May 17, 2022

Fokko commented Jun 14, 2022

potiuk commented Jun 14, 2022

Fokko commented Jun 16, 2022

potiuk commented Jun 16, 2022

potiuk commented Jun 27, 2022

Fokko commented Jun 29, 2022

eladkal Jun 30, 2022

Fokko Jul 6, 2022

josh-fell left a comment

josh-fell Jul 6, 2022

potiuk Jul 6, 2022

Fokko Jul 6, 2022

josh-fell Jul 6, 2022

Fokko Jul 6, 2022

josh-fell Jul 6, 2022

Fokko Jul 6, 2022

josh-fell Jul 6, 2022

potiuk left a comment

kbendick commented Jul 6, 2022

		@@ -0,0 +1,48 @@
		# Licensed to the Apache Software Foundation (ASF) under one

	You can install this package on top of an existing Airflow 2.1+ installation via
	You can install this package on top of an existing Airflow 2.2+ installation via

		from airflow.providers.tabular.hooks.tabular import TabularHook


		class TestTabularHook(unittest.TestCase):

		@@ -0,0 +1,39 @@
		.. Licensed to the Apache Software Foundation (ASF) under one

Add Tabular provider #23704

Add Tabular provider #23704

Conversation

Fokko commented May 13, 2022 • edited Loading

kbendick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eladkal left a comment

Choose a reason for hiding this comment

Fokko commented May 17, 2022

kaxil left a comment

Choose a reason for hiding this comment

kaxil commented May 17, 2022

Fokko commented Jun 14, 2022

potiuk commented Jun 14, 2022

Fokko commented Jun 16, 2022

potiuk commented Jun 16, 2022

potiuk commented Jun 27, 2022

Fokko commented Jun 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josh-fell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

potiuk left a comment

Choose a reason for hiding this comment

kbendick commented Jul 6, 2022

Fokko commented May 13, 2022 •

edited

Loading