Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update SQLAlchemy docs #416

Merged
merged 3 commits into from
Sep 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions cookbook/integrations/flytekit_plugins/sql/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
# /Applications/Xcode.app/Contents/Developer/usr/bin/make requirements.txt
# /Library/Developer/CommandLineTools/usr/bin/make requirements.txt
#
attrs==21.2.0
# via scantree
certifi==2021.5.30
# via requests
charset-normalizer==2.0.5
charset-normalizer==2.0.6
# via requests
click==7.1.2
# via flytekit
Expand Down Expand Up @@ -36,7 +36,7 @@ flytekit==0.22.2
# via
# -r ../../../common/requirements-common.in
# flytekitplugins-sqlalchemy
flytekitplugins-sqlalchemy==0.22.2
flytekitplugins-sqlalchemy==0.22.3
# via -r requirements.in
greenlet==1.1.1
# via sqlalchemy
Expand Down Expand Up @@ -76,7 +76,7 @@ pathspec==0.9.0
# via scantree
pillow==8.3.2
# via matplotlib
protobuf==3.17.3
protobuf==3.18.0
# via
# flyteidl
# flytekit
Expand Down Expand Up @@ -119,7 +119,6 @@ six==1.16.0
# cycler
# flytekit
# grpcio
# protobuf
# python-dateutil
# responses
# scantree
Expand Down

This file was deleted.

19 changes: 8 additions & 11 deletions cookbook/integrations/flytekit_plugins/sql/sql-alchemy.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,13 @@

SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.

That being said, Flyte provides an easy-to-use interface to utilize SQLAlchemy to connect to various SQL Databases.
Flyte provides an easy-to-use interface to utilize SQLAlchemy to connect to various SQL Databases. In this example, we use a Postgres DB to understand how SQLAlchemy can be used within Flyte.

In this example, we'll use a Postgres DB to understand how you can use SQLAlchemy with Flyte.
This task will run with a pre-built container, and thus users needn't build one.
You can simply implement the task, then register and execute it immediately.

Install the following packages before running this example (works locally only):

* `Postgres <https://www.postgresql.org/download/>`__
* ``pip install flytekitplugins-sqlalchemy``
* ``pip install psycopg2``
.. note::
This example works locally only. For remote, you need to input a valid URI.
"""

# %%
Expand Down Expand Up @@ -52,7 +48,7 @@ def create_db(


# %%
# We define a ``run_query`` function to create a table and insert data entries into the table.
# We define a ``run_query`` function to create a table and insert data entries into it.
def run_query(
sql_query: str,
conn: psycopg2.extensions.connection,
Expand All @@ -73,7 +69,7 @@ def run_query(


# %%
# We define a helper function to call the required functions.
# We define a helper function.
def pg_server() -> str:
conn_info = {"database": "flights", "user": "postgres", "password": "postgres"}

Expand Down Expand Up @@ -131,7 +127,7 @@ def get_length(df: pandas.DataFrame) -> int:
and duration <= {{ .inputs.upper_duration_cap }}
""",
inputs=kwtypes(lower_duration_cap=int, upper_duration_cap=int),
task_config=SQLAlchemyConfig(uri=local_pg_server),
task_config=SQLAlchemyConfig(uri="postgresql://localhost/flights"),
)


Expand All @@ -147,4 +143,5 @@ def my_wf(lower_duration_cap: int, upper_duration_cap: int) -> int:
if __name__ == "__main__":
print(f"Starting pg server at {pg_server()}")
print(f"Running {__file__} main...")
print(my_wf(lower_duration_cap=600, upper_duration_cap=800))
print(f"Starting Postgres DB at {pg_server()}")
print(my_wf(lower_duration_cap=300, upper_duration_cap=800))