feat: Postgres supported as Registry, Online store, and Offline store (…

…#2401) * Feast-postgres added to Feast repo Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Add a template for postgres to allow feast init -t postgres Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Split contrib repo configuration to separate file for each Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Moved the postgres type maps into feast.type_map Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Upgrade to latest feast, get postgres tests running, pip lock files Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Fix the ci requirements Signed-off-by: Kevin Zhang <[email protected]> * Fix repo_config for rest of sources Signed-off-by: Kevin Zhang <[email protected]> * Make backwards compatible with python 3.8- Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * address comments Signed-off-by: Danny Chiao <[email protected]> * update lock files Signed-off-by: Danny Chiao <[email protected]> * fix Signed-off-by: Danny Chiao <[email protected]> * lock files Signed-off-by: Danny Chiao <[email protected]> * fix Signed-off-by: Danny Chiao <[email protected]> * cleanup Signed-off-by: Danny Chiao <[email protected]> * missed refactor name Signed-off-by: Danny Chiao <[email protected]> * Remove snowflake references from test.py for postgres Signed-off-by: Gunnar Sv Sigurbjörnsson <[email protected]> * update repo to match new API Signed-off-by: Danny Chiao <[email protected]> * Add documentation Signed-off-by: Danny Chiao <[email protected]> * rename utils Signed-off-by: Danny Chiao <[email protected]> * update roadmap docs Signed-off-by: Danny Chiao <[email protected]> * update documentation Signed-off-by: Danny Chiao <[email protected]> Co-authored-by: Kevin Zhang <[email protected]> Co-authored-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]>
feast-dev · Apr 20, 2022 · ed2f979 · ed2f979
1 parent d66c931
commit ed2f979
Show file tree

Hide file tree

Showing 49 changed files with 2,010 additions and 309 deletions.
diff --git a/Makefile b/Makefile
@@ -74,6 +74,19 @@ test-python-integration-container:
 test-python-universal-contrib:
 	PYTHONPATH='.' FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.contrib_repo_configuration FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests
 
+test-python-universal-postgres:
+	PYTHONPATH='.' \
+		FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.postgres_repo_configuration \
+		FEAST_USAGE=False \
+		IS_TEST=True \
+		python -m pytest --integration --universal \
+			-k "not test_historical_retrieval_fails_on_validation and \
+				not test_historical_retrieval_with_validation and \
+				not test_historical_features_persisting and \
+				not test_historical_retrieval_fails_on_validation and \
+				not test_universal_cli" \
+			sdk/python/tests
+
 test-python-universal-local:
 	FEAST_USAGE=False IS_TEST=True FEAST_IS_LOCAL_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests
 

diff --git a/README.md b/README.md
@@ -146,8 +146,8 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
   * [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
   * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
-  * [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
-  * [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
+  * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
+  * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
   * [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
   * [ ] HTTP source
 * **Offline Stores**
@@ -156,9 +156,9 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
   * [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
   * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
-  * [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
-  * [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
-  * [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
+  * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
+  * [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
+  * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
   * [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
   * [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
 * **Online Stores**
@@ -167,7 +167,7 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
   * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
   * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
-  * [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
+  * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
   * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
   * [ ] Bigtable (in progress)
   * [ ] Cassandra

diff --git a/docs/getting-started/third-party-integrations.md b/docs/getting-started/third-party-integrations.md
@@ -19,8 +19,8 @@ Don't see your offline store or online store of choice here? Check out our guide
 * [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
 * [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
 * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
-* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
-* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
+* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
+* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
 * [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
 * [ ] HTTP source
 
@@ -31,9 +31,9 @@ Don't see your offline store or online store of choice here? Check out our guide
 * [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
 * [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
 * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
-* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
-* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
-* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
+* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
+* [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
+* [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
 * [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
 * [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
 
@@ -44,7 +44,7 @@ Don't see your offline store or online store of choice here? Check out our guide
 * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
 * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
 * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
-* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
+* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
 * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
 * [ ] Bigtable (in progress)
 * [ ] Cassandra

diff --git a/docs/reference/data-sources/postgres.md b/docs/reference/data-sources/postgres.md
@@ -0,0 +1,25 @@
+# PostgreSQL
+
+## Description
+
+**NOTE**: The Postgres plugin is a contrib plugin. This means it may not be fully stable.
+
+
+The PostgreSQL data source allows for the retrieval of historical feature values from a PostgreSQL database for building training datasets as well as materializing features into an online store.
+
+## Examples
+
+Defining a Postgres source
+
+```python
+from feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source import (
+    PostgreSQLSource,
+)
+
+driver_stats_source = PostgreSQLSource(
+    name="feast_driver_hourly_stats",
+    query="SELECT * FROM feast_driver_hourly_stats",
+    timestamp_field="event_timestamp",
+    created_timestamp_column="created",
+)
+```
diff --git a/docs/reference/data-sources/spark.md b/docs/reference/data-sources/spark.md
@@ -1,4 +1,4 @@
-# Spark
+# Spark (contrib)
 
 ## Description
 

diff --git a/docs/reference/offline-stores/postgres.md b/docs/reference/offline-stores/postgres.md
@@ -0,0 +1,34 @@
+# PostgreSQL (contrib)
+
+## Description
+
+The PostgreSQL offline store is an offline store that provides support for reading [PostgreSQL](../data-sources/postgres.md) data sources.
+
+
+**DISCLAIMER**: This PostgreSQL offline store still does not achieve full test coverage.
+
+* Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. Pandas dataframes will be converted to a Spark dataframe and processed as a temporary view.
+* A `SparkRetrievalJob` is returned when calling `get_historical_features()`.
+  * This allows you to call
+     * `to_df` to retrieve the pandas dataframe.
+     * `to_arrow` to retrieve the dataframe as a PyArrow table.
+
+## Example
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: my_project
+registry: data/registry.db
+provider: local
+offline_store:
+  type: postgres
+  host: DB_HOST
+  port: DB_PORT
+  database: DB_NAME
+  db_schema: DB_SCHEMA
+  user: DB_USERNAME
+  password: DB_PASSWORD
+online_store:
+    path: data/online_store.db
+```
+{% endcode %}
diff --git a/docs/reference/offline-stores/spark.md b/docs/reference/offline-stores/spark.md
@@ -1,4 +1,4 @@
-# Spark
+# Spark (contrib)
 
 ## Description
 

diff --git a/docs/reference/online-stores/postgres.md b/docs/reference/online-stores/postgres.md
@@ -0,0 +1,27 @@
+# PostgreSQL (contrib)
+
+## Description
+
+The PostgreSQL online store provides support for materializing feature values into a PostgreSQL database for serving online features.
+
+* Only the latest feature values are persisted
+
+## Example
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: my_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: postgres
+    host: DB_HOST
+    port: DB_PORT
+    database: DB_NAME
+    db_schema: DB_SCHEMA
+    user: DB_USERNAME
+    password: DB_PASSWORD
+```
+{% endcode %}
+
+Configuration options are available [here](https://rtd.feast.dev/en/latest/#feast.repo_config.SqliteOnlineStoreConfig).
diff --git a/docs/roadmap.md b/docs/roadmap.md
@@ -14,8 +14,8 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file)
   * [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
   * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
-  * [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
-  * [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
+  * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres)
+  * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark)
   * [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
   * [ ] HTTP source
 * **Offline Stores**
@@ -24,9 +24,9 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
   * [x] [Synapse (community plugin)](https://github.com/Azure/feast-azure)
   * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
-  * [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
-  * [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
-  * [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
+  * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres)
+  * [x] [Trino (contrib plugin)](https://github.com/Shopify/feast-trino)
+  * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark)
   * [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
   * [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
 * **Online Stores**
@@ -35,7 +35,7 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
   * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
   * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
-  * [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
+  * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
   * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
   * [ ] Bigtable (in progress)
   * [ ] Cassandra

diff --git a/sdk/python/docs/index.rst b/sdk/python/docs/index.rst
@@ -59,6 +59,12 @@ Trino Source
     :members:
     :exclude-members: TrinoOptions
 
+PostgreSQL Source
+------------------
+
+.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source
+    :members:
+    :exclude-members: PostgreSQLOptions
 
 File Source
 ------------------
@@ -192,6 +198,12 @@ Trino Offline Store
 .. automodule:: feast.infra.offline_stores.contrib.trino_offline_store.trino
     :members:
 
+PostgreSQL Offline Store
+------------------------
+
+.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres
+    :members:
+
 
 Online Store
 ==================
@@ -223,4 +235,11 @@ Redis Online Store
 
 .. automodule:: feast.infra.online_stores.redis
     :members:
-    :noindex:
+    :noindex:
+
+PostgreSQL Online Store
+-----------------------
+
+.. automodule:: feast.infra.online_stores.contrib.postgres
+    :members:
+    :noindex:
diff --git a/...ython/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst b/...ython/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst
@@ -0,0 +1,29 @@
+feast.infra.offline\_stores.contrib.postgres\_offline\_store package
+====================================================================
+
+Submodules
+----------
+
+feast.infra.offline\_stores.contrib.postgres\_offline\_store.postgres module
+----------------------------------------------------------------------------
+
+.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+feast.infra.offline\_stores.contrib.postgres\_offline\_store.postgres\_source module
+------------------------------------------------------------------------------------
+
+.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.rst
@@ -7,6 +7,7 @@ Subpackages
 .. toctree::
    :maxdepth: 4
 
+   feast.infra.offline_stores.contrib.postgres_offline_store
    feast.infra.offline_stores.contrib.spark_offline_store
    feast.infra.offline_stores.contrib.trino_offline_store
 
@@ -21,6 +22,14 @@ feast.infra.offline\_stores.contrib.contrib\_repo\_configuration module
    :undoc-members:
    :show-inheritance:
 
+feast.infra.offline\_stores.contrib.postgres\_repo\_configuration module
+------------------------------------------------------------------------
+
+.. automodule:: feast.infra.offline_stores.contrib.postgres_repo_configuration
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 Module contents
 ---------------
 

diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
@@ -0,0 +1,21 @@
+feast.infra.online\_stores.contrib package
+==========================================
+
+Submodules
+----------
+
+feast.infra.online\_stores.contrib.postgres module
+--------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.postgres
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: feast.infra.online_stores.contrib
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.online_stores.rst b/sdk/python/docs/source/feast.infra.online_stores.rst
@@ -1,6 +1,14 @@
 feast.infra.online\_stores package
 ==================================
 
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 4
+
+   feast.infra.online_stores.contrib
+
 Submodules
 ----------
 

diff --git a/sdk/python/docs/source/feast.infra.utils.postgres.rst b/sdk/python/docs/source/feast.infra.utils.postgres.rst
@@ -0,0 +1,29 @@
+feast.infra.utils.postgres package
+==================================
+
+Submodules
+----------
+
+feast.infra.utils.postgres.connection\_utils module
+---------------------------------------------------
+
+.. automodule:: feast.infra.utils.postgres.connection_utils
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+feast.infra.utils.postgres.postgres\_config module
+--------------------------------------------------
+
+.. automodule:: feast.infra.utils.postgres.postgres_config
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: feast.infra.utils.postgres
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.utils.rst b/sdk/python/docs/source/feast.infra.utils.rst
@@ -1,6 +1,14 @@
 feast.infra.utils package
 =========================
 
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 4
+
+   feast.infra.utils.postgres
+
 Submodules
 ----------