chore: Update offline and online store docs (#3048)

* Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> Signed-off-by: Kevin Zhang <[email protected]>
feast-dev · Aug 12, 2022 · 66d2c76 · 66d2c76
1 parent 492a90d
commit 66d2c76
Show file tree

Hide file tree

Showing 3 changed files with 48 additions and 8 deletions.
diff --git a/Makefile b/Makefile
@@ -209,6 +209,7 @@ test-python-universal-postgres-online:
 test-python-universal-cassandra:
 	PYTHONPATH='.' \
 	FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \
+	PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.cassandra \
 	FEAST_USAGE=False \
 	IS_TEST=True \
 	python -m pytest -x --integration \
@@ -217,6 +218,7 @@ test-python-universal-cassandra:
 test-python-universal-cassandra-no-cloud-providers:
 	PYTHONPATH='.' \
 	FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \
+	PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.cassandra \
 	FEAST_USAGE=False \
 	IS_TEST=True \
 	python -m pytest -x --integration \

diff --git a/docs/how-to-guides/customizing-feast/adding-a-new-offline-store.md b/docs/how-to-guides/customizing-feast/adding-a-new-offline-store.md
@@ -406,7 +406,37 @@ Even if you have created the `OfflineStore` class in a separate repo, you can st
     ```
 
     If the integration tests fail, this indicates that there is a mistake in the implementation of this offline store!
-5. Remember to add your datasource to `repo_config.py` similar to how we added `spark`, `trino`, etc, to the dictionary `OFFLINE_STORE_CLASS_FOR_TYPE` and add the necessary configuration to `repo_configuration.py`. Namely, `AVAILABLE_OFFLINE_STORES` should load your repo configuration module.
+
+5. Remember to add your datasource to `repo_config.py` similar to how we added `spark`, `trino`, etc, to the dictionary `OFFLINE_STORE_CLASS_FOR_TYPE`. This will allow Feast to load your class from the `feature_store.yaml`.
+
+6. Finally, add a Makefile target to the Makefile to run your datastore specific tests by setting the `FULL_REPO_CONFIGS_MODULE` and `PYTEST_PLUGINS` environment variable. The `PYTEST_PLUGINS` environment variable allows pytest to load in the `DataSourceCreator` for your datasource. You can remove certain tests that are not relevant or still do not work for your datastore using the `-k` option.
+
+{% code title="Makefile" %}
+```Makefile
+test-python-universal-spark:
+	PYTHONPATH='.' \
+	FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.spark_repo_configuration \
+	PYTEST_PLUGINS=feast.infra.offline_stores.contrib.spark_offline_store.tests \
+ 	FEAST_USAGE=False IS_TEST=True \
+ 	python -m pytest -n 8 --integration \
+ 	 	-k "not test_historical_retrieval_fails_on_validation and \
+			not test_historical_retrieval_with_validation and \
+			not test_historical_features_persisting and \
+			not test_historical_retrieval_fails_on_validation and \
+			not test_universal_cli and \
+			not test_go_feature_server and \
+			not test_feature_logging and \
+			not test_reorder_columns and \
+			not test_logged_features_validation and \
+			not test_lambda_materialization_consistency and \
+			not test_offline_write and \
+			not test_push_features_to_offline_store.py and \
+			not gcs_registry and \
+			not s3_registry and \
+			not test_universal_types" \
+ 	 sdk/python/tests
+```
+{% endcode %}
 
 ### 7. Dependencies
 

diff --git a/docs/how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md b/docs/how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md
@@ -318,7 +318,7 @@ online_store: feast_custom_online_store.mysql.MySQLOnlineStore
 
 ## 4. Testing the OnlineStore class
 
-### Integrating with the integration test suite and unit test suite.
+### 4.1 Integrating with the integration test suite and unit test suite.
 
 Even if you have created the `OnlineStore` class in a separate repo, you can still test your implementation against the Feast test suite, as long as you have Feast as a submodule in your repo.
 
@@ -352,7 +352,7 @@ If you are planning to start the online store up locally(e.g spin up a local Red
 }
 ```
 
-If you are planning instead to use a Dockerized container to run your tests against your online store, you can define a `OnlineStoreCreator` and replace the `None` object above with your `OnlineStoreCreator` class.
+If you are planning instead to use a Dockerized container to run your tests against your online store, you can define a `OnlineStoreCreator` and replace the `None` object above with your `OnlineStoreCreator` class. You should make this class available to pytest through the `PYTEST_PLUGINS` environment variable.
 
 If you create a containerized docker image for testing, developers who are trying to test with your online store will not have to spin up their own instance of the online store for testing. An example of an `OnlineStoreCreator` is shown below:
 
@@ -372,12 +372,20 @@ class RedisOnlineStoreCreator(OnlineStoreCreator):
 ```
 {% endcode %}
 
-3\. You should swap out the `FULL_REPO_CONFIGS` environment variable and run the integration tests against your online store. In the example repo, the file that overwrites `FULL_REPO_CONFIGS` is `feast_custom_online_store/feast_tests.py`, so you would run:
-
-```bash
-export FULL_REPO_CONFIGS_MODULE='feast_custom_online_store.feast_tests'
-make test-python-universal
+3\. Add a Makefile target to the Makefile to run your datastore specific tests by setting the `FULL_REPO_CONFIGS_MODULE` environment variable. Add `PYTEST_PLUGINS` if pytest is having trouble loading your `DataSourceCreator`. You can remove certain tests that are not relevant or still do not work for your datastore using the `-k` option.
+
+{% code title="Makefile" %}
+```Makefile
+test-python-universal-cassandra:
+	PYTHONPATH='.' \
+	FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \
+	PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.cassandra \
+	FEAST_USAGE=False \
+	IS_TEST=True \
+	python -m pytest -x --integration \
+	sdk/python/tests
 ```
+{% endcode %}
 
 * If there are some tests that fail, this indicates that there is a mistake in the implementation of this online store!