Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add mysql as online store #3190

Merged
merged 17 commits into from
Sep 22, 2022
21 changes: 21 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,27 @@ test-python-universal-postgres-online:
not test_snowflake" \
sdk/python/tests

test-python-universal-mysql-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.mysql_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.mysql \
FEAST_USAGE=False \
IS_TEST=True \
python -m pytest -n 8 --integration \
-k "not test_universal_cli and \
not test_go_feature_server and \
not test_feature_logging and \
not test_reorder_columns and \
not test_logged_features_validation and \
not test_lambda_materialization_consistency and \
not test_offline_write and \
not test_push_features_to_offline_store and \
not gcs_registry and \
not s3_registry and \
not test_universal_types and \
not test_snowflake" \
sdk/python/tests

test-python-universal-cassandra:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \
Expand Down
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@
* [DynamoDB](reference/online-stores/dynamodb.md)
* [PostgreSQL (contrib)](reference/online-stores/postgres.md)
* [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md)
* [MySQL (contrib)](reference/online-stores/mysql.md)
* [Providers](reference/providers/README.md)
* [Local](reference/providers/local.md)
* [Google Cloud Platform](reference/providers/google-cloud-platform.md)
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/online-stores/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,7 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
[cassandra.md](cassandra.md)
{% endcontent-ref %}

{% content-ref url="mysql.md" %}
[mysql.md](mysql.md)
{% endcontent-ref %}

53 changes: 53 additions & 0 deletions docs/reference/online-stores/mysql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# MySQL online store (contrib)

## Description

The MySQL online store provides support for materializing feature values into a MySQL database for serving online features.

* Only the latest feature values are persisted


## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: mysql
host: DB_HOST
port: DB_PORT
database: DB_NAME
user: DB_USERNAME
password: DB_PASSWORD
```
{% endcode %}

The full set of configuration options is available in [MySQLOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.mysql.MySQLOnlineStoreConfig).

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Mys online store.

| | Mys |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
feast.infra.online\_stores.contrib.mysql\_online\_store package
===============================================================

Submodules
----------

feast.infra.online\_stores.contrib.mysql\_online\_store.mysql module
--------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.mysql_online_store.mysql
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: feast.infra.online_stores.contrib.mysql_online_store
:members:
:undoc-members:
:show-inheritance:
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Subpackages

feast.infra.online_stores.contrib.cassandra_online_store
feast.infra.online_stores.contrib.hbase_online_store
feast.infra.online_stores.contrib.mysql_online_store

Submodules
----------
Expand All @@ -29,6 +30,14 @@ feast.infra.online\_stores.contrib.hbase\_repo\_configuration module
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.mysql\_repo\_configuration module
--------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.mysql_repo_configuration
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.postgres module
--------------------------------------------------

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Mysql Online Store
Mysql is not included in current [Feast](https://github.com/feast-dev/feast) roadmap, this project intends to add Mysql support for Online Store.
We create a table <project_name>_<feature_view_name> which gets updated with data on every materialize call


#### Create a feature repository

```shell
feast init feature_repo
cd feature_repo
```

#### Edit `feature_store.yaml`

set `online_store` type to be `mysql`

```yaml
project: feature_repo
registry: data/registry.db
provider: local
online_store:
type: mysql
host: 127.0.0.1 # mysql endpoint, default to 127.0.0.1
port: 3306 # mysql port, default to 3306
user: test # mysql user, default to test
password: test # mysql password, default to test
database: feast # mysql database, default to feast
```

#### Apply the feature definitions in `example.py`

```shell
feast -c feature_repo apply
```
##### Output
```
Registered entity driver_id
Registered feature view driver_hourly_stats_view
Deploying infrastructure for driver_hourly_stats_view
```

### Materialize Latest Data to Online Feature Store (Mysql)
```
$ CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
$ feast -c feature_repo materialize-incremental $CURRENT_TIME
```
#### Output
```
Materializing 1 feature views from 2022-04-16 15:30:39+05:30 to 2022-04-19 15:31:04+05:30 into the mysql online store.

driver_hourly_stats_view from 2022-04-16 15:30:39+05:30 to 2022-04-19 15:31:04+05:30:
100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 120.59it/s]
```

### Fetch the latest features for some entity id
```python
from pprint import pprint
from feast import FeatureStore

store = FeatureStore(repo_path=".")
feature_vector = store.get_online_features(
features=[
"driver_hourly_stats:conv_rate",
"driver_hourly_stats:acc_rate",
"driver_hourly_stats:avg_daily_trips",
],
entity_rows=[
{"driver_id": 1004},
{"driver_id": 1005},
],
).to_dict()
pprint(feature_vector)

```
#### Output
```
{'acc_rate': [0.01390857808291912, 0.4063614010810852],
'avg_daily_trips': [69, 706],
'conv_rate': [0.6624961495399475, 0.7595928311347961],
'driver_id': [1004, 1005]}
```
Loading