feat: Feast Spark Offline Store #2349

kevjumba · 2022-03-01T01:58:05Z

What this PR does / why we need it:

Add spark offline store plugin from Adyen Spark to spark offline store. https://github.com/Adyen/feast-spark-offline-store
Currently 159/194 integration tests for spark source work.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Initial cut of Spark based Feast offline store

sdk/python/feast/type_map.py

sdk/python/feast/infra/offline_stores/third_party/spark.py

sdk/python/feast/infra/offline_stores/third_party/spark_source.py

sdk/python/feast/type_map.py

sdk/python/tests/integration/registration/test_universal_types.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

sdk/python/feast/infra/offline_stores/third_party/spark.py

codecov-commenter · 2022-03-02T23:31:05Z

Codecov Report

Merging #2349 (1acc088) into master (74f887f) will decrease coverage by 0.14%.
The diff coverage is 46.40%.

@@            Coverage Diff             @@
##           master    #2349      +/-   ##
==========================================
- Coverage   58.41%   58.27%   -0.15%     
==========================================
  Files         117      118       +1     
  Lines        9764     9885     +121     
==========================================
+ Hits         5704     5760      +56     
- Misses       4060     4125      +65

Flag	Coverage Δ
unittests	`58.27% <46.40%> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/cli.py	`38.65% <ø> (ø)`
sdk/python/feast/inference.py	`60.60% <0.00%> (ø)`
sdk/python/feast/repo_config.py	`77.09% <ø> (ø)`
sdk/python/feast/type_map.py	`39.42% <37.50%> (-0.70%)`	⬇️
...stores/contrib/spark_offline_store/spark_source.py	`47.66% <47.66%> (ø)`
sdk/python/feast/__init__.py	`91.30% <100.00%> (+0.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74f887f...1acc088. Read the comment docs.

java/datatypes/src/main/proto/feast/core/DatastoreTable.proto

docs/reference/data-sources/spark.md

docs/reference/offline-stores/README.md

docs/reference/offline-stores/spark.md

sdk/python/feast/infra/offline_stores/third_party/spark_offline_store/spark.py

sdk/python/feast/type_map.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

sdk/python/tests/integration/feature_repos/repo_configuration.py

kevjumba

@adchia addressed changes PTAL!

docs/reference/offline-stores/spark.md

sdk/python/feast/infra/offline_stores/third_party/spark_offline_store/spark.py

sdk/python/tests/integration/feature_repos/repo_configuration.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

sdk/python/feast/type_map.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

Signed-off-by: Kevin Zhang <[email protected]>

adchia

/lgtm

feast-ci-bot · 2022-03-04T20:16:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [adchia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.19.0](v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f)) * Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb)) * Event timestamps response ([#2355](#2355)) ([5481caf](5481caf)) * Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d)) * Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59)) * Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230))

woop · 2022-03-05T06:13:04Z

🎉 This PR is included in version 0.19.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

* State of feast Signed-off-by: Kevin Zhang <[email protected]> * Clean up changes Signed-off-by: Kevin Zhang <[email protected]> * Fix random incorrect changes Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix build errors Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Rename to pass checks Signed-off-by: Kevin Zhang <[email protected]> * Fix issues Signed-off-by: Kevin Zhang <[email protected]> * Fix type checking issues Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Clean up print statements for first review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <[email protected]> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <[email protected]> * Format Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add file store functionality Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * Add example feature repo Signed-off-by: Kevin Zhang <[email protected]> * Update data source creator Signed-off-by: Kevin Zhang <[email protected]> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <[email protected]> * Update the docs Signed-off-by: Kevin Zhang <[email protected]> * Clean up code Signed-off-by: Kevin Zhang <[email protected]> * Clean up more code Signed-off-by: Kevin Zhang <[email protected]> * Uncomment repo configs Signed-off-by: Kevin Zhang <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Update dependencies Signed-off-by: Kevin Zhang <[email protected]> * Fix ci dependencies Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Realign with master Signed-off-by: Kevin Zhang <[email protected]> * Fix accidental changes Signed-off-by: Kevin Zhang <[email protected]> * Make type map change cleaner Signed-off-by: Kevin Zhang <[email protected]> * Address review comments Signed-off-by: Kevin Zhang <[email protected]> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <[email protected]> * Add comments Signed-off-by: Kevin Zhang <[email protected]> * Reformat Signed-off-by: Kevin Zhang <[email protected]> * Fix logger Signed-off-by: Kevin Zhang <[email protected]> * Remove unused imports Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Fix CI dependencies Signed-off-by: Danny Chiao <[email protected]> * Prefix destinations with project name Signed-off-by: Kevin Zhang <[email protected]> * Update comment Signed-off-by: Kevin Zhang <[email protected]> * Fix 3.8 Signed-off-by: Kevin Zhang <[email protected]> * temporary fix Signed-off-by: Kevin Zhang <[email protected]> * rollback Signed-off-by: Kevin Zhang <[email protected]> * update Signed-off-by: Kevin Zhang <[email protected]> * Update ci? Signed-off-by: Kevin Zhang <[email protected]> * Move third party to contrib Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Remove third_party refactor Signed-off-by: Kevin Zhang <[email protected]> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <[email protected]> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]>

# [0.19.0](feast-dev/feast@v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([feast-dev#2361](feast-dev#2361)) ([d202d51](feast-dev@d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([feast-dev#2363](feast-dev#2363)) ([74f887f](feast-dev@74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([feast-dev#2344](feast-dev#2344)) ([fdb5f21](feast-dev@fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([feast-dev#2354](feast-dev#2354)) ([993616f](feast-dev@993616f)) * Rollback Redis-py to Redis-py-cluster ([feast-dev#2347](feast-dev#2347)) ([1ba86fb](feast-dev@1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([feast-dev#2350](feast-dev#2350)) ([c2a6c6c](feast-dev@c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](feast-dev@412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([feast-dev#2356](feast-dev#2356)) ([ec7385c](feast-dev@ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](feast-dev@0af8adb)) * Event timestamps response ([feast-dev#2355](feast-dev#2355)) ([5481caf](feast-dev@5481caf)) * Feast Spark Offline Store ([feast-dev#2349](feast-dev#2349)) ([98b8d8d](feast-dev@98b8d8d)) * Initial merge of Web UI logic ([feast-dev#2352](feast-dev#2352)) ([ce3bc59](feast-dev@ce3bc59)) * Key ttl setting for redis online store ([feast-dev#2341](feast-dev#2341)) ([236a108](feast-dev@236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([feast-dev#2336](feast-dev#2336)) ([43da230](feast-dev@43da230)) Signed-off-by: Achal Shah <[email protected]>

* State of feast Signed-off-by: Kevin Zhang <[email protected]> * Clean up changes Signed-off-by: Kevin Zhang <[email protected]> * Fix random incorrect changes Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix build errors Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Rename to pass checks Signed-off-by: Kevin Zhang <[email protected]> * Fix issues Signed-off-by: Kevin Zhang <[email protected]> * Fix type checking issues Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Clean up print statements for first review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <[email protected]> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <[email protected]> * Format Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add file store functionality Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * Add example feature repo Signed-off-by: Kevin Zhang <[email protected]> * Update data source creator Signed-off-by: Kevin Zhang <[email protected]> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <[email protected]> * Update the docs Signed-off-by: Kevin Zhang <[email protected]> * Clean up code Signed-off-by: Kevin Zhang <[email protected]> * Clean up more code Signed-off-by: Kevin Zhang <[email protected]> * Uncomment repo configs Signed-off-by: Kevin Zhang <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Update dependencies Signed-off-by: Kevin Zhang <[email protected]> * Fix ci dependencies Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Realign with master Signed-off-by: Kevin Zhang <[email protected]> * Fix accidental changes Signed-off-by: Kevin Zhang <[email protected]> * Make type map change cleaner Signed-off-by: Kevin Zhang <[email protected]> * Address review comments Signed-off-by: Kevin Zhang <[email protected]> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <[email protected]> * Add comments Signed-off-by: Kevin Zhang <[email protected]> * Reformat Signed-off-by: Kevin Zhang <[email protected]> * Fix logger Signed-off-by: Kevin Zhang <[email protected]> * Remove unused imports Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Fix CI dependencies Signed-off-by: Danny Chiao <[email protected]> * Prefix destinations with project name Signed-off-by: Kevin Zhang <[email protected]> * Update comment Signed-off-by: Kevin Zhang <[email protected]> * Fix 3.8 Signed-off-by: Kevin Zhang <[email protected]> * temporary fix Signed-off-by: Kevin Zhang <[email protected]> * rollback Signed-off-by: Kevin Zhang <[email protected]> * update Signed-off-by: Kevin Zhang <[email protected]> * Update ci? Signed-off-by: Kevin Zhang <[email protected]> * Move third party to contrib Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Remove third_party refactor Signed-off-by: Kevin Zhang <[email protected]> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <[email protected]> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]>

# [0.19.0](feast-dev/feast@v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([feast-dev#2361](feast-dev#2361)) ([d202d51](feast-dev@d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([feast-dev#2363](feast-dev#2363)) ([74f887f](feast-dev@74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([feast-dev#2344](feast-dev#2344)) ([fdb5f21](feast-dev@fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([feast-dev#2354](feast-dev#2354)) ([993616f](feast-dev@993616f)) * Rollback Redis-py to Redis-py-cluster ([feast-dev#2347](feast-dev#2347)) ([1ba86fb](feast-dev@1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([feast-dev#2350](feast-dev#2350)) ([c2a6c6c](feast-dev@c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](feast-dev@412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([feast-dev#2356](feast-dev#2356)) ([ec7385c](feast-dev@ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](feast-dev@0af8adb)) * Event timestamps response ([feast-dev#2355](feast-dev#2355)) ([5481caf](feast-dev@5481caf)) * Feast Spark Offline Store ([feast-dev#2349](feast-dev#2349)) ([98b8d8d](feast-dev@98b8d8d)) * Initial merge of Web UI logic ([feast-dev#2352](feast-dev#2352)) ([ce3bc59](feast-dev@ce3bc59)) * Key ttl setting for redis online store ([feast-dev#2341](feast-dev#2341)) ([236a108](feast-dev@236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([feast-dev#2336](feast-dev#2336)) ([43da230](feast-dev@43da230)) Signed-off-by: Achal Shah <[email protected]>

* State of feast Signed-off-by: Kevin Zhang <[email protected]> * Clean up changes Signed-off-by: Kevin Zhang <[email protected]> * Fix random incorrect changes Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix build errors Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Rename to pass checks Signed-off-by: Kevin Zhang <[email protected]> * Fix issues Signed-off-by: Kevin Zhang <[email protected]> * Fix type checking issues Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Clean up print statements for first review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <[email protected]> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <[email protected]> * Format Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add file store functionality Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * Add example feature repo Signed-off-by: Kevin Zhang <[email protected]> * Update data source creator Signed-off-by: Kevin Zhang <[email protected]> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <[email protected]> * Update the docs Signed-off-by: Kevin Zhang <[email protected]> * Clean up code Signed-off-by: Kevin Zhang <[email protected]> * Clean up more code Signed-off-by: Kevin Zhang <[email protected]> * Uncomment repo configs Signed-off-by: Kevin Zhang <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Update dependencies Signed-off-by: Kevin Zhang <[email protected]> * Fix ci dependencies Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Realign with master Signed-off-by: Kevin Zhang <[email protected]> * Fix accidental changes Signed-off-by: Kevin Zhang <[email protected]> * Make type map change cleaner Signed-off-by: Kevin Zhang <[email protected]> * Address review comments Signed-off-by: Kevin Zhang <[email protected]> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <[email protected]> * Add comments Signed-off-by: Kevin Zhang <[email protected]> * Reformat Signed-off-by: Kevin Zhang <[email protected]> * Fix logger Signed-off-by: Kevin Zhang <[email protected]> * Remove unused imports Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Fix CI dependencies Signed-off-by: Danny Chiao <[email protected]> * Prefix destinations with project name Signed-off-by: Kevin Zhang <[email protected]> * Update comment Signed-off-by: Kevin Zhang <[email protected]> * Fix 3.8 Signed-off-by: Kevin Zhang <[email protected]> * temporary fix Signed-off-by: Kevin Zhang <[email protected]> * rollback Signed-off-by: Kevin Zhang <[email protected]> * update Signed-off-by: Kevin Zhang <[email protected]> * Update ci? Signed-off-by: Kevin Zhang <[email protected]> * Move third party to contrib Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Remove third_party refactor Signed-off-by: Kevin Zhang <[email protected]> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <[email protected]> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]>

* Added additional value types to UI parser and removed references to registry-bq.json ([feast-dev#2361](feast-dev#2361)) ([d202d51](feast-dev@d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([feast-dev#2363](feast-dev#2363)) ([74f887f](feast-dev@74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([feast-dev#2344](feast-dev#2344)) ([fdb5f21](feast-dev@fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([feast-dev#2354](feast-dev#2354)) ([993616f](feast-dev@993616f)) * Rollback Redis-py to Redis-py-cluster ([feast-dev#2347](feast-dev#2347)) ([1ba86fb](feast-dev@1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([feast-dev#2350](feast-dev#2350)) ([c2a6c6c](feast-dev@c2a6c6c)) * Add owner field to Entity and rename labels to tags ([412d625](feast-dev@412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([feast-dev#2356](feast-dev#2356)) ([ec7385c](feast-dev@ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](feast-dev@0af8adb)) * Event timestamps response ([feast-dev#2355](feast-dev#2355)) ([5481caf](feast-dev@5481caf)) * Feast Spark Offline Store ([feast-dev#2349](feast-dev#2349)) ([98b8d8d](feast-dev@98b8d8d)) * Initial merge of Web UI logic ([feast-dev#2352](feast-dev#2352)) ([ce3bc59](feast-dev@ce3bc59)) * Key ttl setting for redis online store ([feast-dev#2341](feast-dev#2341)) ([236a108](feast-dev@236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([feast-dev#2336](feast-dev#2336)) ([43da230](feast-dev@43da230)) Signed-off-by: Achal Shah <[email protected]>

* Remove sdk/go Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add interfaces and overall skeleton (incl working protobufs) Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Wire up FeatureStore & OnlineStore, implement RedisOnlineStore constructor, add tests Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Introduce RepoConfig struct instead of using raw map[string]interface{} Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Pass feast project name to RedisOnlineStore Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add Redis Online Read Test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Implement gRPC server start logic Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Update OnlineRead interface Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Redis reader with hashing implemented Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added implementation for GetOnlineFeatures in featurestore and a simple test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Iterate on key building Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Implement gRPC server Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix some warnings Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix bugs & tests Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Implement redis key builder and reader Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add more TODO comments Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Rebuilt protos Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Ignore read test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add more TODO comments Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Clean up serialize function Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fixed GetOnlineFeatures to so that each EntityKey is a row in the first param of OnlineRead, added values to SerializeValue Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Remove comment Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * return copy of Features from OnlineStore instead of reusing original copy + add response for OnlineStore Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Delete go/protos directory from git, since it's in .gitignore Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix go/server compilation Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add integration test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added http json endpoint + add validation for feature names and entity keys for each requested feature view in GetOnlineFeatures Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Call go subprocess from Python Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fixed GetOnlineFeatures bug Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * removed grpc-gateway + add go http server on fs.serve() Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * experimenting python connector Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * working version of python connector Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fixed python connector error + add a script to build plugin binary Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added a simple connector config type to RepoConfig in python sdk to support feast alpha enable go_feature_server Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * migrating python's sdk get_online_features and its helper functions Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added redis pipeline + passed go tests, haven't tested feature service Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added integration test w/o odfv + dummy entity Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * accidentally removed entity from request Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * passed integration test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * first go server clean version, keep python functions. Next commit will diverge from python function calls for more optimized calls Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added 'optimized' functions Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added goroutines to OnlineRead Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added goroutines to redis OnlineRead, getFeaturesToUse, getEntityMaps and entity struct Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Ensure the correct tests are run for the go feature server Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Format and lint Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Remove Go feature server implementation of Python feature server Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Remove incorrect comments Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Clean up Go feature server test logic Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add docstrings for GoServer class Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Remove unused GoServerError Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Remove unused ConnectorOnlineStore Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Run Go feature server tests separately from main integration tests Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * lint-go + cleaned go code + handle BrokenPipeError in go_server.py Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added create ./go/protos to compile-protos-go in Makefile Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * removed 1 unneeded comment Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix small bug in go_server raise if error is not brokenpipe in stop function Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add Snowflake back in for Go feature server tests Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * formatted go Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add CI for Go integration tests Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Modify setup.py to build Go protos and clean up Makefile Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Move go build logic to a different file to not trigger on pip install Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix Go protoc command Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * separated create config from json and yaml + changed go server _connect method name + add TODO to user proper logging + fix go_server exception on stop() + change go layout Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added go server lifecycle test + local infra for go server + experiment with new go server implementations Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Only run Go tests on FileSource Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix Makefile commands Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fixed test to use go_feature_server flag in test + added option to 2 go server implementations + fix parsedKind Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fixed goserverlifecyle test implementation Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fixed goserverlifecycle --- still need to review this test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added a command to test goserver with thread, works in reasonable time now but need to review go server thread implementation Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Print output Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Clean up Makefile and fix errors in setup.py Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Pin Golang protobuf dependency to specific version Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Format + lint Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Revert previous change Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add back Python version Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix test errors due to rebase Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add Go version to unit test workflow Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix .gitignore Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Format + lint Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add setup Go action to set correct Go version for unit tests Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Test Gopath in workflow Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add Github action for installing protoc during unit tests Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Print stderr for Go proto generation Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Check for PATH and GOBIN in unit tests Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Lint Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Switch from using protoc binary for Go protos to grpcio-tools Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add psutil==5.9.0 dependency for test_go_server_life_cycle test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Specify Go version during Github actions Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add back protoc setup Github action Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Add GOPATH to GITHUB_PATH so that protoc-gen-go is discovered Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Set PATH so that protoc-gen-go can be found Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Update Python3.7 ci-requirements file Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Update all requirements Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix test Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix protoc-gen-go bug for all workflows Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Clean up existing test and mark new test as integration Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Lint Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Correctly set GOPATH in workflows Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Modify AWS Lambda Dockerfile to copy the necessary Go files Signed-off-by: Felix Wang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * merge infra package into feast package Signed-off-by: Achal Shah <[email protected]> * optimized go server + address some PR comments Signed-off-by: Ly Cao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added newServingServiceServer to comply with the rest of the code style Signed-off-by: Ly Cao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * removed an extra err check Signed-off-by: Ly Cao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * added pre-allocation to online response metadata Signed-off-by: Ly Cao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Fix protoc compilation for go and use a unix domain socket file Signed-off-by: Achal Shah <[email protected]> * Use a flag for compilation Signed-off-by: Achal Shah <[email protected]> * Change flag in Signed-off-by: Achal Shah <[email protected]> * fix: Remove redis service to prevent more conflicts and add redis node to master_only (#2354) * Fix redis on master.yml Signed-off-by: Kevin Zhang <[email protected]> * Remove hack in make_feature_store_yaml Signed-off-by: Kevin Zhang <[email protected]> * Fix error Signed-off-by: Kevin Zhang <[email protected]> * Fix tests Signed-off-by: Kevin Zhang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Added additional value types to UI parser and removed references to registry-bq.json (#2361) Signed-off-by: Yun Nan Liu <[email protected]> Co-authored-by: Yun Nan Liu <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix:revert back to jdk 11 for datatypes package Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix:upgrade feast ui packages to fix critical vulnerabilities Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Add support for semantic release (#2332) * Add support for semantic release Signed-off-by: Willem Pienaar <[email protected]> * Fix typos Signed-off-by: Willem Pienaar <[email protected]> * Require a Personal Access Token Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Allow all snowflake python connector connection methods to be available to Feast (#2356) * Allow all snowflake python connector connection methods to be available to feast Signed-off-by: Miles Adkins <[email protected]> * format/lint Signed-off-by: Miles Adkins <[email protected]> * feat: have feast using snowflake python connector for authentication Signed-off-by: Miles Adkins <[email protected]> * fix random print Signed-off-by: Miles Adkins <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Fix Redshift bug that stops waiting on statements after 5 minutes (#2363) Signed-off-by: Tsotne Tabidze <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Feast Spark Offline Store (#2349) * State of feast Signed-off-by: Kevin Zhang <[email protected]> * Clean up changes Signed-off-by: Kevin Zhang <[email protected]> * Fix random incorrect changes Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix build errors Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Rename to pass checks Signed-off-by: Kevin Zhang <[email protected]> * Fix issues Signed-off-by: Kevin Zhang <[email protected]> * Fix type checking issues Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Clean up print statements for first review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <[email protected]> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <[email protected]> * Format Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add file store functionality Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * Add example feature repo Signed-off-by: Kevin Zhang <[email protected]> * Update data source creator Signed-off-by: Kevin Zhang <[email protected]> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <[email protected]> * Update the docs Signed-off-by: Kevin Zhang <[email protected]> * Clean up code Signed-off-by: Kevin Zhang <[email protected]> * Clean up more code Signed-off-by: Kevin Zhang <[email protected]> * Uncomment repo configs Signed-off-by: Kevin Zhang <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Update dependencies Signed-off-by: Kevin Zhang <[email protected]> * Fix ci dependencies Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Realign with master Signed-off-by: Kevin Zhang <[email protected]> * Fix accidental changes Signed-off-by: Kevin Zhang <[email protected]> * Make type map change cleaner Signed-off-by: Kevin Zhang <[email protected]> * Address review comments Signed-off-by: Kevin Zhang <[email protected]> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <[email protected]> * Add comments Signed-off-by: Kevin Zhang <[email protected]> * Reformat Signed-off-by: Kevin Zhang <[email protected]> * Fix logger Signed-off-by: Kevin Zhang <[email protected]> * Remove unused imports Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Fix CI dependencies Signed-off-by: Danny Chiao <[email protected]> * Prefix destinations with project name Signed-off-by: Kevin Zhang <[email protected]> * Update comment Signed-off-by: Kevin Zhang <[email protected]> * Fix 3.8 Signed-off-by: Kevin Zhang <[email protected]> * temporary fix Signed-off-by: Kevin Zhang <[email protected]> * rollback Signed-off-by: Kevin Zhang <[email protected]> * update Signed-off-by: Kevin Zhang <[email protected]> * Update ci? Signed-off-by: Kevin Zhang <[email protected]> * Move third party to contrib Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Remove third_party refactor Signed-off-by: Kevin Zhang <[email protected]> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <[email protected]> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Update versions of helm charts + java pom files Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Event timestamps response (#2355) * ability to get event timestamps from online response Signed-off-by: Vitaly Sergeyev <[email protected]> * fix event timestamp bugs Signed-off-by: Vitaly Sergeyev <[email protected]> * python formatting Signed-off-by: Vitaly Sergeyev <[email protected]> * optional param to retrieve event_timestamp in online_reponse Signed-off-by: Vitaly Sergeyev <[email protected]> * formatting Signed-off-by: Vitaly Sergeyev <[email protected]> * renaming param Signed-off-by: Vitaly Sergeyev <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Metadata changes & making data sources top level objects to power Feast UI (#2336) * Squash commits for metadata changes Signed-off-by: Danny Chiao <[email protected]> * tests Signed-off-by: Danny Chiao <[email protected]> * Add more tests Signed-off-by: Danny Chiao <[email protected]> * lint Signed-off-by: Danny Chiao <[email protected]> * Add apply test Signed-off-by: Danny Chiao <[email protected]> * Add apply test Signed-off-by: Danny Chiao <[email protected]> * Add apply test Signed-off-by: Danny Chiao <[email protected]> * lint Signed-off-by: Danny Chiao <[email protected]> * fix bigquery source Signed-off-by: Danny Chiao <[email protected]> * fix test Signed-off-by: Danny Chiao <[email protected]> * fix spark source Signed-off-by: Danny Chiao <[email protected]> * fix spark source Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Remove buggy branch identification Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix action and set go version Signed-off-by: Achal Shah <[email protected]> * Complie python protos correctly Signed-off-by: Achal Shah <[email protected]> * Set up go in the pr integration test workflow Signed-off-by: Achal Shah <[email protected]> * Nuke go requirements for python ci (for the moment) Signed-off-by: Achal Shah <[email protected]> * Update redis port Signed-off-by: Achal Shah <[email protected]> * ci: Add version bump script Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Fix broken release preparation Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Add ci bot as committer and fix typo in dry-run config Signed-off-by: feast-ci-bot <[email protected]> Signed-off-by: Achal Shah <[email protected]> * chore(release): release 0.19.0 # [0.19.0](v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f)) * Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb)) * Event timestamps response ([#2355](#2355)) ([5481caf](5481caf)) * Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d)) * Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59)) * Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230)) Signed-off-by: Achal Shah <[email protected]> * chore: Fix changelog and disable release comments Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Fix unhashable Snowflake and Redshift sources Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * chore: Remove needs-kind and release not requirement Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * chore: Remove release note block Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Uncheck prerelease status for patch releases Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Fix default feast apply path without any extras (#2373) * fix: Fix default feast apply path without any extras Signed-off-by: Danny Chiao <[email protected]> * revert removing ge Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * docs: Add short section about github actions (#2315) * Add short section about github actions Signed-off-by: Kevin Zhang <[email protected]> * Move to contributing.md Signed-off-by: Kevin Zhang <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Use the correct dockerhub image tag when building feature servers (#2372) * fix: Use the correct dockerhub image tag when building feature servers Signed-off-by: Achal Shah <[email protected]> * Print attempt Signed-off-by: Achal Shah <[email protected]> * Increase sleep interval Signed-off-by: Achal Shah <[email protected]> * More logging and sleep Signed-off-by: Achal Shah <[email protected]> * change logging to debug Signed-off-by: Achal Shah <[email protected]> * Change the dockerimage name Signed-off-by: Achal Shah <[email protected]> * Change version to have periods Signed-off-by: Achal Shah <[email protected]> * Change version entirely Signed-off-by: Achal Shah <[email protected]> * Undo unintended changes Signed-off-by: Achal Shah <[email protected]> * Remove dead branch Signed-off-by: Achal Shah <[email protected]> * compile go as part of python ci setup Signed-off-by: Achal Shah <[email protected]> * set path with gopath/bin Signed-off-by: Achal Shah <[email protected]> * Set path for commands correctly Signed-off-by: Achal Shah <[email protected]> * Set up go for lint-go Signed-off-by: Achal Shah <[email protected]> * Remove makefile stuff Signed-off-by: Achal Shah <[email protected]> * Fix workflows and Makefile commands Signed-off-by: Achal Shah <[email protected]> * protoc-gen-go-grpc Signed-off-by: Achal Shah <[email protected]> * install go deps first Signed-off-by: Achal Shah <[email protected]> * Compile go protos as well Signed-off-by: Achal Shah <[email protected]> * Remove go mod tidy from the deps target Signed-off-by: Achal Shah <[email protected]> * Better python tooling Signed-off-by: Achal Shah <[email protected]> * Remove container Signed-off-by: Achal Shah <[email protected]> * Upgrade lint go as well Signed-off-by: Achal Shah <[email protected]> * Upgrade lint go as well Signed-off-by: Achal Shah <[email protected]> * More logging for unit-go Signed-off-by: Achal Shah <[email protected]> * Install using python setup.py develop Signed-off-by: Achal Shah <[email protected]> * install-python-ci-dependencies Signed-off-by: Achal Shah <[email protected]> * Skip tests that need a feature repo Signed-off-by: Achal Shah <[email protected]> * Skip a scary forking test Signed-off-by: Achal Shah <[email protected]> * Remove all the crusty port stuff and disable the lifecycle test Signed-off-by: Achal Shah <[email protected]> * Remove more crusty port stuff Signed-off-by: Achal Shah <[email protected]> * Update skip message Signed-off-by: Achal Shah <[email protected]> * Introduce a sleep before reading values Signed-off-by: Achal Shah <[email protected]> * Disable one more test configuration since background thread-based monitoring doesn't currently work Signed-off-by: Achal Shah <[email protected]> * Disable the test entirely Signed-off-by: Achal Shah <[email protected]> * ci: Add support for semantic release (#2332) * Add support for semantic release Signed-off-by: Willem Pienaar <[email protected]> * Fix typos Signed-off-by: Willem Pienaar <[email protected]> * Require a Personal Access Token Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Feast Spark Offline Store (#2349) * State of feast Signed-off-by: Kevin Zhang <[email protected]> * Clean up changes Signed-off-by: Kevin Zhang <[email protected]> * Fix random incorrect changes Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix build errors Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Rename to pass checks Signed-off-by: Kevin Zhang <[email protected]> * Fix issues Signed-off-by: Kevin Zhang <[email protected]> * Fix type checking issues Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Clean up print statements for first review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <[email protected]> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <[email protected]> * Format Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Address review Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Add file store functionality Signed-off-by: Kevin Zhang <[email protected]> * lint Signed-off-by: Kevin Zhang <[email protected]> * Add example feature repo Signed-off-by: Kevin Zhang <[email protected]> * Update data source creator Signed-off-by: Kevin Zhang <[email protected]> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <[email protected]> * Update the docs Signed-off-by: Kevin Zhang <[email protected]> * Clean up code Signed-off-by: Kevin Zhang <[email protected]> * Clean up more code Signed-off-by: Kevin Zhang <[email protected]> * Uncomment repo configs Signed-off-by: Kevin Zhang <[email protected]> * Fix setup.py Signed-off-by: Kevin Zhang <[email protected]> * Update dependencies Signed-off-by: Kevin Zhang <[email protected]> * Fix ci dependencies Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Screwed up rebase Signed-off-by: Kevin Zhang <[email protected]> * Realign with master Signed-off-by: Kevin Zhang <[email protected]> * Fix accidental changes Signed-off-by: Kevin Zhang <[email protected]> * Make type map change cleaner Signed-off-by: Kevin Zhang <[email protected]> * Address review comments Signed-off-by: Kevin Zhang <[email protected]> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <[email protected]> * Add comments Signed-off-by: Kevin Zhang <[email protected]> * Reformat Signed-off-by: Kevin Zhang <[email protected]> * Fix logger Signed-off-by: Kevin Zhang <[email protected]> * Remove unused imports Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Fix CI dependencies Signed-off-by: Danny Chiao <[email protected]> * Prefix destinations with project name Signed-off-by: Kevin Zhang <[email protected]> * Update comment Signed-off-by: Kevin Zhang <[email protected]> * Fix 3.8 Signed-off-by: Kevin Zhang <[email protected]> * temporary fix Signed-off-by: Kevin Zhang <[email protected]> * rollback Signed-off-by: Kevin Zhang <[email protected]> * update Signed-off-by: Kevin Zhang <[email protected]> * Update ci? Signed-off-by: Kevin Zhang <[email protected]> * Move third party to contrib Signed-off-by: Kevin Zhang <[email protected]> * Fix imports Signed-off-by: Kevin Zhang <[email protected]> * Remove third_party refactor Signed-off-by: Kevin Zhang <[email protected]> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <[email protected]> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Update versions of helm charts + java pom files Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Metadata changes & making data sources top level objects to power Feast UI (#2336) * Squash commits for metadata changes Signed-off-by: Danny Chiao <[email protected]> * tests Signed-off-by: Danny Chiao <[email protected]> * Add more tests Signed-off-by: Danny Chiao <[email protected]> * lint Signed-off-by: Danny Chiao <[email protected]> * Add apply test Signed-off-by: Danny Chiao <[email protected]> * Add apply test Signed-off-by: Danny Chiao <[email protected]> * Add apply test Signed-off-by: Danny Chiao <[email protected]> * lint Signed-off-by: Danny Chiao <[email protected]> * fix bigquery source Signed-off-by: Danny Chiao <[email protected]> * fix test Signed-off-by: Danny Chiao <[email protected]> * fix spark source Signed-off-by: Danny Chiao <[email protected]> * fix spark source Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Remove buggy branch identification Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Add version bump script Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Fix broken release preparation Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Add ci bot as committer and fix typo in dry-run config Signed-off-by: feast-ci-bot <[email protected]> Signed-off-by: Achal Shah <[email protected]> * chore(release): release 0.19.0 * Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f)) * Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c)) * Add owner field to Entity and rename labels to tags ([412d625](412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb)) * Event timestamps response ([#2355](#2355)) ([5481caf](5481caf)) * Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d)) * Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59)) * Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230)) Signed-off-by: Achal Shah <[email protected]> * chore: Fix changelog and disable release comments Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * ci: Uncheck prerelease status for patch releases Signed-off-by: Willem Pienaar <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Fix default feast apply path without any extras (#2373) * fix: Fix default feast apply path without any extras Signed-off-by: Danny Chiao <[email protected]> * revert removing ge Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Making a name for data sources not a breaking change (#2379) * fix: Making a name for data sources not a breaking change Signed-off-by: Danny Chiao <[email protected]> * fix test Signed-off-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Update some fields optional in UI parser (#2380) Signed-off-by: Yun Nan Liu <[email protected]> Co-authored-by: Yun Nan Liu <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * fix: Fix spark docs (#2382) * Fix spark docs Signed-off-by: Kevin Zhang <[email protected]> * Fix readme.md Signed-off-by: Kevin Zhang <[email protected]> * Fix md Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * feat: Makefile for contrib for Issue #2364 (#2366) * Add make file Signed-off-by: Kevin Zhang <[email protected]> * Fix makefile Signed-off-by: Kevin Zhang <[email protected]> * Clean up Signed-off-by: Kevin Zhang <[email protected]> * Add documentation Signed-off-by: Kevin Zhang <[email protected]> * Clean up Signed-off-by: Kevin Zhang <[email protected]> * Update documentation Signed-off-by: Kevin Zhang <[email protected]> * update make target Signed-off-by: Kevin Zhang <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Signed-off-by: Achal Shah <[email protected]> * Closes threadpool resources upon datastore online_write_batch completion (#2386) Signed-off-by: Pamela Toman <[email protected]> Signed-off-by: Achal Shah <[email protected]> * CR comments Signed-off-by: Achal Shah <[email protected]> Co-authored-by: Tsotne Tabidze <[email protected]> Co-authored-by: Willem Pienaar <[email protected]> Co-authored-by: Ly Cao <[email protected]> Co-authored-by: Willem Pienaar <[email protected]> Co-authored-by: Ly Cao <[email protected]> Co-authored-by: Felix Wang <[email protected]> Co-authored-by: Achal Shah <[email protected]> Co-authored-by: Kevin Zhang <[email protected]> Co-authored-by: mickey-liu <[email protected]> Co-authored-by: Yun Nan Liu <[email protected]> Co-authored-by: Danny Chiao <[email protected]> Co-authored-by: Willem Pienaar <[email protected]> Co-authored-by: sfc-gh-madkins <[email protected]> Co-authored-by: Vitaly Sergeyev <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: ptoman-pa <[email protected]>

# [0.20.0](v0.19.0...v0.20.0) (2022-04-14) ### Bug Fixes * Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a)) * Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e)) * Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63)) * Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f)) * Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0)) * Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73)) * Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5)) * Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938)) * Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d)) * Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441)) * Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13)) * Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b)) * Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e)) * Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7)) * Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a)) * Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c)) * Fix Java Master ([#2499](#2499)) ([e083458](e083458)) * Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2)) * Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf)) * Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a)) * Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd)) * Fix the feature repo fixture used by java tests ([#2469](#2469)) ([32e925e](32e925e)) * Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9)) * Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59)) * Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d)) * Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de)) * Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2)) * Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27)) * Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435)) * Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92)) * Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835)) * Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22)) * Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16)) * Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7)) * Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2)) * Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2)) * Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e)) * Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1)) * Timestamp update ([#2486](#2486)) ([bf23111](bf23111)) * Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d)) * Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7)) * Update file api ([#2470](#2470)) ([83a11c6](83a11c6)) * Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155)) * Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9)) * Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3)) * Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6)) * Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49)) * Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d)) ### Features * Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048)) * Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928)) * Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8)) * Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252)) * Add Field class ([#2500](#2500)) ([1279612](1279612)) * Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49)) * Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1)) * Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088)) * Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375)) * Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765)) * Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21)) * Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386) * Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7)) * Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) * Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b)) * Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274)) * Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed)) * Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))

rehevkor5 · 2024-05-13T17:13:28Z

sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py

+                    SELECT {field_string},
+                    ROW_NUMBER() OVER({partition_by_join_key_string} ORDER BY {timestamp_desc_string}) AS feast_row_
+                    FROM {from_expression} t1
+                    WHERE {event_timestamp_column} BETWEEN TIMESTAMP('{start_date_str}') AND TIMESTAMP('{end_date_str}')


I'm not sure if it was intentional that both the start and the end are inclusive, but IMO the starting point should be inclusive and the ending point should be exclusive. That way, if you have ranges of tuples like (a, b) and (b, c), they are contiguous and non-overlapping. By using BETWEEN the ranges will instead overlap at b.

kevjumba added kind/feature New feature or request do-not-merge/work-in-progress labels Mar 1, 2022

kevjumba requested a review from a team as a code owner March 1, 2022 01:58

kevjumba requested review from pyalex and removed request for a team March 1, 2022 01:58

feast-ci-bot added release-note-none and removed do-not-merge/work-in-progress labels Mar 1, 2022

github-actions bot assigned adchia Mar 1, 2022

feast-ci-bot added the size/XL label Mar 1, 2022

kevjumba force-pushed the feast_offline_store branch from cd74282 to 4eeef08 Compare March 1, 2022 02:01

kevjumba unassigned adchia Mar 1, 2022

kevjumba requested review from adchia and removed request for pyalex March 1, 2022 02:02

kevjumba changed the title ~~feat: [WIP] Feast Spark Offline Store~~ feat: Feast Spark Offline Store Mar 1, 2022

kevjumba added the ok-to-test label Mar 1, 2022

kevjumba commented Mar 1, 2022

View reviewed changes

sdk/python/feast/type_map.py Outdated Show resolved Hide resolved

sdk/python/feast/type_map.py Show resolved Hide resolved

adchia reviewed Mar 1, 2022

View reviewed changes

feast-ci-bot added size/XXL and removed size/XL labels Mar 2, 2022

kevjumba added the do-not-merge/work-in-progress label Mar 2, 2022

kevjumba force-pushed the feast_offline_store branch from 733d6bb to 0e76808 Compare March 2, 2022 22:19

kevjumba force-pushed the feast_offline_store branch from 3279f1f to 49c7dbf Compare March 3, 2022 00:28

pyalex reviewed Mar 3, 2022

View reviewed changes

java/datatypes/src/main/proto/feast/core/DatastoreTable.proto Outdated Show resolved Hide resolved

adchia reviewed Mar 3, 2022

View reviewed changes

kevjumba commented Mar 3, 2022

View reviewed changes

adchia reviewed Mar 3, 2022

View reviewed changes

sdk/python/feast/type_map.py Outdated Show resolved Hide resolved

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py Outdated Show resolved Hide resolved

kevjumba removed the do-not-merge/work-in-progress label Mar 3, 2022

kevjumba force-pushed the feast_offline_store branch from cbbc57f to 12eb130 Compare March 3, 2022 23:24

feast-ci-bot removed the size/XXL label Mar 3, 2022

kevjumba added 3 commits March 4, 2022 12:05

Fix imports

05aaeb8

Signed-off-by: Kevin Zhang <[email protected]>

Remove third_party refactor

29a60d7

Signed-off-by: Kevin Zhang <[email protected]>

Revert ci requirements and update comment in type map

b43417e

Signed-off-by: Kevin Zhang <[email protected]>

kevjumba force-pushed the feast_offline_store branch from 9c6f6fa to b43417e Compare March 4, 2022 20:05

Revert 3.8-requirements

1acc088

Signed-off-by: Kevin Zhang <[email protected]>

adchia approved these changes Mar 4, 2022

View reviewed changes

feast-ci-bot assigned adchia Mar 4, 2022

feast-ci-bot added the lgtm label Mar 4, 2022

feast-ci-bot added approved release-note and removed release-note-none labels Mar 4, 2022

kevjumba added ok-to-test and removed ok-to-test labels Mar 4, 2022

adchia merged commit 98b8d8d into feast-dev:master Mar 4, 2022

rehevkor5 reviewed May 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Feast Spark Offline Store #2349

feat: Feast Spark Offline Store #2349

kevjumba commented Mar 1, 2022 •

edited by adchia

Loading

codecov-commenter commented Mar 2, 2022 •

edited

Loading

kevjumba left a comment

adchia left a comment

feast-ci-bot commented Mar 4, 2022

woop commented Mar 5, 2022

rehevkor5 May 13, 2024

feat: Feast Spark Offline Store #2349

feat: Feast Spark Offline Store #2349

Conversation

kevjumba commented Mar 1, 2022 • edited by adchia Loading

codecov-commenter commented Mar 2, 2022 • edited Loading

Codecov Report

kevjumba left a comment

Choose a reason for hiding this comment

adchia left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Mar 4, 2022

woop commented Mar 5, 2022

rehevkor5 May 13, 2024

Choose a reason for hiding this comment

kevjumba commented Mar 1, 2022 •

edited by adchia

Loading

codecov-commenter commented Mar 2, 2022 •

edited

Loading