fix: Schema update #2509

kevjumba · 2022-04-07T19:47:10Z

What this PR does / why we need it:
Update schema in RequestDataSource.
Which issue(s) this PR fixes:

Fixes #

sdk/python/feast/field.py

achals · 2022-04-07T20:34:28Z

sdk/python/tests/integration/feature_repos/universal/feature_views.py

-from feast.data_source import DataSource, RequestSource
-
-
-def driver_feature_view(


Was this not being used anywhere?

I think my rebase must have accidentally deleted it, i added it back.

sdk/python/feast/on_demand_feature_view.py

achals · 2022-04-07T20:36:25Z

sdk/python/feast/data_source.py

        description (optional): A human-readable description.
        tags (optional): A dictionary of key-value pairs to store arbitrary metadata.
        owner (optional): The owner of the request data source, typically the email of the primary
            maintainer.
    """

    name: str
-    schema: Dict[str, ValueType]
+    schema: Union[Dict[str, ValueType], List[Field]]


Maybe the internals should always be using List[Field], and we convert to that in __init__ and from_proto? Or is this field exposed to users directly, and it would break them if we changed it's type?

Actually looking at the from_proto method we're changing the type of this field anyway

Yup, removing union.

achals · 2022-04-07T20:40:40Z

sdk/python/feast/data_source.py

+                    if field1 != field2:
+                        return False
+                return True
+            elif isinstance(self.schema, Dict) and isinstance(other.schema, Dict):


You should check here that set(self.schema.keys()).symmetric_difference(set(other.schema.keys()) is empty; otherwise this check will incorrectly pass if other.schema has keys that self.schema does not.

Updated, by just setting schema to list instead.

codecov-commenter · 2022-04-13T00:35:34Z

Codecov Report

Merging #2509 (21481f3) into master (6f22f22) will decrease coverage by 24.16%.
The diff coverage is 75.00%.

@@             Coverage Diff             @@
##           master    #2509       +/-   ##
===========================================
- Coverage   81.78%   57.61%   -24.17%     
===========================================
  Files         141      140        -1     
  Lines       11939    11885       -54     
===========================================
- Hits         9764     6848     -2916     
- Misses       2175     5037     +2862

Flag	Coverage Δ
integrationtests	`?`
unittests	`57.61% <75.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...tegration/feature_repos/universal/feature_views.py	`36.36% <0.00%> (-46.97%)`	⬇️
sdk/python/feast/request_feature_view.py	`62.50% <25.00%> (-5.07%)`	⬇️
sdk/python/feast/data_source.py	`65.44% <70.45%> (-1.93%)`	⬇️
sdk/python/feast/on_demand_feature_view.py	`67.71% <72.72%> (-25.81%)`	⬇️
sdk/python/feast/types.py	`89.39% <83.33%> (-0.61%)`	⬇️
sdk/python/feast/field.py	`90.62% <100.00%> (ø)`
...n/tests/integration/registration/test_inference.py	`85.12% <100.00%> (-14.88%)`	⬇️
...on/tests/integration/registration/test_registry.py	`62.88% <100.00%> (-32.97%)`	⬇️
sdk/python/tests/unit/test_data_sources.py	`100.00% <100.00%> (ø)`
.../integration/online_store/test_online_retrieval.py	`16.84% <0.00%> (-83.16%)`	⬇️
... and 72 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f22f22...21481f3. Read the comment docs.

achals · 2022-04-13T15:30:32Z

java/datatypes/pom.xml

                    </protocArtifact>
                    <pluginId>grpc-java</pluginId>
                    <pluginArtifact>
-                        io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}
+                        io.grpc:protoc-gen-grpc-java:1.0.0:exe:osx-x86_64


Why are we hard coding this?

Fixed, I need to patch this normally in order to have the java tests work on my local machine.

achals · 2022-04-13T15:33:39Z

sdk/python/feast/feature_store.py

@@ -679,6 +679,7 @@ def apply(

        for odfv in odfvs_to_update:
            for v in odfv.source_request_sources.values():
+                print(type(v))


achals · 2022-04-13T15:34:20Z

sdk/python/feast/on_demand_feature_view.py

+                schema.update(request_source.schema)
+            else:
+                raise Exception(
+                    "Request source schema is not correct type: "


Nit, use f-strings

achals · 2022-04-13T15:35:46Z

sdk/python/tests/integration/registration/test_registry.py

@@ -247,7 +247,8 @@ def test_modify_feature_views_success(test_registry):
    )

    request_source = RequestSource(
-        name="request_source", schema={"my_input_1": ValueType.INT32}
+        name="request_source",
+        schema=[Field(name="my_input_1", dtype=PrimitiveFeastType.INT32)],


Can you also have a test for the old to make sure it still works?

achals · 2022-04-13T15:36:22Z

sdk/python/tests/integration/registration/test_inference.py

@@ -227,7 +228,8 @@ def test_view_with_missing_feature(features_df: pd.DataFrame) -> pd.DataFrame:
 def test_datasource_inference():
    # Create Feature Views
    date_request = RequestSource(
-        name="date_request", schema={"some_date": ValueType.UNIX_TIMESTAMP}
+        name="date_request",
+        schema=[Field(name="some_date", dtype=PrimitiveFeastType.UNIX_TIMESTAMP)],


Same, keep a test using the dict style to make sure it keeps working.

pyalex · 2022-04-13T19:16:58Z

go/internal/feast/model/ondemandfeatureview.go

@@ -55,8 +55,8 @@ func (fs *OnDemandFeatureView) ProjectWithFeatures(featureNames []string) (*OnDe
 func (fs *OnDemandFeatureView) GetRequestDataSchema() map[string]types.ValueType_Enum {
 	schema := make(map[string]types.ValueType_Enum)
 	for _, requestDataSource := range fs.SourceRequestDataSources {
-		for fieldName, fieldValueType := range requestDataSource.Schema {
-			schema[fieldName] = fieldValueType
+		for _, spec := range requestDataSource.Schema {


featureSpec is a better name here

or feature. but spec is to vague

Signed-off-by: Kevin Zhang <[email protected]>

achals

/lgtm

feast-ci-bot · 2022-04-13T21:07:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [achals,kevjumba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.20.0](v0.19.0...v0.20.0) (2022-04-14) ### Bug Fixes * Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a)) * Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e)) * Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63)) * Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f)) * Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0)) * Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73)) * Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5)) * Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938)) * Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d)) * Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441)) * Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13)) * Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b)) * Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e)) * Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7)) * Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a)) * Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c)) * Fix Java Master ([#2499](#2499)) ([e083458](e083458)) * Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2)) * Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf)) * Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a)) * Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd)) * Fix the feature repo fixture used by java tests ([#2469](#2469)) ([32e925e](32e925e)) * Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9)) * Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59)) * Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d)) * Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de)) * Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2)) * Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27)) * Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435)) * Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92)) * Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835)) * Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22)) * Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16)) * Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7)) * Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2)) * Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2)) * Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e)) * Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1)) * Timestamp update ([#2486](#2486)) ([bf23111](bf23111)) * Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d)) * Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7)) * Update file api ([#2470](#2470)) ([83a11c6](83a11c6)) * Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155)) * Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9)) * Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3)) * Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6)) * Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49)) * Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d)) ### Features * Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048)) * Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928)) * Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8)) * Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252)) * Add Field class ([#2500](#2500)) ([1279612](1279612)) * Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49)) * Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1)) * Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088)) * Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375)) * Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765)) * Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21)) * Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386) * Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7)) * Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) * Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b)) * Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274)) * Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed)) * Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))

kevjumba added do-not-merge/work-in-progress area/core kind/housekeeping labels Apr 7, 2022

kevjumba requested a review from achals April 7, 2022 19:47

kevjumba requested a review from a team as a code owner April 7, 2022 19:47

feast-ci-bot added the approved label Apr 7, 2022

github-actions bot assigned felixwang9817 Apr 7, 2022

feast-ci-bot added the size/L label Apr 7, 2022

achals reviewed Apr 7, 2022

View reviewed changes

kevjumba force-pushed the schema_update branch from 5ebe397 to cceddbc Compare April 12, 2022 23:32

kevjumba changed the title ~~WIP: fix: Schema update~~ fix: Schema update Apr 12, 2022

feast-ci-bot removed the do-not-merge/work-in-progress label Apr 12, 2022

achals reviewed Apr 13, 2022

View reviewed changes

kevjumba force-pushed the schema_update branch 2 times, most recently from f78e47d to b0818d8 Compare April 13, 2022 19:14

pyalex reviewed Apr 13, 2022

View reviewed changes

kevjumba added 13 commits April 13, 2022 13:59

Fix

1802ac4

Signed-off-by: Kevin Zhang <[email protected]>

Fix

5fd414d

Signed-off-by: Kevin Zhang <[email protected]>

fix lint

9f677c7

Signed-off-by: Kevin Zhang <[email protected]>

indent fix

2122172

Signed-off-by: Kevin Zhang <[email protected]>

Fix

53f3439

Signed-off-by: Kevin Zhang <[email protected]>

Fix lint

d404a4a

Signed-off-by: Kevin Zhang <[email protected]>

Fix lint

f7005d7

Signed-off-by: Kevin Zhang <[email protected]>

Fix go

a7a7703

Signed-off-by: Kevin Zhang <[email protected]>

Fix tests

156b171

Signed-off-by: Kevin Zhang <[email protected]>

Fix

512e00f

Signed-off-by: Kevin Zhang <[email protected]>

Fix

8d62f0c

Signed-off-by: Kevin Zhang <[email protected]>

Fix

eb6bf3c

Signed-off-by: Kevin Zhang <[email protected]>

Revert

fba1388

Signed-off-by: Kevin Zhang <[email protected]>

kevjumba added 6 commits April 13, 2022 13:59

fixes

a25a3b5

Signed-off-by: Kevin Zhang <[email protected]>

Fix lint

7d073d5

Signed-off-by: Kevin Zhang <[email protected]>

Fix and clean up

6fcdaff

Signed-off-by: Kevin Zhang <[email protected]>

Lint

f91d9bd

Signed-off-by: Kevin Zhang <[email protected]>

Fix

a71eaf9

Signed-off-by: Kevin Zhang <[email protected]>

Patch fix, will change definitions in separate pr

a8880cf

Signed-off-by: Kevin Zhang <[email protected]>

kevjumba force-pushed the schema_update branch from 21481f3 to a8880cf Compare April 13, 2022 21:00

achals approved these changes Apr 13, 2022

View reviewed changes

feast-ci-bot assigned achals Apr 13, 2022

feast-ci-bot added the lgtm label Apr 13, 2022

feast-ci-bot merged commit cf7bbc2 into feast-dev:master Apr 13, 2022

kevjumba mentioned this pull request Apr 13, 2022

fix: Fix definitions.py with new definition #2541

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Schema update #2509

fix: Schema update #2509

kevjumba commented Apr 7, 2022

achals Apr 7, 2022

kevjumba Apr 12, 2022

achals Apr 7, 2022

achals Apr 7, 2022

kevjumba Apr 12, 2022

achals Apr 7, 2022

kevjumba Apr 12, 2022

codecov-commenter commented Apr 13, 2022 •

edited

Loading

achals Apr 13, 2022

kevjumba Apr 13, 2022

achals Apr 13, 2022

kevjumba Apr 13, 2022

achals Apr 13, 2022

kevjumba Apr 13, 2022

achals Apr 13, 2022

kevjumba Apr 13, 2022

achals Apr 13, 2022

kevjumba Apr 13, 2022

pyalex Apr 13, 2022

pyalex Apr 13, 2022

kevjumba Apr 13, 2022

achals left a comment

feast-ci-bot commented Apr 13, 2022

		from feast.data_source import DataSource, RequestSource


		def driver_feature_view(

fix: Schema update #2509

fix: Schema update #2509

Conversation

kevjumba commented Apr 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Apr 13, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

achals left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Apr 13, 2022

codecov-commenter commented Apr 13, 2022 •

edited

Loading