feat: Add Support for DynamodbOnlineStoreConfig endpoint_url parameter #2485

TremaMiguel · 2022-04-04T17:53:16Z

What this PR does / why we need it:

Which issue(s) this PR fixes:

Context

DynamoDBOnlineStoreConfig accepts an endpoint_url parameter , i.e. http://localhost:8000 to enable local DynamoDB development.

Signed-off-by: Miguel Trejo <[email protected]>

achals

Thanks for the contribution @TremaMiguel ! Quick question about local dynamo setup

sdk/python/tests/integration/feature_repos/repo_configuration.py

Signed-off-by: Miguel Trejo <[email protected]>

codecov-commenter · 2022-04-04T18:25:11Z

Codecov Report

Merging #2485 (77c9f13) into master (0c9e5b7) will decrease coverage by 25.37%.
The diff coverage is 93.45%.

@@             Coverage Diff             @@
##           master    #2485       +/-   ##
===========================================
- Coverage   84.57%   59.19%   -25.38%     
===========================================
  Files         130      133        +3     
  Lines       11043    11186      +143     
===========================================
- Hits         9340     6622     -2718     
- Misses       1703     4564     +2861

Flag	Coverage Δ
integrationtests	`?`
unittests	`59.19% <93.45%> (+0.22%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/online_stores/dynamodb.py	`69.36% <79.41%> (-18.77%)`	⬇️
...t/infra/online_store/test_dynamodb_online_store.py	`100.00% <100.00%> (ø)`
.../integration/online_store/test_online_retrieval.py	`17.39% <0.00%> (-82.61%)`	⬇️
sdk/python/tests/utils/online_read_write_test.py	`20.68% <0.00%> (-79.32%)`	⬇️
.../integration/online_store/test_universal_online.py	`14.85% <0.00%> (-77.72%)`	⬇️
...fline_store/test_universal_historical_retrieval.py	`23.73% <0.00%> (-76.27%)`	⬇️
sdk/python/feast/infra/online_stores/redis.py	`27.04% <0.00%> (-69.19%)`	⬇️
...gration/registration/test_feature_service_apply.py	`31.25% <0.00%> (-68.75%)`	⬇️
sdk/python/tests/integration/e2e/test_usage_e2e.py	`33.87% <0.00%> (-66.13%)`	⬇️
sdk/python/tests/data/data_creator.py	`34.78% <0.00%> (-65.22%)`	⬇️
... and 74 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c9e5b7...77c9f13. Read the comment docs.

Signed-off-by: Miguel Trejo <[email protected]>

achals · 2022-04-06T17:19:18Z

sdk/python/feast/infra/online_stores/dynamodb.py

+            region=self.region, endpoint_url=None
+        )
+        dynamodb_resource = _initialize_dynamodb_resource(
+            region=self.region, endpoint_url=None


Why is the endpoint_url None here?

@achals my mistake, should we add it as an attribute to DynamoDBTable ?

I'm not familiar with how the user interacts with the DynamoDBTable class, and how can the user configure it.

Lets add it for consistency, it's not currently used right now but will be used as part of upcoming features.

achals · 2022-04-06T17:19:25Z

sdk/python/feast/infra/online_stores/dynamodb.py

@@ -384,5 +403,7 @@ def update(self):
        dynamodb_client.get_waiter("table_exists").wait(TableName=f"{self.name}")

    def teardown(self):
-        dynamodb_resource = _initialize_dynamodb_resource(region=self.region)
+        dynamodb_resource = _initialize_dynamodb_resource(
+            region=self.region, endpoint_url=None


same question

I've made some changes based on your suggestions, if you don't mind giving it a second check, please.

Signed-off-by: Miguel Trejo <[email protected]>

achals

/lgtm

feast-ci-bot · 2022-04-08T04:23:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, TremaMiguel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [achals]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

achals · 2022-04-08T04:56:26Z

Thanks for the contribution @TremaMiguel !

# [0.20.0](v0.19.0...v0.20.0) (2022-04-14) ### Bug Fixes * Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a)) * Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e)) * Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63)) * Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f)) * Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0)) * Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73)) * Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5)) * Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938)) * Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d)) * Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441)) * Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13)) * Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b)) * Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e)) * Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7)) * Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a)) * Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c)) * Fix Java Master ([#2499](#2499)) ([e083458](e083458)) * Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2)) * Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf)) * Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a)) * Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd)) * Fix the feature repo fixture used by java tests ([#2469](#2469)) ([32e925e](32e925e)) * Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9)) * Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59)) * Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d)) * Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de)) * Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2)) * Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27)) * Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435)) * Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92)) * Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835)) * Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22)) * Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16)) * Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7)) * Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2)) * Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2)) * Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e)) * Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1)) * Timestamp update ([#2486](#2486)) ([bf23111](bf23111)) * Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d)) * Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7)) * Update file api ([#2470](#2470)) ([83a11c6](83a11c6)) * Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155)) * Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9)) * Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3)) * Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6)) * Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49)) * Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d)) ### Features * Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048)) * Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928)) * Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8)) * Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252)) * Add Field class ([#2500](#2500)) ([1279612](1279612)) * Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49)) * Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1)) * Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088)) * Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375)) * Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765)) * Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21)) * Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386) * Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7)) * Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) * Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b)) * Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274)) * Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed)) * Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))

TremaMiguel added 4 commits April 4, 2022 11:50

feat: support dynamodb client and resource with endpoint_url

af305e5

Signed-off-by: Miguel Trejo <[email protected]>

fix: overwrite by partitionkey batchwrite operation

2180e2c

Signed-off-by: Miguel Trejo <[email protected]>

docs: how to configure local dynamodb

02f8071

Signed-off-by: Miguel Trejo <[email protected]>

fix: DynamoDBonlineStore endpoint_url defaults to None

76efcea

Signed-off-by: Miguel Trejo <[email protected]>

TremaMiguel requested a review from a team as a code owner April 4, 2022 17:53

TremaMiguel requested review from achals and removed request for a team April 4, 2022 17:53

github-actions bot assigned achals Apr 4, 2022

feast-ci-bot added the size/M label Apr 4, 2022

achals reviewed Apr 4, 2022

View reviewed changes

sdk/python/tests/integration/feature_repos/repo_configuration.py Outdated Show resolved Hide resolved

docs: setup dummy aws credentials

0440598

Signed-off-by: Miguel Trejo <[email protected]>

achals assigned TremaMiguel and unassigned achals Apr 5, 2022

test: DynamoDBOnlineStoreConfig endpoint_url configuration

7d23380

Signed-off-by: Miguel Trejo <[email protected]>

feast-ci-bot added size/L and removed size/M labels Apr 6, 2022

Merge branch 'master' into feat/dynamodb-endpoint-url

4b9fcc6

TremaMiguel changed the title ~~feat: Add Support for Dynamodb local development~~ feat: Add Support for DynamodbOnlineStoreConfig endpoint_url parameter Apr 6, 2022

achals reviewed Apr 6, 2022

View reviewed changes

feat: DynamoDBTable support endpoint_url

77c9f13

Signed-off-by: Miguel Trejo <[email protected]>

achals approved these changes Apr 8, 2022

View reviewed changes

feast-ci-bot assigned achals Apr 8, 2022

feast-ci-bot added the lgtm label Apr 8, 2022

feast-ci-bot added the approved label Apr 8, 2022

feast-ci-bot merged commit 7b863d1 into feast-dev:master Apr 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Support for DynamodbOnlineStoreConfig endpoint_url parameter #2485

feat: Add Support for DynamodbOnlineStoreConfig endpoint_url parameter #2485

TremaMiguel commented Apr 4, 2022 •

edited

Loading

achals left a comment

codecov-commenter commented Apr 4, 2022 •

edited

Loading

achals Apr 6, 2022

TremaMiguel Apr 6, 2022

TremaMiguel Apr 6, 2022

achals Apr 7, 2022

TremaMiguel Apr 8, 2022

achals Apr 6, 2022

TremaMiguel Apr 8, 2022

achals left a comment

feast-ci-bot commented Apr 8, 2022

achals commented Apr 8, 2022

feat: Add Support for DynamodbOnlineStoreConfig endpoint_url parameter #2485

feat: Add Support for DynamodbOnlineStoreConfig endpoint_url parameter #2485

Conversation

TremaMiguel commented Apr 4, 2022 • edited Loading

achals left a comment

Choose a reason for hiding this comment

codecov-commenter commented Apr 4, 2022 • edited Loading

Codecov Report

achals Apr 6, 2022

Choose a reason for hiding this comment

TremaMiguel Apr 6, 2022

Choose a reason for hiding this comment

TremaMiguel Apr 6, 2022

Choose a reason for hiding this comment

achals Apr 7, 2022

Choose a reason for hiding this comment

TremaMiguel Apr 8, 2022

Choose a reason for hiding this comment

achals Apr 6, 2022

Choose a reason for hiding this comment

TremaMiguel Apr 8, 2022

Choose a reason for hiding this comment

achals left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Apr 8, 2022

achals commented Apr 8, 2022

TremaMiguel commented Apr 4, 2022 •

edited

Loading

codecov-commenter commented Apr 4, 2022 •

edited

Loading