Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Go Sql Online Store #2446

Merged
merged 43 commits into from
Mar 29, 2022
Merged

Conversation

kevjumba
Copy link
Collaborator

@kevjumba kevjumba commented Mar 24, 2022

What this PR does / why we need it:
Implements a basic sql online store that we can use to perform integration testing within Go instead of requiring entry point to be through Python. (This makes it quite hard to do Go specific testing). Also contains integration tests for the sql online store and the getonlinefeatures grpc service to make sure the correct data is retrieved.

Which issue(s) this PR fixes:

Fixes #

@kevjumba kevjumba requested a review from a team as a code owner March 24, 2022 19:36
@kevjumba kevjumba requested review from MattDelac and removed request for a team March 24, 2022 19:36
@kevjumba kevjumba marked this pull request as draft March 24, 2022 19:37
@woop
Copy link
Member

woop commented Mar 24, 2022

I dont agree with the idea of rewriting online stores into Go. We will end up duplicating a lot of logic that should only live in Python in my opinion. What is the motivation behind this change?

Comment on lines +134 to +132
if s.db == nil {
if s.path == "" {
return nil, errors.New("no database path available")
}
db, err := initializeConnection(s.path)
s.db = db
if err != nil {
return nil, err
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be guarded by a lock to prevent concurrent access creating multiple dangling connections

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a mutex.

go/internal/feast/sqliteonlinestore.go Outdated Show resolved Hide resolved
go/internal/feast/sqliteonlinestore.go Outdated Show resolved Hide resolved
@kevjumba
Copy link
Collaborator Author

I dont agree with the idea of rewriting online stores into Go. We will end up duplicating a lot of logic that should only live in Python in my opinion. What is the motivation behind this change?

Adding @pyalex and @achals. Oleksii raised the concern of difficulties in writing certain tests for the Go feature server specifically and we talked about it and decided to use sqlite as a path as a simple testing backbone to do certain integration/unit testing.

@kevjumba kevjumba force-pushed the go_sql_lite_store branch from e82ef9c to 5b5babd Compare March 24, 2022 21:15
@woop
Copy link
Member

woop commented Mar 24, 2022

I dont agree with the idea of rewriting online stores into Go. We will end up duplicating a lot of logic that should only live in Python in my opinion. What is the motivation behind this change?

Adding @pyalex and @achals. Oleksii raised the concern of difficulties in writing certain tests for the Go feature server specifically and we talked about it and decided to use sqlite as a path as a simple testing backbone to do certain integration/unit testing.

Can you be more specific on the types of testing that isn't possible? Is the idea that we would reimplement all these update/destroy methods from the online store interface and make the Go implementation more than a reader/writer?

@achals
Copy link
Member

achals commented Mar 24, 2022

I dont agree with the idea of rewriting online stores into Go. We will end up duplicating a lot of logic that should only live in Python in my opinion. What is the motivation behind this change?

Adding @pyalex and @achals. Oleksii raised the concern of difficulties in writing certain tests for the Go feature server specifically and we talked about it and decided to use sqlite as a path as a simple testing backbone to do certain integration/unit testing.

Can you be more specific on the types of testing that isn't possible? Is the idea that we would reimplement all these update/destroy methods from the online store interface and make the Go implementation more than a reader/writer?

I believe the idea was to have only the read path implemented for SQLite to make it easy to run a full test in pure go using a prebuilt SQLite file without crossing the boundary from python to go, or spinning up redis and writing to it before testing the go code that read it.

I agree that the update method should not be included in this pr, unless I'm missing something.

go.mod Outdated Show resolved Hide resolved
@codecov-commenter
Copy link

codecov-commenter commented Mar 25, 2022

Codecov Report

Merging #2446 (941aea8) into master (ad5694e) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2446      +/-   ##
==========================================
- Coverage   84.94%   84.92%   -0.02%     
==========================================
  Files         127      127              
  Lines       10841    10841              
==========================================
- Hits         9209     9207       -2     
- Misses       1632     1634       +2     
Flag Coverage Δ
integrationtests 75.23% <ø> (-0.24%) ⬇️
unittests 58.45% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ython/feast/embedded_go/online_features_service.py 95.08% <0.00%> (-1.64%) ⬇️
.../integration/online_store/test_universal_online.py 93.59% <0.00%> (-0.28%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad5694e...941aea8. Read the comment docs.

@kevjumba kevjumba force-pushed the go_sql_lite_store branch from ba29126 to 0d1d152 Compare March 25, 2022 22:03
kevjumba added 19 commits March 29, 2022 11:58
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: Kevin Zhang <[email protected]>
@kevjumba kevjumba force-pushed the go_sql_lite_store branch from fdf28b2 to 941aea8 Compare March 29, 2022 19:00
@kevjumba kevjumba requested a review from achals March 29, 2022 19:12
Copy link
Member

@achals achals left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit ff7c7fa into feast-dev:master Mar 29, 2022
jbvaningen pushed a commit to jbvaningen/feast that referenced this pull request Mar 30, 2022
* Initial structure for go sqlite online store

Signed-off-by: Kevin Zhang <[email protected]>

* Somewhat intermediate state

Signed-off-by: Kevin Zhang <[email protected]>

* Add sqlite online store for go for testing

Signed-off-by: Kevin Zhang <[email protected]>

* Revert

Signed-off-by: Kevin Zhang <[email protected]>

* Revert

Signed-off-by: Kevin Zhang <[email protected]>

* Clean up

Signed-off-by: Kevin Zhang <[email protected]>

* Address review issues

Signed-off-by: Kevin Zhang <[email protected]>

* Fix/address issues

Signed-off-by: Kevin Zhang <[email protected]>

* lint

Signed-off-by: Kevin Zhang <[email protected]>

* Fix

Signed-off-by: Kevin Zhang <[email protected]>

* Fix

Signed-off-by: Kevin Zhang <[email protected]>

* Make integration test work

Signed-off-by: Kevin Zhang <[email protected]>

* Fix tests

Signed-off-by: Kevin Zhang <[email protected]>

* Fix tests

Signed-off-by: Kevin Zhang <[email protected]>

* debugging

Signed-off-by: Kevin Zhang <[email protected]>

* Debug

Signed-off-by: Kevin Zhang <[email protected]>

* Debug

Signed-off-by: Kevin Zhang <[email protected]>

* Debug

Signed-off-by: Kevin Zhang <[email protected]>

* Debug

Signed-off-by: Kevin Zhang <[email protected]>

* Debug

Signed-off-by: Kevin Zhang <[email protected]>

* Debug

Signed-off-by: Kevin Zhang <[email protected]>

* Remove feature_repo files

Signed-off-by: Kevin Zhang <[email protected]>

* update gitignore

Signed-off-by: Kevin Zhang <[email protected]>

* Clean up code

Signed-off-by: Kevin Zhang <[email protected]>

* Update go mod

Signed-off-by: Kevin Zhang <[email protected]>

* Update makefile

Signed-off-by: Kevin Zhang <[email protected]>

* Fix gitignore issue

Signed-off-by: Kevin Zhang <[email protected]>

* Update makefile

Signed-off-by: Kevin Zhang <[email protected]>

* Update makefile

Signed-off-by: Kevin Zhang <[email protected]>

* Update makefile

Signed-off-by: Kevin Zhang <[email protected]>

* Update makefile

Signed-off-by: Kevin Zhang <[email protected]>

* Update makefile

Signed-off-by: Kevin Zhang <[email protected]>

* Revert worfklow

Signed-off-by: Kevin Zhang <[email protected]>

* Update build path

Signed-off-by: Kevin Zhang <[email protected]>

* remove

Signed-off-by: Kevin Zhang <[email protected]>

* rename

Signed-off-by: Kevin Zhang <[email protected]>

* Address review

Signed-off-by: Kevin Zhang <[email protected]>

* fix tests

Signed-off-by: Kevin Zhang <[email protected]>

* Fix

Signed-off-by: Kevin Zhang <[email protected]>

* see if this fixes test

Signed-off-by: Kevin Zhang <[email protected]>

* Fix

Signed-off-by: Kevin Zhang <[email protected]>

* revert

Signed-off-by: Kevin Zhang <[email protected]>

* Will add in separate pr to update cryptography

Signed-off-by: Kevin Zhang <[email protected]>
Signed-off-by: joostvan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants