Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abstract ibis-based Offline Store #3979

Closed
tokoko opened this issue Mar 1, 2024 · 0 comments · Fixed by #4102
Closed

Abstract ibis-based Offline Store #3979

tokoko opened this issue Mar 1, 2024 · 0 comments · Fixed by #4102
Labels
kind/feature New feature or request

Comments

@tokoko
Copy link
Collaborator

tokoko commented Mar 1, 2024

Is your feature request related to a problem? Please describe.

  • Each feast offline store has to more or less reimplement the same data transformation logic (mainly point-in-time join) in their respective technologies. It's hard to ensure feature parity.
  • It's next to impossible to properly incorporate ODFV logic into offline store operations.

Describe the solution you'd like

  • Implement abstract offline store with ibis that other feature stores can extend and customize. Ibis is a lightweight python library that offers a universal dataframe API with pluggable query engine backends. As of now it can interface with 15+ engines with varying level of API coverage, including BigQuery, Snowflake, Pyspark, PosgreSQL, Trino and others.
  • ibis-based offline stores should be able to turn Substrait-based on demand feature view transformations into ibis expressions and execute them right alongside core offline store logic in one go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant