Feast 0.5 release

[woop]

We're going to try and make a habit of discussing and evolving our roadmaps using GitHub issues. For each release this will allow folks to see what changes are upcoming, propose new changes to include in the release, and also understand why certain decisions have been made.

I have moved the current roadmap for Feast 0.5 to Gitbook. It is available over here https://docs.feast.dev/introduction/roadmap

The roadmap as committed above is already out of date, and this is part of the reason why I am creating this issue. We are planning to increase the scope of the 0.5 release. The original Feast 0.5 release was mostly focused on

• Batch and streaming feature statistics and validation (Add Feature Validation to Feast #172)
• User authentication/authorization (Authentication and authorization #504)

The release was scheduled for early March

However, we are planning to include some pretty significant changes to the Feast code base

• We want to include the following operational improvements
  • Support for Redis Clusters (Add Support for Redis Cluster #502)
  • Support for handling out of order ingestion (Filter out of order feature rows #273 )
  • Support for compression or storage of features without field names to reduce the memory footprint in Redis (Optimize Redis memory foot print for ingestion / serving #515)
• Remove feature versions and enable automatic data migration. This is a big one. It would be a massive QOL improvement for our users. They wouldnt have to worry about feature versions any more, and any changes they make to a feature set would not drop their existing data. However, this requires a significant amount of development (Feature set versioning #386, Remove feature set versions from Feast serving API #462)
• Given the above change, we would need to refactor the existing storage code to be more modular. This modularity would come in the form of a storage module that is more cleanly separated from the existing codebase. This would make adding new stores possible. (Factor out storage key generation logic, to eliminate code duplication #402)
• Refactoring the current configuration files for both Feast Serving and Feast Core. (Clean up Feast configuration #525)
• Inclusion of a basic job management API. This would allow us to query/stop/restart jobs that population jobs that are currently being managed by Feast Core. (Add job management APIs #302)

The above won't be the only changes we propose, but they are the most significant ones. Given the scope of work, we don't expect 0.5 will land in March. Best estimate is mid-April.

The good news is that a lot of the functionality our users would have been waiting for in Feast 0.6 will now land in Feast 0.5.

Comments welcome!

[Jeffwan]

1. This link https://docs.feast.dev/roadmap is broken
2. Is storage API part of 0.5 release? Can we think 0.5 will give required interface to add cloud support for different vendors?

[woop]

1. This link https://docs.feast.dev/roadmap is broken
2. Is storage API part of 0.5 release? Can we think 0.5 will give required interface to add cloud support for different vendors?

1. Done.
2. Yes, it would make adding new stores possible. Please have a look and comment on Add general storage API and refactor existing store implementations #567.

[woop]

We're close to cutting this release and we are now trying to determine what goes into 0.5 and what goes into 0.6. There have been suggestions of reducing the scope of 0.5 to only the functionality that breaks APIs (version removal, data model changes, configuration changes), then after that we can release and merge new functionality (auth, batch statistics). These releases would follow directly on each other, so we wouldnt extend 0.6 by adding new scope.

The goal is to make the jump to 0.5 easier from 0.4, since many teams run Feast in production.

Based on the current PRs that we have open and the work that is complete, I'll suggest the following

0.5: "Technical" release

1. Everything that is in master (streaming statistics, Redis Cluster, compression, storage refactor, job management API).
2. Helm configuration refactor Refactor Feast Helm charts for better end user install experience #533
3. Feature & feature set labels Add feature and feature set labels, for metadata #536
4. Feature/Entity/Field data model changes Split Field model into distinct Feature and Entity objects #655
5. Unique ingestion ID for all batch ingestions Add unique ingestion id for all batch ingestions #656
6. We may also add constraints if necessary to ensure forward compatibility (Validate feature reference and project functionality #631)
7. Feature version removal Feature set versioning #386

0.6: Feature release

1. Batch statistics & validation Add computation and retrieval of batch feature statistics #612
2. Authentication & Authorization Add support for authentication and authorization #554
3. Online feature/entity status metadata Add Field Status metadata to Online Serving #658

Looking at this now, it seems like we have just swapped the original 0.5 and 0.6 😅

Would really love thoughts on this.