Redis Online Store - Truncate and Load

[umesh532]

Expected Behavior

For every materialize run, expected it to overwrite the redis online DB with new data

Current Behavior

It appends the data with every materialize run and also TTL configuration is not working.

Steps to reproduce

Materialize the data for overlapping dates ( start or end date)

Specifications

For our ML use case:

• Ingesting the data into Big query (offline store) and redis (online store)
• With every run, only the latest data is supposed to be served to Model.

My observations are:

• There is no direct way to read all data records from feast Online DB.

• It requires to pass the list of entity values when reading the data from online.

• This makes it more complicated as we need to build this list by querying the recent data from offline DB.

• Version: 0.15

• Platform: Python

• Subsystem:

Possible Solution

Is there any plan to implement SQL Engine like interface to read the data from online DB? Building the entity DF is making it more complex.
Implement the solution with an ability to overwrite the data in online DB by passing overwrite property when executing the materialize command.

[pyalex]

Hey @umesh532, indeed TTL is not currently assigned to Redis key. We know about this issue and we're still discussing how this functionality should be implemented. However, you still can overwrite your existing keys with new values. A new row with the same key just needs to have a more recent timestamp. I believe there was no intention by design to wipe out all existing keys before each materialization but rather update existing ones there's a new value. All materializations are essentially incremental.

[pyalex]

Regarding the second part, it's currently impossible to make open range or SQL requests to an online store and it's not designed to support that. It's specifically optimized for key-oriented requests. And at least as of now, there are no plans to change that.