Does Qdrant have a cleanup process or job (or vector expiration)

[anthonyolazabal]

Hi,
I'm planning to use Qdrant as my main Vector Store and I'm wondering if there is any option to define an expiration of the validity of a vector in order to have it removed automatically or via a job so I limit the storage and memory consumption ?
Thanks in advance for your answer
Best regards

[timvisee]

Yes. This is handled by the optimizers which run in the background. This can be controlled by the vacuum threshold in the optimizers config for the vacuum optimizer.

By default it's 20% meaning that if 20% or more of the data is removed, it'll reoptimize and prune the removed data. Note that this is on segment level, not across your whole collection. I'd recommend to leave it at 20% if possible.

[anthonyolazabal]

So if I understand well I need to create a external job that search for points based on a timestamp, delete them and the vacuum will release the storage. There is no alternative to automatically delete the points after a certain period (like 1 day or more) ?

[timvisee]

Sorry, I misunderstood that you wanted to expire vectors yourself.

Currently, yes. You'd need your own job for this. Adding a timestamp and using a separate job to do the deletes should be perfectly fine. You're right, the vacuum optimizer will prune the vectors from storage.

Please make sure you define a payload index on the timestamp field to ensure fast deletion: https://qdrant.tech/documentation/concepts/indexing/#payload-index

There is no alternative to automatically delete the points after a certain period (like 1 day or more) ?

Correct. We're evaluating various options to achieve TLL/expiry. But this is quite hard to pull off in a distributed environment, and so we did not settle on anything yet.

[anthonyolazabal]

Ok noted, thanks for the quick answer Tim. Will go with the external job.