Feature: remove vector points when unassociating datasource from collection

[kunwar31]

Solves #317

[chiragjn]

This change itself is fine I guess

Ideally,

• delete should be implemented directly instead of having to fetch the data points first. E.g. https://qdrant.tech/documentation/concepts/points/#delete-points
• Our vector store implementations have a hidden limit of 1e6 on points scroll. We should not have any limits, rather pagination with a batch size

  cognita/backend/modules/vector_db/qdrant.py

  Lines 235 to 237 in 783634c

  	if len(data_point_vectors) > MAX_SCROLL_LIMIT:
  	stop = True
  	break

• This deletion should also be async worker job because the operation might take much more longer

[chiragjn]

That being said this change might still be useful for collections with less than 10k data points

[kunwar31]

• This deletion should also be async worker job because the operation might take much more longer

can use the same process_pool here used in #321 , with the same caveats.
If we already have 2 use cases for this, better to think about queue & worker based approach

[kunwar31]

Changes:

1. Deletion uses run_in_executor, so no blocking of event loop
2. added yield_data_point_vector_batches. list_data_point_vectors is now implemented using it.
3. used yield_data_point_vector_batches and delete per batch, to reduce memory footprint
4. Created delete_data_point_vectors_by_data_source method directly in base vectorDB class, so it can be used for all DBs

Caveats:

1. delete should be implemented directly instead of having to fetch the data points first, atleast for the vectorDBs that support it.
   • will need to overide delete_data_point_vectors_by_data_source method for each vectorDB
   • create a new issue for this? too much going on in this PR
2. singlestore does not have batched yield method. so the 1e6 scroll limit applies

Ready for review @chiragjn

[Blakeinstein]

@kunwar31 Apologies for the delay, can you look into rebasing this PR?

[mnvsk97]

@kunwar31 Apologies for the delay, can you look into rebasing this PR?

Hi @kunwar31, I've rebased this branch with the main branch. Please validate again with all the vector databases that we support and let us know.

[mnvsk97]

I am closing this PR since many changes have been made to collections and data sources APIs since this PR was creted.