Qdrant indexing time

[nairajay2k]

How does the indexing time vary

for a single node with say 10 million records
vs
10 nodes with 10 million records EACH in a cluster(So total records is 10 times).

Please let me know the answer for

1. vector indexing
2. payload indexing

[timvisee]

In this case, the indexing time would be about the same. That is because indexing happens on each node separately. Whether you have a single node with 10m records, or 10 nodes with each 10m records, each node still has the same amount to index.

If you'd have 10m records in total across all nodes, and have some replication across them, then indexing would likely be a lot faster. That is because you'd have a lot more CPU available per record for indexing.

[nairajay2k]

Thanks Tim for confirming.

[nairajay2k]

What about replication? If there is there are n replicas, would indexing time vary as n(as the total data has now gone up by n times)??
And how does RAM required for vector and payload indexing vary as m(number of records)??

[timvisee]

What about replication? If there is there are n replicas, would indexing time vary as n(as the total data has now gone up by n times)?? And how does RAM required for vector and payload indexing vary as m(number of records)??

The more points you have to index, the longer indexing will take. Indexing time is likely O(n log n). RAM requirements likely grow linearly. You may choose to use quantization and/or put data on disk to make RAM requirements less extreme.

[nairajay2k]

Is there a sequence in which indexing is recommended?? If both payload and vector indexes are required, should one do vector first and then payload or the other way around?? Does it even matter??