Add Faiss Agent Docs

docs/overview/component/README.md

@@ -67,6 +67,7 @@ In this section, we will describe what is Vald Agent and the corresponding compo
 
 Vald Agent provides functionalities to perform approximate nearest neighbor search.
 Agent-NGT uses [yahoojapan/NGT](https://github.com/yahoojapan/NGT) as a core library.
+And Agent-Faiss uses [facebookresearch/faiss](https://github.com/facebookresearch/faiss) as a core library.
 
 Each Vald Agent pod has its own vector data space because only several Vald Agents are selected to be inserted/updated in a single insert/update request.
 

docs/overview/component/agent.md

@@ -78,6 +78,56 @@ This image shows the mechanism to create NGT index.
 
 Please refer to [Go Doc](https://pkg.go.dev/github.com/vdaas/[email protected]/pkg/agent/core/ngt/service) for other functions.
 
+#### Vald Agent Faiss
+
+Vald Agent Faiss uses [Faiss](https://github.com/facebookresearch/faiss) as an algorithm.
+
+The main functions are the followings:
+
+- Insert
+  - Request to insert new vectors into the Faiss.
+  - Requested vectors are stored in the `vqueue`.
+  - Cache a fixed number of verctors for Faiss training.
+  - Once Faiss trained in CreateIndex, the vector is never cached for Faiss training.
+- Search
+  - Get the nearest neighbor vectors of the request vector from Faiss indexes.
+  - radius/epsilon is search config for NGT and has no meaning in Faiss.
+- Update
+  - Create a request to update the specific vectors to the new vectors.
+  - Requested vectors are stored in the `vqueue`.
+- Remove
+  - Create a request to remove the specific vectors from Faiss indexes.
+  - Requested vectors are stored in the `vqueue`.
+- Exist
+  - Check whether the specific vectors are already inserted or not.
+- CreateIndex
+  - Create a new Faiss index structure in memory using vectors stored in the `vqueue` and the existing Faiss index structure if it exists.
+  - If a certain number of vectors required for Faiss training are not cached, they will not be trained.
+  - If Faiss is not trained, no index is generated.
+- SaveIndex
+  - Save metadata about Faiss index information to the internal storage.
+
+Unimplemented functions are the followings:
+
+- GetObject
+- SearchByID
+- StreamXXX
+- MultiXXX
+
+<div class="notice">
+Same as Agent NGT, You have to control the duration of CreateIndex and SaveIndex by configuration.
+
+These methods don’t always run when getting the request.
+
+</div>
+
+<div class="warning">
+As you see, Vald Agent Faiss can only search the nearest neighbors from the Faiss index.
+
+You have to wait to complete the CreateIndex and SaveIndex functions before searching.
+
+</div>
+
 ### Sidecar
 
 `Sidecar` saves the index metadata file to external storage like Amazon S3 or Google Cloud Storage.

docs/user-guides/configuration.md

@@ -175,6 +175,42 @@ When the setting parameter of Vald Agent NGT is shorter than the setting value o
 If this happens, the Index Manager may not function properly.
 </div>
 
+#### Faiss
+
+Vald Agent Faiss uses [facebookresearch/faiss][faiss] as a core library for searching vectors.
+The behaviors of Faiss can be configured by setting `agent.faiss` field object.
+
+The important parameters are the followings:
+
+- `agent.faiss.dimension`
+- `agent.faiss.distance_type`
+- `agent.faiss.m`
+- `agent.faiss.metric_type`
+- `agent.faiss.nbits_per_idx`
+- `agent.faiss.nlist`
+
+Users should configure these parameters first to fit their use case.
+For further details, please read [the Fiass wiki][faiss-wiki].
+
+Vald Agent Faiss has a feature to start indexing automatically.
+The behavior of this feature can be configured with these parameters:
+
+- `agent.faiss.auto_index_duration_limit`
+- `agent.faiss.auto_index_check_duration`
+- `agent.faiss.auto_index_length`
+
+<div class="notice">
+While the Vald Agent Faiss is in the process of creating indexes, it will ignore all search requests to the target pods.
+</div>
+
+<div class="warning">
+When deploying Vald Index Manager, the above parameters should be set much longer than the Vald Index Manager settings (Please refer to the Vald Index Manager section) or minus value.<BR>
+E.g., set agent.faiss.auto_index_duration_limit to "720h" or "-1h" and agent.faiss.auto_index_check_duration to "24h" or "-1h".<BR>
+This is because the Vald Index Manager accurately grasps the index information of each Vald Agent Faiss and controls the execution timing of indexing.<BR><BR>
+When the setting parameter of Vald Agent Faiss is shorter than the setting value of Vald Index Manager, Vald Agent Faiss may start indexing by itself without the execution command from Vald Index Manager.
+If this happens, the Index Manager may not function properly.
+</div>
+
 #### Resource requests and limits, Pod priorities
 
 Because the Vald Agent pod places indexes on memory, termination of agent pods causes loss of indexes.
@@ -371,3 +407,5 @@ For further details, there are references to the Helm values in the Vald GitHub
 [kubernetes-topology-spread-constraints]: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
 [yj-ngt]: https://github.com/yahoojapan/NGT
 [yj-ngt-wiki]: https://github.com/yahoojapan/NGT/wiki
+[faiss]: https://github.com/facebookresearch/faiss
+[faiss-wiki]: https://github.com/facebookresearch/faiss/wiki