diff --git a/content/docs/user-guides/sdks.md b/content/docs/user-guides/sdks.md index eef5b9d12..316bc1d7f 100644 --- a/content/docs/user-guides/sdks.md +++ b/content/docs/user-guides/sdks.md @@ -1,6 +1,6 @@ --- title: "Sdks_user Guides" -date: 2024-02-15T17:10:16+09:00 +date: 2024-10-28T06:57:15Z draft: false weight: 1400 description: Sends request and receives response from the Vald cluster @@ -27,7 +27,6 @@ Here is the list of Official SDKs. - [Java](https://github.com/vdaas/vald-client-java) - [Python](https://github.com/vdaas/vald-client-python) - [Node.js](https://github.com/vdaas/vald-client-node) -- [Clojure](https://github.com/vdaas/vald-client-clj) ## How to use SDKs diff --git a/content/docs/v1.7/api/_index.md b/content/docs/v1.7/api/_index.md index d1ddbb62b..a80a43882 100644 --- a/content/docs/v1.7/api/_index.md +++ b/content/docs/v1.7/api/_index.md @@ -1,6 +1,6 @@ --- title: "Api_v1.7/Api" -date: 2024-08-08T10:35:18+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 0 --- @@ -22,7 +22,6 @@ The APIs overview tables is here: | Upsert Service | Update the exists vector(s) in the Vald Agent Pods or Insert new vector(s) into the Vald Agent Pods if the vector is not exists. | [Upsert](/docs/v1.7/api/upsert#upsert-rpc)
[StreamUpsert](/docs/v1.7/api/upsert#streamupsert-rpc)
[MultiUpdate](/docs/v1.7/api/upsert#multiupsert-rpc) | [Vald Upsert APIs](/docs/v1.7/api/upsert) | | Search Service | Search nearest neighbor vectors using query (vectors or IDs).
There are 2 types of Search methods: Search (means ANN Search) and LinearSearch. | [Search](/docs/v1.7/api/search#search-rpc)
[SearchByID](/docs/v1.7/api/search#searchbyid-rpc)
[StreamSearch](/docs/v1.7/api/search#streamsearch-rpc)
[StreamSearchByID](/docs/v1.7/api/search#streamsearchbyid-rpc)
[MultiSearch](/docs/v1.7/api/search#multisearch-rpc)
[MultiSearchByID](/docs/v1.7/api/search#multisearchbyid-rpc)
[LinearSearch](/docs/v1.7/api/search#linearsearch-rpc)
[LinearSearchByID](/docs/v1.7/api/search#linearsearchbyid-rpc)
[StreamLinearSearch](/docs/v1.7/api/search#streamlinearsearch-rpc)
[StreamLinearSearchByID](/docs/v1.7/api/search#streamlinearsearchbyid-rpc)
[MultiLinearSearch](/docs/v1.7/api/search#multilinearsearch-rpc)
[MultiLinearSearchByID](/docs/v1.7/api/search#multilinearsearchbyid-rpc) | [Vald Search APIs](/docs/v1.7/api/search) | | Remove Service | Remove the exists vector(s) from the Vald Agent Pods. | [Remove](/docs/v1.7/api/remove#remove-rpc)
[StreamRemove](/docs/v1.7/api/remove#streamremove-rpc)
[MultiRemove](/docs/v1.7/api/remove#multiremove-rpc) | [Vald Remove APIs](/docs/v1.7/api/remove) | -| Flush Service | Remove the exists all vector(s) from the Vald Agent Pods. | [Flush](/docs/v1.7/api/flush#flush-rpc)
| [Vald Flush APIs](/docs/v1.7/api/flush) | | Object Service | Get information of vector(s) indexed into the Vald Agent Pods. | [Exists](/docs/v1.7/api/object#exists-rpc)
[GetObject](/docs/v1.7/api/object#getobject-rpc)
[StreamGetObject](/docs/v1.7/api/object#streamgetobject-rpc) | [Vald Object APIs](/docs/v1.7/api/object) |
diff --git a/content/docs/v1.7/api/build_proto.md b/content/docs/v1.7/api/build_proto.md index 07dfa0cbc..24c4005d8 100644 --- a/content/docs/v1.7/api/build_proto.md +++ b/content/docs/v1.7/api/build_proto.md @@ -1,6 +1,6 @@ --- title: "Build_proto_v1.7/Api" -date: 2024-08-08T10:35:18+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 1100 description: How to build gRPC proto files for calling API to your Vald cluster @@ -238,9 +238,9 @@ There are many tools for building proto in Rust, we use [tonic](https://github.c } ``` - 1. `src/client.rs` + 1. `src/clinet.rs` - There are 4 steps in `src/client.rs`: + There are 4 steps in `src/clinet.rs`: 1. Load dataset 1. Insert vector to Vald cluster diff --git a/content/docs/v1.7/api/filter-gateway.md b/content/docs/v1.7/api/filter-gateway.md index 243ca5f2d..a23c3125b 100644 --- a/content/docs/v1.7/api/filter-gateway.md +++ b/content/docs/v1.7/api/filter-gateway.md @@ -1,6 +1,6 @@ --- title: "Filter Gateway_v1.7/Api" -date: 2024-08-08T10:35:18+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 800 description: How to use CRUD API with filter gateway @@ -13,7 +13,7 @@ menu: ## Overview -Filter Server is responsible for providing insert, update, upsert and search interface for `Vald Filter Gateway`. +Filter Servie is responsible for providing insert, update, upsert and search interface for `Vald Filter Gateway`. Vald Filter Gateway forward user request to user-defined ingress/egress filter components allowing user to run custom logic. @@ -1488,17 +1488,6 @@ service Filter { | id | string | | the vector ID | | distance | float | | the distance between result vector and request vector | -### Status Code - -| code | desc. | -| :--: | :---------------- | -| 0 | OK | -| 1 | CANCELLED | -| 3 | INVALID_ARGUMENT | -| 4 | DEADLINE_EXCEEDED | -| 5 | NOT_FOUND | -| 13 | INTERNAL | - ## MultiSearch RPC MultiSearch RPC is the method to search objects with multiple objects in **1** request. diff --git a/content/docs/v1.7/api/flush.md b/content/docs/v1.7/api/flush.md deleted file mode 100644 index c276a392f..000000000 --- a/content/docs/v1.7/api/flush.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: "Flush_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 -draft: false -weight: 600 -description: Remove all indexes from the Vald cluster -menu: - api: - parent: Api ---- - -# Vald Flush APIs - -## Overview - -Flush Service is responsible for removing all vectors that are indexed and uncommitted in the `vald-agent`. - -```rpc -service Flush { - - rpc Flush(payload.v1.Flush.Request) returns (payload.v1.Info.Index.Count) {} - -} -``` - -## Flush RPC - -Flush RPC is the method to remove all vectors. - -### Input - -- the scheme of `payload.v1.Flush.Request` - - ```rpc - message Flush { - message Request { - } - } - ``` - - - Flush.Request - - empty - -### Output - -- the scheme of `payload.v1.Info.Index.Count` - - ```rpc - message Object { - message Info_Index_Count { - uint32 stored = 0; - uint32 uncommitted = 0; - bool indexing = false; - bool saving = false; - } - } - ``` - - Object.Info_Index_Count - - | field | type | label | desc. | - | :---------: | :----- | :---- | :------------------------------------------------------------------------- | - | stored | uint32 | | count of indices. | - | uncommitted | uint32 | | count of uncommitted indices. | - | indexing | bool | | the state indicating whether `vald-agent` pods is present in the indexing. | - | saving | bool | | the state indicating whether `vald-agent` pods is present in the saving. | - -### Status Code - -| code | desc. | -| :--: | :---------------- | -| 0 | OK | -| 1 | CANCELLED | -| 3 | INVALID_ARGUMENT | -| 4 | DEADLINE_EXCEEDED | -| 5 | NOT_FOUND | -| 13 | INTERNAL | diff --git a/content/docs/v1.7/api/insert.md b/content/docs/v1.7/api/insert.md index ae0b252f0..f48bb63ac 100644 --- a/content/docs/v1.7/api/insert.md +++ b/content/docs/v1.7/api/insert.md @@ -1,6 +1,6 @@ --- title: "Insert_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 100 description: Insert vectors to the Vald cluster @@ -108,10 +108,9 @@ Inset RPC is the method to add a new single vector. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -234,10 +233,9 @@ It's the recommended method to insert a large number of vectors. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -357,10 +355,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting diff --git a/content/docs/v1.7/api/mirror-gateway.md b/content/docs/v1.7/api/mirror-gateway.md deleted file mode 100644 index b0dae82a1..000000000 --- a/content/docs/v1.7/api/mirror-gateway.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: "Mirror Gateway_v1.7/Api" -date: 2024-02-15T17:12:23+09:00 -draft: false -weight: 900 -description: API description when operating Vald Mirror Gateway -menu: - api: - parent: Api ---- - -# Vald Mirror Gateway APIs - -## Overview - -Mirror Service is responsible for providing the `Register` interface for the Vald Mirror Gateway. - -```rpc -service Mirror { - rpc Register(payload.v1.Mirror.Targets) returns (payload.v1.Mirror.Targets) {} -} -``` - -## Register RPC - -Register RPC is the method to register other Vald Mirror Gateway targets. - -### Input - -- the scheme of `payload.v1.Mirror.Targets`. - - ```rpc - message Mirror { - message Target { - string host = 1; - uint32 port = 2; - } - - message Targets { - repeated Target targets = 1; - } - } - ``` - - - Mirror.Targets - - | field | type | label | required | desc. | - | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | - | targets | Mirror.Target | repeated(Array[Mirror.Target]) | \* | The multiple target information. | - - - Mirror.Target - - | field | type | label | required | desc. | - | :---: | :----- | :---- | :------: | :------------------- | - | host | string | | \* | The target hostname. | - | port | uint32 | | \* | The target port. | - -### Output - -- the scheme of `payload.v1.Mirror.Targets`. - - ```rpc - message Mirror { - message Target { - string host = 1; - uint32 port = 2; - } - - message Targets { - repeated Target targets = 1; - } - } - ``` - - - Mirror.Targets - - | field | type | label | required | desc. | - | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | - | targets | Mirror.Target | repeated(Array[Mirror.Target]) | | The multiple target information. | - - - Mirror.Target - - | field | type | label | required | desc. | - | :---: | :----- | :---- | :------: | :------------------- | - | host | string | | \* | The target hostname. | - | port | uint32 | | \* | The target port. | - -### Status Code - -| code | desc. | -| :--: | :---------------- | -| 0 | OK | -| 1 | CANCELLED | -| 3 | INVALID_ARGUMENT | -| 4 | DEADLINE_EXCEEDED | -| 13 | INTERNAL | diff --git a/content/docs/v1.7/api/object.md b/content/docs/v1.7/api/object.md index f75e89ff8..3e8484c51 100644 --- a/content/docs/v1.7/api/object.md +++ b/content/docs/v1.7/api/object.md @@ -1,6 +1,6 @@ --- title: "Object_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 700 description: Get object information in the Vald cluster @@ -78,7 +78,7 @@ Exists RPC is the method to check that a vector exists in the `vald-agent`. | 5 | NOT_FOUND | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -159,7 +159,7 @@ GetObject RPC is the method to get the metadata of a vector inserted into the `v | 5 | NOT_FOUND | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -253,7 +253,7 @@ Each Upsert request and response are independent. | 5 | NOT_FOUND | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting diff --git a/content/docs/v1.7/api/remove.md b/content/docs/v1.7/api/remove.md index 583c33734..c70689ab8 100644 --- a/content/docs/v1.7/api/remove.md +++ b/content/docs/v1.7/api/remove.md @@ -1,6 +1,6 @@ --- title: "Remove_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 500 description: Remove indexes from the Vald cluster @@ -109,10 +109,9 @@ Remove RPC is the method to remove a single vector. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -171,10 +170,10 @@ RemoveByTimestamp RPC is the method to remove vectors based on timestamp. - Remove.Timestamp - | field | type | label | required | description | - | :-------: | :------------------------ | :---- | :------: | :------------------------------------------------- | - | timestamp | int64 | | \* | The timestamp. | - | operator | Remove.Timestamp.Operator | | | The conditional operator. (default value is `Eq`). | + | field | type | label | required | description | + | :-------: | :------------------------ | :---- | :------: | :------------------------------------------------ | + | timestamp | int64 | | \* | The timestamp. | + | operator | Remove.Timestamp.Operator | | | The conditionl operator. (default value is `Eq`). | - Remove.Timestamp.Operator @@ -233,7 +232,7 @@ RemoveByTimestamp RPC is the method to remove vectors based on timestamp. | 5 | NOT_FOUND | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -352,10 +351,9 @@ It's the recommended method to remove a large number of vectors. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -380,7 +378,7 @@ gRPC has a message size limitation.
Please be careful that the size of the request exceeds the limit. -### Input +## Input - the scheme of `payload.v1.Remove.MultiRequest` @@ -473,10 +471,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting diff --git a/content/docs/v1.7/api/search.md b/content/docs/v1.7/api/search.md index 5f4711cc9..3e8f3dbb9 100644 --- a/content/docs/v1.7/api/search.md +++ b/content/docs/v1.7/api/search.md @@ -1,6 +1,6 @@ --- title: "Search_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 400 description: Search ANN vectors from the Vald cluster @@ -154,10 +154,9 @@ Search RPC is the method to search vector(s) similar to the request vector. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -275,10 +274,9 @@ The vector with the same requested ID should be indexed into the `vald-agent` be | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -411,10 +409,9 @@ Each Search request and response are independent. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -547,10 +544,9 @@ Each SearchByID request and response are independent. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -692,10 +688,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -711,7 +706,7 @@ Here are some common reasons and how to resolve each error. | NOT_FOUND | Search result is empty or insufficient to request result length. | Send a request with another vector or set min_num to a smaller value. | | INTERNAL | Target Vald cluster or network route has some critical error. | Check target Vald cluster first and check network route including ingress as second. | -## MultiSearchByID RPC +### MultiSearchByID RPC MultiSearchByID RPC is the method to search vectors with multiple IDs in **1** request. @@ -838,10 +833,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -954,10 +948,9 @@ LinearSearch RPC is the method to linear search vector(s) similar to the request | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -1072,10 +1065,9 @@ You will get a `NOT_FOUND` error if the vector isn't stored. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -1204,10 +1196,9 @@ Each LinearSearch request and response are independent. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -1336,10 +1327,9 @@ Each LinearSearchByID request and response are independent. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -1477,10 +1467,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -1496,7 +1485,7 @@ Here are some common reasons and how to resolve each error. | NOT_FOUND | Search result is empty or insufficient to request result length. | Send a request with another vector or set min_num to a smaller value. | | INTERNAL | Target Vald cluster or network route has some critical error. | Check target Vald cluster first and check network route including ingress as second. | -## MultiLinearSearchByID RPC +### MultiLinearSearchByID RPC MultiLinearSearchByID RPC is the method to linear search vectors with multiple IDs in **1** request. @@ -1619,10 +1608,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting diff --git a/content/docs/v1.7/api/status.md b/content/docs/v1.7/api/status.md index cfbfd7bbe..7f7a09b7b 100644 --- a/content/docs/v1.7/api/status.md +++ b/content/docs/v1.7/api/status.md @@ -1,6 +1,6 @@ --- title: "Status_v1.7/Api" -date: 2024-08-08T10:35:19+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 1000 description: API response description for each status @@ -27,7 +27,6 @@ The sections below describe the meaning of each code and why API returns. | 5 | [NOT_FOUND](#NOT_FOUND) | | 6 | [ALREADY_EXISTS](#ALREADY_EXISTS) | | 8 | [RESOURCE_EXHAUSTED](#RESOURCE_EXHAUSTED) | -| 10 | [ABORTED](#ABORTED) | | 13 | [INTERNAL](#INTERNAL) | | 14 | [UNAVAILABLE](#UNAVAILABLE) | @@ -113,25 +112,6 @@ Services that return status are all services. The most case in the Vald is that the query vector is too large. In other words, the vector dimension size in configuration is too large. -## ABORTED - -`ABORTED` means that the operation was aborted, usually due to a concurrency issue. - -It appears when: - -- A request is made during the process of creating indices. -- A request is made during the flushing process. - -Services that return status are: - -- [Insert Service](/docs/v1.7/api/insert) -- [Update Service](/docs/v1.7/api/update) -- [Upsert Service](/docs/v1.7/api/upsert) -- [Remove Service](/docs/v1.7/api/remove) -- [Search Service](/docs/v1.7/api/search) - -You have to wait for the completion of creating indices or flushing process before making the request again. - ## INTERNAL `INTERNAL` appears when some wrong happens in the Vald cluster. diff --git a/content/docs/v1.7/api/update.md b/content/docs/v1.7/api/update.md index e7647c9c4..4f84d8319 100644 --- a/content/docs/v1.7/api/update.md +++ b/content/docs/v1.7/api/update.md @@ -1,6 +1,6 @@ --- title: "Update_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 200 description: Update vectors indexed in the Vald cluster @@ -110,10 +110,9 @@ Update RPC is the method to update a single vector. | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -239,10 +238,9 @@ It's the recommended method to update the large amount of vectors. | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -365,10 +363,9 @@ Please be careful that the size of the request exceeds the limit. | 4 | DEADLINE_EXCEEDED | | 5 | NOT_FOUND | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting diff --git a/content/docs/v1.7/api/upsert.md b/content/docs/v1.7/api/upsert.md index e96f1d851..7c2338f05 100644 --- a/content/docs/v1.7/api/upsert.md +++ b/content/docs/v1.7/api/upsert.md @@ -1,6 +1,6 @@ --- title: "Upsert_v1.7/Api" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 300 description: Update or Insert vectors to the Vald cluster @@ -113,10 +113,9 @@ Upsert RPC is the method to update the inserted vector to a new single vector or | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -240,10 +239,9 @@ It’s the recommended method to upsert a large number of vectors. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting @@ -364,10 +362,9 @@ Please be careful that the size of the request exceeds the limit. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 6 | ALREADY_EXISTS | -| 10 | ABORTED | | 13 | INTERNAL | -Please refer to [Response Status Code](/docs/v1.7/status) for more details. +Please refer to [Response Status Code](./status) for more details. ### Troubleshooting diff --git a/content/docs/v1.7/contributing/coding-style.md b/content/docs/v1.7/contributing/coding-style.md index a0d056f88..bede5e85f 100644 --- a/content/docs/v1.7/contributing/coding-style.md +++ b/content/docs/v1.7/contributing/coding-style.md @@ -1,6 +1,6 @@ --- title: "Coding Style_v1.7/Contributing" -date: 2024-10-16T16:05:58+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 300 description: Coding style for Vald project @@ -25,7 +25,7 @@ Please also read the [Contribution guideline](/docs/v1.7/contributing/contributi Code formatting and naming conventions affect coding readability and maintainability. Every developer has a different coding style. Luckily Go provides tools to format source code and check for potential issues in the source code. -We recommend using [golines](https://github.com/segmentio/golines), [gofumpt](https://github.com/mvdan/gofumpt), and [goimports](https://github.com/golang/tools/tree/master/cmd/goimports) and [crlfmt](https://github.com/cockroachdb/crlfmt) to format the source code in Vald and [golangci-lint](https://github.com/golangci/golangci-lint) with the `--enable-all` option. +We recommend using [golines](https://github.com/segmentio/golines), [gofumpt](https://github.com/mvdan/gofumpt), and [goimports](https://github.com/golang/tools/tree/master/cmd/goimports) to format the source code in Vald and [golangci-lint](https://github.com/golangci/golangci-lint) with the `--enable-all` option. We suggest everyone install the plugin for your editor to format the code once you edit the code automatically and use `make format/go` command if you want to format the source code manually. But having tools to format source code does not mean you do not need to care about the formatting of the code, for example: @@ -102,7 +102,7 @@ Here are the naming conventions of the package: ```go // bad - package encode + package encodebase64 // good package base64 // inside the encoding/base64 folder @@ -284,7 +284,7 @@ The variable and the constant should be named as: ```go // bad - yamlProcessor := new(something) + yamlprocessor := new(something) // good yamlProcessor := new(something) @@ -326,7 +326,7 @@ In this section, rules also apply to the `function` (without receiver). The meth ```go // bad - func (s *something) someMethod() {} + func (s *something) somemethod() {} // bad func (s *something) some_method() {} @@ -340,7 +340,7 @@ In this section, rules also apply to the `function` (without receiver). The meth ```go // bad - func (s *something) generateRandomNumber() int {} + func (s *something) genereateRandomNumber() int {} // good func (s *something) genRandNum() int {} @@ -886,7 +886,7 @@ tests := []test { for _, tc := range tests { test := tc - t.Run(test.name, func(tt *testing.T) { + t.Run(test.name, func(tt *tesint.T) { checkFunc = defaultCheckFunc if test.checkFunc != nil { checkFunc = test.checkFunc @@ -1257,7 +1257,7 @@ In Vald, we can apply it to the different helper functions like `beforeFunc()` o ```go type test struct { ... - beforeFunc func(*testing.T) // helper function to initialize testing + beforeFunc func(*testing.T) // helper function to initialze testing afterFunc func(*testing.T) // helper function to cleanup } @@ -1329,7 +1329,7 @@ For example, we decided to mock the following implementation `Encoder`. package json type Encoder interface { - Encode(any) ([]byte, error) + Encode(interface{}) ([]byte, error) } ``` @@ -1338,7 +1338,7 @@ type encoder struct { encoder json.Encoder } -func (e *encoder) Encode(obj any) ([]byte, error) { +func (e *encoder) Encode(obj interface{}) ([]byte, error) { return e.encoder.Encode(obj) } ``` @@ -1349,10 +1349,10 @@ The following is an example of mock implementation: package json type MockEncoder struct { - EncoderFunc func(any) ([]byte, error) + EncoderFunc func(interface{}) ([]byte, error) } -func (m *MockEncoder) Encode(obj any) ([]byte, error) { +func (m *MockEncoder) Encode(obj interface{}) ([]byte, error) { return m.EncodeFunc(obj) } ``` @@ -1365,7 +1365,7 @@ tests := []test { name: "returns (byte{}, nil) when encode success" fields: fields { encoding: &json.MockEncoder { - EncoderFunc: func(any) ([]byte, error) { + EncoderFunc: func(interface{}) ([]byte, error) { return []byte{}, nil }, }, @@ -1418,4 +1418,4 @@ Since each package has its purpose, we decided to apply different strategies to For the rest of the `./pkg` packages, we decided to implement the unit test for the exported function only. -Please follow the [unit test guideline](/docs/v1.7/unit-test-guideline) for more details on how to implement good unit test. +Please follow the [unit test guideline](./unit-test-guideline) for more details on how to implement good unit test. diff --git a/content/docs/v1.7/contributing/development.md b/content/docs/v1.7/contributing/development.md index 8cc1134af..3f44c28a1 100644 --- a/content/docs/v1.7/contributing/development.md +++ b/content/docs/v1.7/contributing/development.md @@ -1,6 +1,6 @@ --- title: "Development_v1.7/Contributing" -date: 2024-08-08T10:35:16+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 200 description: Basic development guide for the Vald project @@ -36,7 +36,7 @@ This is the easiest way to start developing `Vald`. You can just open our [devco We don't officially have a setup documentation for now, but you can take a look at the [`Dockerfile`](https://github.com/vdaas/vald/blob/main/dockers/dev/Dockerfile). That's everything you need to build and test `Vald`, so you can use it as a reference. -> If you would like to use the `Dockerfile` directly, please note that `docker-in-docker` environment is required to run our E2E tests. +> If you would like to use the `Dockerfile` directlly, please note that `docker-in-docker` environment is required to run our E2E tests. > In devcontainer, [`VS Code` handles it for us](https://github.com/devcontainers/features/tree/main/src/docker-in-docker). ## Run tests diff --git a/content/docs/v1.7/contributing/reviewer-guideline.md b/content/docs/v1.7/contributing/reviewer-guideline.md deleted file mode 100644 index 95879ce21..000000000 --- a/content/docs/v1.7/contributing/reviewer-guideline.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: "Reviewer Guideline_v1.7/Contributing" -date: 2024-08-08T10:35:16+09:00 -draft: false -weight: 0 -description: "" -menu: - contributing: - parent: Contributing ---- - -# Guideline for the Pull Request reviewers - -This document serves as a guideline for those reviewing Pull Requests. - -This guideline is intended for internal developers. A separate guideline for external contributors is planned. - -## Purpose - -The purpose of this guideline is to clarify what reviewers should be aware of and to optimize communication, thereby facilitating efficient development of Vald. - -## Preparation - -This guideline assumes that the content of the Pull Request is appropriate. - -Consider splitting the Pull Request if the changes are extensive, have multiple intentions, or if the tests are insufficient. This makes it easier to review. - -Regarding the amount of change, opinions may vary, but in this project, we aim to keep the implementation of logic under 1000 lines, excluding automatically generated code such as Protocol Buffers. - -## Review request - -In this project, merging a Pull Request requires approval from two people. If you want a specific person to review, assign them explicitly; otherwise, assign random people. - -### Request notification - -People may notice requests through different means, but in this project, we prioritize the following methods of communication: - -1. If you are available to communicate with us via online meeting systems like Zoom, please feel free to notify us directly. -1. Send a mention in some way. For example, we use [Slack](https://join.slack.com/t/vald-community/shared_invite/zt-db2ky9o4-R_9p2sVp8xRwztVa8gfnPA) and we have [X account](https://twitter.com/vdaas_vald). -1. Email from GitHub is automatically sent when someone is assigned. - -If the reviewer does not notice the request, we follow the same priority for re-notifying. - -#### When Requested for Review - -Unless there are high-priority issues, you are in a meeting, or deeply focused on development, start the review immediately. - -## Approve condition - -- The content of the Pull Request is understood. -- The content can be understood at a glance. - - If it cannot be understood at a glance, request additional comments. -- There are no deficiencies in the test cases. -- The quality is suitable for release. - - Compatibility - - Security - - Performance - -Regarding quality, while there is always room for improvement, if the quality is sufficient for release, it is acceptable to approve. If there is potential for improvement, suggest it in the comments, and let the implementer decide whether to address it in the same Pull Request or a separate one. - -### What Reviewers Should Do - -- If the title or description is unclear, ask questions before reviewing the code. -- If any part of the code is not understood after a brief review, ask for clarification or additional comments instead of trying to understand it yourself. -- Verify if the test cases are sufficient. -- Approve immediately if there are no comments. -- If interrupted, respond even if the review is incomplete. -- Ignore other reviewers' comments. - -### What Reviewers Should Not Do - -- Try to decipher something that is not immediately understandable. -- Close the browser or tab or start something else if it is not understood. -- Attempt to verify the implementation in detail. - -### Communication Methods - -All communications do not need to be completed on GitHub. Using Slack or Zoom can make the review process more efficient. - -The following criteria often determine the choice of communication tool: - -- On GitHub Mainly used for specific questions or comments about the code. If the exchange is unsuitable for comments, use Slack or Zoom. -- On Slack, used for brief exchanges. If the discussion is likely to be lengthy or complex, switch to Zoom. -- On Zoom Used when more complex communication is necessary. - -Regardless of where communication takes place, ensure that the results are documented on GitHub or in the code as a record of the communication. - -## Effect measurement - -Once a month, a report is created at [Vald Issues](https://github.com/vdaas/vald/issues). - -This report includes the time taken from when a Pull Request is opened to when it is merged and a summary of this data. Regularly reflect on this information. diff --git a/content/docs/v1.7/contributing/unit-test-guideline.md b/content/docs/v1.7/contributing/unit-test-guideline.md index 558c01e3d..ab99bb349 100644 --- a/content/docs/v1.7/contributing/unit-test-guideline.md +++ b/content/docs/v1.7/contributing/unit-test-guideline.md @@ -1,6 +1,6 @@ --- title: "Unit Test Guideline_v1.7/Contributing" -date: 2024-10-16T16:05:58+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 400 description: Unit test guideline for Vald project @@ -99,7 +99,7 @@ It is not only a single input, but also multi inputs. If a function or method accepts multiple inputs, we should try to create test cases to cover all the inputs. -(Note: The below `calcSum()` is different function from `calcSum(val ...int32)` which we mention before.) +(Note: The below `calcSum()` is diffrent function from `calsSum(val ...int32)` which we mention before.) ```go func calcAverageDiff(val1 []int32, val2 []int32) (diff float64) { @@ -139,7 +139,7 @@ You have to create unit tests for error patterns as the same as success patterns #### Advanced -##### Robust boundary test +##### Robust boudary test The previous section is about the basic test cases. The (robust) boundary test should be applied to cover more test coverage. @@ -191,7 +191,7 @@ And, you may also create unit tests based on robust boundary tests or equivalenc But, we have to take care the Vald is developed using Go. As you know as Go has many coding features as other languages. -One of the features is that Go will convert a single value to a slice value when the Function or Method receives a variadic argument (e.g., `...[]int`, `...[]string`, `...any`, or etc.) as the input. +One of the features is that Go will convert a single value to a slice value when the Function or Method receives a variadic argument (e.g., `...[]int`, `...[]string`, `...interface{}`, or etc.) as the input. And we apply the table-driven test for running unit tests. For example, when we create the unit test of `func getMeta(...[]int)`, the test code will be more complex than other functions' tests which don't use variadic argument as the input, if we create the test for all input patterns. @@ -200,15 +200,15 @@ So, we define the basic unit case slightly different from [the basic test case]( This change is very clear and you can apply it easily. Our basic test case depends on the type of two variadic arguments. -1. When input is `...any` - - We have to write all test cases which satisfy `...any` as same as a [basic test case](#Basic). For example, `val = 1`, `val = "input"`, `val = []float64{2020.12}` and so on. -1. When input is not `...any` but `...[]int`, `...[]string` or etc. +1. When input is `...interface{}` + - We have to write all test cases which satisfy `...interface{}` as same as a [basic test case](#Basic). For example, `val = 1`, `val = "input"`, `val = []float64{2020.12}` and so on. +1. When input is not `...interface{}` but `...[]int`, `...[]string` or etc. - We have to create only slice pattern test cases, which is the same as not creating test cases with a single value. - We should test with boundary cases, for example, we should test with `val = []int{math.MaxInt64()}` when the input value is `...[]int`. Summarize Vald unit test guideline: -- Apply basic test case, but take care of input variable pattern, in particular, the variadic argument (`...any` or not) +- Apply basic test case, but take care of input variable pattern, in particular, the variadic argument (`...interface{}` or not) - Apply robust boundary tests, including edge cases (e.g., `math.MaxInt64()`) - Apply equivalence class testing when needed. diff --git a/content/docs/v1.7/overview/architecture.md b/content/docs/v1.7/overview/architecture.md index 32b67a4dd..73b1a9fb1 100644 --- a/content/docs/v1.7/overview/architecture.md +++ b/content/docs/v1.7/overview/architecture.md @@ -1,6 +1,6 @@ --- title: "Architecture_v1.7/Overview" -date: 2024-08-08T10:35:19+09:00 +date: 2024-10-28T06:58:08Z draft: false weight: 200 description: High-Level architecture design and overview of each component @@ -58,7 +58,7 @@ Here are the concepts of Vald. - Distributed vector spaces - All the vector data and indexes are distributed to Vald Agents in the Vald cluster. Whenever you search a vector in Vald cluster, all Vald agents can process parallel and merge the result by Vald LB Gateway. + All the vector data and indexes are distributed to Vald Agents in the Vald cluster. Whenever you search a vector in Vald cluster, all Vald agents can process parallelly and merge the result by Vald LB Gateway. - Kubernetes based diff --git a/content/docs/v1.7/overview/component/_index.md b/content/docs/v1.7/overview/component/_index.md index d85ccdd41..aa5a245e6 100644 --- a/content/docs/v1.7/overview/component/_index.md +++ b/content/docs/v1.7/overview/component/_index.md @@ -1,6 +1,6 @@ --- title: "Component_v1.7/Overview/Component" -date: 2024-02-15T17:12:26+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 0 --- @@ -74,7 +74,6 @@ In this section, we will describe what is Vald Agent and the corresponding compo Vald Agent provides functionalities to perform approximate nearest neighbor search. Agent-NGT uses [yahoojapan/NGT](https://github.com/yahoojapan/NGT) as a core library. -And Agent-Faiss uses [facebookresearch/faiss](https://github.com/facebookresearch/faiss) as a core library. Each Vald Agent pod has its own vector data space because only several Vald Agents are selected to be inserted/updated in a single insert/update request. diff --git a/content/docs/v1.7/overview/component/agent.md b/content/docs/v1.7/overview/component/agent.md index e35e60a88..a3c020efa 100644 --- a/content/docs/v1.7/overview/component/agent.md +++ b/content/docs/v1.7/overview/component/agent.md @@ -1,6 +1,6 @@ --- title: "Agent_v1.7/Overview/Component" -date: 2024-10-16T16:06:01+09:00 +date: 2024-10-28T06:58:08Z draft: false weight: 100 description: Stores index data and Search ANN vectors @@ -85,59 +85,9 @@ You have to wait to complete the CreateIndex and SaveIndex functions before sear This image shows the mechanism to create NGT index. -NGT Index Creation Mechanism + -Please refer to [Go Doc](https://pkg.go.dev/github.com/vdaas/vald@v1.7.14/pkg/agent/core/ngt/service) for other functions. - -#### Vald Agent Faiss - -Vald Agent Faiss uses [Faiss](https://github.com/facebookresearch/faiss) as an algorithm. - -The main functions are the followings: - -- Insert - - Request to insert new vectors into the Faiss. - - Requested vectors are stored in the `vqueue`. - - Cache a fixed number of verctors for Faiss training. - - Once Faiss trained in CreateIndex, the vector is never cached for Faiss training. -- Search - - Get the nearest neighbor vectors of the request vector from Faiss indexes. - - radius/epsilon is search config for NGT and has no meaning in Faiss. -- Update - - Create a request to update the specific vectors to the new vectors. - - Requested vectors are stored in the `vqueue`. -- Remove - - Create a request to remove the specific vectors from Faiss indexes. - - Requested vectors are stored in the `vqueue`. -- Exist - - Check whether the specific vectors are already inserted or not. -- CreateIndex - - Create a new Faiss index structure in memory using vectors stored in the `vqueue` and the existing Faiss index structure if it exists. - - If a certain number of vectors required for Faiss training are not cached, they will not be trained. - - If Faiss is not trained, no index is generated. -- SaveIndex - - Save metadata about Faiss index information to the internal storage. - -Unimplemented functions are the followings: - -- GetObject -- SearchByID -- StreamXXX -- MultiXXX - -
-Same as Agent NGT, You have to control the duration of CreateIndex and SaveIndex by configuration. - -These methods don’t always run when getting the request. - -
- -
-As you see, Vald Agent Faiss can only search the nearest neighbors from the Faiss index. - -You have to wait to complete the CreateIndex and SaveIndex functions before searching. - -
+Please refer to [Go Doc](https://pkg.go.dev/github.com/vdaas/vald@v1.3.1/pkg/agent/core/ngt/service) for other functions. ### Sidecar diff --git a/content/docs/v1.7/overview/component/mirror-gateway.md b/content/docs/v1.7/overview/component/mirror-gateway.md deleted file mode 100644 index bcc562960..000000000 --- a/content/docs/v1.7/overview/component/mirror-gateway.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: "Mirror Gateway_v1.7/Overview/Component" -date: 2024-02-15T17:12:27+09:00 -draft: false -weight: 600 -description: Gateway for operating Vald cluster compatible with Multi-region and Multi AZ -menu: - component: - parent: Component ---- - -# Vald Mirror Gateway - -Vald Mirror Gateway is an optional component of the Vald, allowing the vector data to be synchronized across multiple Vald clusters. - -This component makes it possible to enhance availability during a cluster failure. - - - -## Responsibility - -Vald Mirror Gateway is responsible for the followings: - -- Forward user requests ([Insert](/docs/v1.7/api/insert) / [Upsert](/docs/v1.7/api/upsert) / [Update](/docs/v1.7/api/update) / [Remove](/docs/v1.7/api/remove)) to the other Vald Mirror Gateways in the same group. -- Manages the state of indexes stored in all clusters to ensure they are consistent. - -## Features - -This chapter shows the main features to fulfill Vald Mirror Gateway’s role: - -- Full mesh connection -- Request forwarding -- Automatic rollback on failure - -### Full mesh connection - - - -The Vald Mirror Gateway is designed to interconnect with Vald Mirror Gateways in other Vald clusters. - -Vald Mirror Gateway uses a Custom Resource called the `ValdMirrorTarget` to manage the connection destination information between Vald Mirror Gateways. - -The `ValdMirrorTarget` is a Custom Resource related to the connection destination to other Vald Mirror Gateway. - -When two Vald clusters contain Vald Mirror Gateways, Vald Mirror Gateways can send the request to each other by applying `ValdMirrorTarget`. - -For more information about `ValdMirrorTarget` configuration, please refer to [Custom Resource Configuration](/docs/v1.7/user-guides/mirroring-configuration). - -### Request forwarding - - - -The Vald Mirror Gateway forwards the incoming user request ([Insert](/docs/v1.7/api/insert) / [Upsert](/docs/v1.7/api/upsert) / [Update](/docs/v1.7/api/update) / [Remove](/docs/v1.7/api/remove)) to other Vald Mirror Gateways. -Then, while forwarding the user request, the Vald Mirror Gateway bypasses the incoming user request to Vald LB Gateway in its own cluster. - -On the other hand, if the incoming user request is an [Object API](/docs/v1.7/api/object) or [Search API](/docs/v1.7/api/search), it is bypassed to only a Vald LB Gateway in its own cluster without forwarding it to other Vald Mirror Gateways. - -### Continuous processing on failure - -The request may fail at the forwarding destination or the bypass destination. - -If some of the requests fails, the processing continues based on their status code. - -Here's an overview of how the Mirror Gateway handles failures for each type of request. - -For more information about status code, please refer to [Mirror Gateway Troubleshooting](/docs/v1.7/troubleshooting/mirror-gateway). - -- Insert Request - - - If the target host returns a status code of `ALREADY_EXISTS`, the Update request is sent to this host. - - If the target host returns a status code other than `OK`, `ALREADY_EXISTS`, the Mirror Gateway returns that status code without continuous processing. - - If all target hosts return a status code `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - - If all target hosts return a status code `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - -- Update Request - - - If the target host returns a status code `NOT_FOUND`, the Insert request is sent to this host. - - If the target host returns a status code other than `OK`, `ALREADY_EXISTS`, the Mirror Gateway returns that status code without continuous processing. - - If all target hosts return a status code `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - - If all target hosts return a status code `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - -- Upsert Request - - - If all target hosts return a status code `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - - If the target host returns a status code other than `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns that status code without continuous processing. - - If all target hosts return a status code `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - -- Remove/RemoveByTimestamp Request - - - If all target hosts return a status code `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND`. - - If the target host returns a status code other than `OK` or `NOT_FOUND`, the Mirror Gateway returns that status code without continuous processing. - - If all target hosts return a status code `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. diff --git a/content/docs/v1.7/overview/data-flow.md b/content/docs/v1.7/overview/data-flow.md index 1cf3c591b..5df631903 100644 --- a/content/docs/v1.7/overview/data-flow.md +++ b/content/docs/v1.7/overview/data-flow.md @@ -1,6 +1,6 @@ --- title: "Data Flow_v1.7/Overview" -date: 2024-08-08T10:35:19+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 300 description: Data flow inside Vald for each operation @@ -88,7 +88,7 @@ Upsert request updates the existing vector if the same vector ID already exists -When the user upsert a vector to Vald: +When the user upserts a vector to Vald: 1. Vald LB Gateway receives the request from the user, including the user's vector ID(s) and the vector(s). 2. Vald LB Gateway will broadcast an existing check request to the Vald Agent(s) to check if the vector exists. diff --git a/content/docs/v1.7/performance/continuous-benchmark.md b/content/docs/v1.7/performance/continuous-benchmark.md deleted file mode 100644 index dcf63b655..000000000 --- a/content/docs/v1.7/performance/continuous-benchmark.md +++ /dev/null @@ -1,513 +0,0 @@ ---- -title: "Continuous Benchmark_v1.7/Performance" -date: 2024-10-16T16:05:59+09:00 -draft: false -weight: 300 -description: Performance verification using continuous benchmark tool -menu: - performance: - parent: Performance ---- - -# Continuous Benchmark Tool - -## What is the Continuous Benchmark Tool? - -Continuous Benchmark Tool allows you to continuously benchmark the Vald cluster. - -Assumed use case is: - -- Verification with workload close to the production environment -- Verification before service installation when Vald version up - -## Architecture - -Continuous Benchmark Tool has following 2 components: - -- Benchmark Operator: Manages benchmark jobs -- Benchmark Job: Executes CRUD request to the target Vald cluster - -## Benchmark component and its feature - -### Benchmark Operator - -- Manages benchmark jobs according to applied manifest. -- Apply method: - - Scenario method: one manifest with multiple benchmark jobs - - Job method: one manifest with one benchmark job - -### Benchmark Job - -- Executes CRUD request to the target Vald cluster based on defined config. -- Execute steps are: - 1. Load dataset (valid only for HDF5 format) - 1. Execute request with load dataset - -## Benchmark CRD - -Benchmark workload can be set by applying the Kubernetes Custom Resources(CRDs), `ValdBenchmarkScenarioResource` or `ValdBenchmarkJobResource`. -Benchmark Operator manages benchmark job according to the applied manifest. - -### ValdBenchmarkJob - -[`ValdBenchmarkJob`](https://github.com/vdaas/vald/blob/main/charts/vald-benchmark-operator/crds/valdbenchmarkjob.yaml) is used for executing single benchmark job. - -And, Benchmark Operator also applies it to the Kubernetes cluster based on `ValdBenchmarkScenarioResource`. - -**main properties** - -| Name | mandatory | Description | type | sample | -| :------------------------- | :-------- | :-------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | -| target | \* | target Vald cluster | object | ref: [target](#target-prop) | -| dataset | \* | dataset information | object | ref: [dataset](#dataset-prop) | -| job_type | \* | execute job type | string enum: [insert, update, upsert, remove, search, getobject, exists] | search | -| repetition | | the number of job repetitions
default: `1` | integer | 1 | -| replica | | the number of job concurrent job executions
default: `1` | integer | 2 | -| rps | | designed request per sec to the target cluster
default: `1000` | integer | 1000 | -| concurrency_limit | | goroutine count limit for rps adjustment
default: `200` | integer | 20 | -| ttl_seconds_after_finished | | time until deletion of Pod after job end
default: `600` | integer | 120 | -| insert_config | | request config for insert job | object | ref: [config](#insert-cfg-props) | -| update_config | | request config for update job | object | ref: [config](#update-cfg-props) | -| upsert_config | | request config for upsert job | object | ref: [config](#upsert-cfg-props) | -| search_config | | request config for search job | object | ref: [config](#search-cfg-props) | -| remove_config | | request config for remove job | object | ref: [config](#remove-cfg-props) | -| object_config | | request config for object job | object | ref: [config](#object-cfg-props) | -| client_config | | gRPC client config for running benchmark job
Tune if can not getting the expected performance with default config. | object | ref: [defaults.grpc](https://github.com/vdaas/vald/blob/main/charts/vald/values.yaml) | -| server_config | | server config for benchmark job pod
Tune if can not getting the expected performance with default config. | object | ref: [defaults.server_config](https://github.com/vdaas/vald/blob/main/charts/vald/values.yaml) | - - - -**target** - -- target Vald cluster information -- type: object - -| property | mandatory | description | type | sample | -| :------- | :-------- | :-------------------- | :------ | :-------- | -| host | \* | target cluster's host | string | localhost | -| port | \* | target cluster's port | integer | 8081 | - - - -**dataset** - -- dataset which is used for executing job operation -- type: object - -| property | mandatory | description | type | sample | -| :---------- | :-------- | :--------------------------------------------------------------------------------------------------- | :------------------------------------- | :------------ | -| name | \* | dataset name | string enum: [fashion-mnist, original] | fashion-mnist | -| group | \* | group name | string enum: [train, test, neighbors] | train | -| indexes | \* | amount of index size | integer | 1000000 | -| range | \* | range of indexes to be used (if there are many indexes, the range will be corrected on the job side) | object | - | -| range.start | \* | start of range | integer | 1 | -| range.end | \* | end of range | integer | 1000000 | -| url | | the dataset url. It should be set when set `name` as `original` | string | | - - - -**insert_config** - -- rpc config for insert request -- type: object - -| property | mandatory | description | type | sample | -| :---------------------- | :-------- | :----------------------------------------------------------------------------------------------------------- | :----- | :--------- | -| skip_strict_exist_check | | Check whether the same vector is already inserted or not.
The ID should be unique if the value is `true`. | bool | false | -| timestamp | | The timestamp of the vector inserted.
If it is N/A, the current time will be used. | string | 1707272658 | - - - -**update_config** - -- rpc config for update request -- type: object - -| property | mandatory | description | type | sample | -| :---------------------- | :-------- | :----------------------------------------------------------------------------------------------------------- | :----- | :--------- | -| skip_strict_exist_check | | Check whether the same vector is already inserted or not.
The ID should be unique if the value is `true`. | bool | false | -| timestamp | | The timestamp of the vector inserted.
If it is N/A, the current time will be used. | string | 1707272658 | -| disable_balanced_update | | A flag to disable balanced update (split remove -> insert operation) during update operation. | bool | false | - - - -**upsert_config** - -- rpc config for upsert request -- type: object - -| property | mandatory | description | type | sample | -| :---------------------- | :-------- | :----------------------------------------------------------------------------------------------------------- | :----- | :--------- | -| skip_strict_exist_check | | Check whether the same vector is already inserted or not.
The ID should be unique if the value is `true`. | bool | false | -| timestamp | | The timestamp of the vector inserted.
If it is N/A, the current time will be used. | string | 1707272658 | -| disable_balanced_update | | A flag to disable balanced update (split remove -> insert operation) during update operation. | bool | false | - - - -**upsert_config** - -- rpc config for search request -- type: object - -| property | mandatory | description | type | sample | -| :-------------------- | :-------- | :---------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- | :----- | -| radius | | The search radius.
default: `-1` | number | -1 | -| epsilon | | The search coefficient.
default: `0.05` | number | 0.05 | -| num | \* | The maximum number of results to be returned. | integer | 10 | -| min_num | | The minimum number of results to be returned. | integer | 5 | -| timeout | | Search timeout in nanoseconds
default: `10s` | string | 3s | -| enable_linear_search | | A flag to enable linear search operation for estimating search recall.
If it is `true`, search operation with linear operation will execute. | bool | false | -| aggregation_algorithm | | The search aggregation algorithm option.
default: `Unknown` | string enum: ["Unknown", "ConcurrentQueue", "SortSlice", "SortPoolSlice", "PairingHeap"] | | - - - -**remove_config** - -- rpc config for remove request -- type: object - -| property | mandatory | description | type | sample | -| :---------------------- | :-------- | :----------------------------------------------------------------------------------------------------------- | :----- | :--------- | -| skip_strict_exist_check | | Check whether the same vector is already inserted or not.
The ID should be unique if the value is `true`. | bool | false | -| timestamp | | The timestamp of the vector inserted.
If it is N/A, the current time will be used. | string | 1707272658 | - - - -**object_config** - -- rpc config for get object request -- type: object - -| property | mandatory | description | type | sample | -| :-------------------- | :-------- | :---------------------------------------------------------- | :------- | :----- | -| filter_config.targets | | filter target host and port for bypassing filter component. | []object | | - -### ValdBenchmarkScenario - -[`ValdBenchmarkScenario`](https://github.com/vdaas/vald/blob/main/charts/vald-benchmark-operator/crds/valdbenchmarkscenario.yaml) is used for executing single or multiple benchmark job. - -Benchmark Operator decomposes manifest and creates benchmark resources one by one. -The `target` and `dataset` property are the global config for scenario, they can be overwritten when each job has own config. - -**main properties** - -| property | mandatory | description | type | sample | -| :------- | :-------- | :------------------------------------------------------------------------------------- | :----- | :-------------------------------------- | -| target | \* | target Vald cluster information
It will be overwritten when each job has own config | object | ref: [target](#target-prop) | -| dataset | \* | dataset information
It will be overwritten when each job has own config | object | ref: [dataset](#dataset-prop) | -| jobs | \* | benchmark job config
The jobs written above will be executed in order. | object | ref: [benchmark job](#valdbenchmarkjob) | - -## Deploy Benchmark Operator - -Continuous benchmark operator can be applied with `Helm` same as Vald cluster. - -It requires `ValdBenchmarkOperatorRelease` for deploying `vald-benchmark-operator`. - -It is not must to apply, so please edit and apply as necessary. - -
-`job.client_config` is used as the default gRPC client configuration for all benchmark jobs.
-Although it is possible to override using `ValdBenchmarkScenarioRelease` or `ValdBenchmarkJobRelease`, we recommend configuring common setting items here. -
- -
Sample ValdBenchmarkOperatorRelease YAML - -```yaml -# @schema {"name": "name", "type": "string"} -# name -- name of the deployment -name: vald-benchmark-operator -# @schema {"name": "time_zone", "type": "string"} -# time_zone -- time_zone -time_zone: "" -# @schema {"name": "image", "type": "object"} -image: - # @schema {"name": "image.repository", "type": "string"} - # image.repository -- image repository - repository: vdaas/vald-benchmark-operator - # @schema {"name": "image.tag", "type": "string"} - # image.tag -- image tag - tag: v1.7.5 - # @schema {"name": "image.pullPolicy", "type": "string", "enum": ["Always", "Never", "IfNotPresent"]} - # image.pullPolicy -- image pull policy - pullPolicy: Always -# @schema {"name": "job", "type": "object"} -job: - # @schema {"name": "job.image", "type": "object"} - image: - # @schema {"name": "job.image.repository", "type": "string"} - # image.repository -- job image repository - repository: vdaas/vald-benchmark-job - # @schema {"name": "job.image.tag", "type": "string"} - # image.tag -- image tag for job docker image - tag: v1.7.12 - # @schema {"name": "job.image.pullPolicy", "type": "string", "enum": ["Always", "Never", "IfNotPresent"]} - # image.pullPolicy -- image pull policy - pullPolicy: Always - # @schema {"name": "job.client_config", "type": "object"} - # client_config -- gRPC client config for request to the Vald cluster - client_config: - # @schema {"name": "job.client_config.addrs", "type": "array", "items": {"type": "string"}} - # job.client_config.addrs -- gRPC client addresses - addrs: [] - # @schema {"name": "job.client_config.health_check_duration", "type": "string"} - # job.client_config.health_check_duration -- gRPC client health check duration - health_check_duration: "1s" - # @schema {"name": "job.client_config.connection_pool", "type": "object"} - connection_pool: - # @schema {"name": "job.client_config.connection_pool.enable_dns_resolver", "type": "boolean"} - # job.client_config.connection_pool.enable_dns_resolver -- enables gRPC client connection pool dns resolver, when enabled vald uses ip handshake exclude dns discovery which improves network performance - enable_dns_resolver: true - # @schema {"name": "job.client_config.connection_pool.enable_rebalance", "type": "boolean"} - # job.client_config.connection_pool.enable_rebalance -- enables gRPC client connection pool rebalance - enable_rebalance: true - # @schema {"name": "job.client_config.connection_pool.rebalance_duration", "type": "string"} - # job.client_config.connection_pool.rebalance_duration -- gRPC client connection pool rebalance duration - rebalance_duration: 30m - # @schema {"name": "job.client_config.connection_pool.size", "type": "integer"} - # job.client_config.connection_pool.size -- gRPC client connection pool size - size: 3 - # @schema {"name": "job.client_config.connection_pool.old_conn_close_duration", "type": "string"} - # job.client_config.connection_pool.old_conn_close_duration -- makes delay before gRPC client connection closing during connection pool rebalance - old_conn_close_duration: "2m" - # @schema {"name": "job.client_config.backoff", "type": "object", "anchor": "backoff"} - backoff: - # @schema {"name": "job.client_config.backoff.initial_duration", "type": "string"} - # job.client_config.backoff.initial_duration -- gRPC client backoff initial duration - initial_duration: 5ms - # @schema {"name": "job.client_config.backoff.backoff_time_limit", "type": "string"} - # job.client_config.backoff.backoff_time_limit -- gRPC client backoff time limit - backoff_time_limit: 5s - # @schema {"name": "job.client_config.backoff.maximum_duration", "type": "string"} - # job.client_config.backoff.maximum_duration -- gRPC client backoff maximum duration - maximum_duration: 5s - # @schema {"name": "job.client_config.backoff.jitter_limit", "type": "string"} - # job.client_config.backoff.jitter_limit -- gRPC client backoff jitter limit - jitter_limit: 100ms - # @schema {"name": "job.client_config.backoff.backoff_factor", "type": "number"} - # job.client_config.backoff.backoff_factor -- gRPC client backoff factor - backoff_factor: 1.1 - # @schema {"name": "job.client_config.backoff.retry_count", "type": "integer"} - # job.client_config.backoff.retry_count -- gRPC client backoff retry count - retry_count: 100 - # @schema {"name": "job.client_config.backoff.enable_error_log", "type": "boolean"} - # job.client_config.backoff.enable_error_log -- gRPC client backoff log enabled - enable_error_log: true - # @schema {"name": "job.client_config.circuit_breaker", "type": "object"} - circuit_breaker: - # @schema {"name": "job.client_config.circuit_breaker.closed_error_rate", "type": "number"} - # job.client_config.circuit_breaker.closed_error_rate -- gRPC client circuitbreaker closed error rate - closed_error_rate: 0.7 - # @schema {"name": "job.client_config.circuit_breaker.half_open_error_rate", "type": "number"} - # job.client_config.circuit_breaker.half_open_error_rate -- gRPC client circuitbreaker half-open error rate - half_open_error_rate: 0.5 - # @schema {"name": "job.client_config.circuit_breaker.min_samples", "type": "integer"} - # job.client_config.circuit_breaker.min_samples -- gRPC client circuitbreaker minimum sampling count - min_samples: 1000 - # @schema {"name": "job.client_config.circuit_breaker.open_timeout", "type": "string"} - # job.client_config.circuit_breaker.open_timeout -- gRPC client circuitbreaker open timeout - open_timeout: "1s" - # @schema {"name": "job.client_config.circuit_breaker.closed_refresh_timeout", "type": "string"} - # job.client_config.circuit_breaker.closed_refresh_timeout -- gRPC client circuitbreaker closed refresh timeout - closed_refresh_timeout: "10s" - # @schema {"name": "job.client_config.call_option", "type": "object"} - call_option: - # @schema {"name": "job.client_config.wait_for_ready", "type": "boolean"} - # job.client_config.call_option.wait_for_ready -- gRPC client call option wait for ready - wait_for_ready: true - # @schema {"name": "job.client_config.max_retry_rpc_buffer_size", "type": "integer"} - # job.client_config.call_option.max_retry_rpc_buffer_size -- gRPC client call option max retry rpc buffer size - max_retry_rpc_buffer_size: 0 - # @schema {"name": "job.client_config.max_recv_msg_size", "type": "integer"} - # job.client_config.call_option.max_recv_msg_size -- gRPC client call option max receive message size - max_recv_msg_size: 0 - # @schema {"name": "job.client_config.max_send_msg_size", "type": "integer"} - # job.client_config.call_option.max_send_msg_size -- gRPC client call option max send message size - max_send_msg_size: 0 - # @schema {"name": "job.client_config.dial_option", "type": "object"} - dial_option: - # @schema {"name": "job.client_config.dial_option.write_buffer_size", "type": "integer"} - # job.client_config.dial_option.write_buffer_size -- gRPC client dial option write buffer size - write_buffer_size: 0 - # @schema {"name": "job.client_config.dial_option.read_buffer_size", "type": "integer"} - # job.client_config.dial_option.read_buffer_size -- gRPC client dial option read buffer size - read_buffer_size: 0 - # @schema {"name": "job.client_config.dial_option.initial_window_size", "type": "integer"} - # job.client_config.dial_option.initial_window_size -- gRPC client dial option initial window size - initial_window_size: 0 - # @schema {"name": "job.client_config.dial_option.initial_connection_window_size", "type": "integer"} - # job.client_config.dial_option.initial_connection_window_size -- gRPC client dial option initial connection window size - initial_connection_window_size: 0 - # @schema {"name": "job.client_config.dial_option.max_msg_size", "type": "integer"} - # job.client_config.dial_option.max_msg_size -- gRPC client dial option max message size - max_msg_size: 0 - # @schema {"name": "job.client_config.dial_option.backoff_max_delay", "type": "string"} - # job.client_config.dial_option.backoff_max_delay -- gRPC client dial option max backoff delay - backoff_max_delay: "120s" - # @schema {"name": "job.client_config.dial_option.backoff_base_delay", "type": "string"} - # job.client_config.dial_option.backoff_base_delay -- gRPC client dial option base backoff delay - backoff_base_delay: "1s" - # @schema {"name": "job.client_config.dial_option.backoff_multiplier", "type": "number"} - # job.client_config.dial_option.backoff_multiplier -- gRPC client dial option base backoff delay - backoff_multiplier: 1.6 - # @schema {"name": "job.client_config.dial_option.backoff_jitter", "type": "number"} - # job.client_config.dial_option.backoff_jitter -- gRPC client dial option base backoff delay - backoff_jitter: 0.2 - # @schema {"name": "job.client_config.dial_option.min_connection_timeout", "type": "string"} - # job.client_config.dial_option.min_connection_timeout -- gRPC client dial option minimum connection timeout - min_connection_timeout: "20s" - # @schema {"name": "job.client_config.dial_option.enable_backoff", "type": "boolean"} - # job.client_config.dial_option.enable_backoff -- gRPC client dial option backoff enabled - enable_backoff: false - # @schema {"name": "job.client_config.dial_option.insecure", "type": "boolean"} - # job.client_config.dial_option.insecure -- gRPC client dial option insecure enabled - insecure: true - # @schema {"name": "job.client_config.dial_option.timeout", "type": "string"} - # job.client_config.dial_option.timeout -- gRPC client dial option timeout - timeout: "" - # @schema {"name": "job.client_config.dial_option.interceptors", "type": "array", "items": {"type": "string", "enum": ["TraceInterceptor"]}} - # job.client_config.dial_option.interceptors -- gRPC client interceptors - interceptors: [] - # @schema {"name": "job.client_config.dial_option.net", "type": "object", "anchor": "net"} - net: - # @schema {"name": "job.client_config.dial_option.net.dns", "type": "object"} - dns: - # @schema {"name": "job.client_config.dial_option.net.dns.cache_enabled", "type": "boolean"} - # job.client_config.dial_option.net.dns.cache_enabled -- gRPC client TCP DNS cache enabled - cache_enabled: true - # @schema {"name": "job.client_config.dial_option.net.dns.refresh_duration", "type": "string"} - # job.client_config.dial_option.net.dns.refresh_duration -- gRPC client TCP DNS cache refresh duration - refresh_duration: 30m - # @schema {"name": "job.client_config.dial_option.net.dns.cache_expiration", "type": "string"} - # job.client_config.dial_option.net.dns.cache_expiration -- gRPC client TCP DNS cache expiration - cache_expiration: 1h - # @schema {"name": "job.client_config.dial_option.net.dialer", "type": "object"} - dialer: - # @schema {"name": "job.client_config.dial_option.net.dialer.timeout", "type": "string"} - # job.client_config.dial_option.net.dialer.timeout -- gRPC client TCP dialer timeout - timeout: "" - # @schema {"name": "job.client_config.dial_option.net.dialer.keepalive", "type": "string"} - # job.client_config.dial_option.net.dialer.keepalive -- gRPC client TCP dialer keep alive - keepalive: "" - # @schema {"name": "job.client_config.dial_option.net.dialer.dual_stack_enabled", "type": "boolean"} - # job.client_config.dial_option.net.dialer.dual_stack_enabled -- gRPC client TCP dialer dual stack enabled - dual_stack_enabled: true - # @schema {"name": "job.client_config.dial_option.net.socket_option", "type": "object"} - socket_option: - # @schema {"name": "job.client_config.dial_option.net.socket_option.reuse_port", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.reuse_port -- server listen socket option for reuse_port functionality - reuse_port: true - # @schema {"name": "job.client_config.dial_option.net.socket_option.reuse_addr", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.reuse_addr -- server listen socket option for reuse_addr functionality - reuse_addr: true - # @schema {"name": "job.client_config.dial_option.net.socket_option.tcp_fast_open", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.tcp_fast_open -- server listen socket option for tcp_fast_open functionality - tcp_fast_open: true - # @schema {"name": "job.client_config.dial_option.net.socket_option.tcp_no_delay", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.tcp_no_delay -- server listen socket option for tcp_no_delay functionality - tcp_no_delay: true - # @schema {"name": "job.client_config.dial_option.net.socket_option.tcp_cork", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.tcp_cork -- server listen socket option for tcp_cork functionality - tcp_cork: false - # @schema {"name": "job.client_config.dial_option.net.socket_option.tcp_quick_ack", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.tcp_quick_ack -- server listen socket option for tcp_quick_ack functionality - tcp_quick_ack: true - # @schema {"name": "job.client_config.dial_option.net.socket_option.tcp_defer_accept", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.tcp_defer_accept -- server listen socket option for tcp_defer_accept functionality - tcp_defer_accept: true - # @schema {"name": "job.client_config.dial_option.net.socket_option.ip_transparent", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.ip_transparent -- server listen socket option for ip_transparent functionality - ip_transparent: false - # @schema {"name": "job.client_config.dial_option.net.socket_option.ip_recover_destination_addr", "type": "boolean"} - # job.client_config.dial_option.net.socket_option.ip_recover_destination_addr -- server listen socket option for ip_recover_destination_addr functionality - ip_recover_destination_addr: false - # @schema {"name": "job.client_config.dial_option.keepalive", "type": "object"} - keepalive: - # @schema {"name": "job.client_config.dial_option.keepalive.time", "type": "string"} - # job.client_config.dial_option.keepalive.time -- gRPC client keep alive time - time: "120s" - # @schema {"name": "job.client_config.dial_option.keepalive.timeout", "type": "string"} - # job.client_config.dial_option.keepalive.timeout -- gRPC client keep alive timeout - timeout: "30s" - # @schema {"name": "job.client_config.dial_option.keepalive.permit_without_stream", "type": "boolean"} - # job.client_config.dial_option.keepalive.permit_without_stream -- gRPC client keep alive permit without stream - permit_without_stream: true -# @schema {"name": "resources", "type": "object"} -# resources -- kubernetes resources of pod -resources: - # @schema {"name": "resources.limits", "type": "object"} - limits: - cpu: 300m - memory: 300Mi - # @schema {"name": "resources.requests", "type": "object"} - requests: - cpu: 200m - memory: 200Mi -# @schema {"name": "logging", "type": "object"} -logging: - # @schema {"name": "logging.logger", "type": "string", "enum": ["glg", "zap"]} - # logging.logger -- logger name. - logger: glg - # @schema {"name": "logging.level", "type": "string", "enum": ["debug", "info", "warn", "error", "fatal"]} - # logging.level -- logging level. - level: debug - # @schema {"name": "logging.format", "type": "string", "enum": ["raw", "json"]} - # logging.format -- logging format. - format: raw -``` - -
- -For more details of the configuration of `vald-benchmark-operator-release`, please refer to [here](https://github.com/vdaas/vald/blob/main/charts/vald-benchmark-operator/values.yaml) - -1. Add Vald repo into the helm repo - - ```bash - helm repo add vald https://vdaas.vald.org - ``` - -1. Deploy `vald-benchmark-operator-release` - - ```bash - helm install vald-benchmark-operator-release vald/vald-benchmark-operator - ``` - -1. Apply `vbor.yaml` (optional) - - ```bash - kubectl apply -f vbor.yaml - ``` - -## Running Continuous Benchmarks - -After deploy the benchmark operator, you can execute continuous benchmark by applying `ValdBenchmarkScenarioRelease` or `ValdBenchmarkJobRelease`. - -Please configure designed benchmark and apply by `kubectl` command. - -The sample manifests are [here](https://github.com/vdaas/vald/tree/main/example/helm/benchmark). - -## Monitoring Benchmark Job Metrics - -Metrics monitoring can be set in the same way as Vald cluster. -For information on building a monitoring environment, please refer to [Observability Configuration](/docs/v1.7/user-guides/observability-configuration). - -To monitor metrics about continuous benchmarking, please edit `ValdBenchmarkOperatorRelease` as follows: - -```yaml ---- -# @schema {"name": "observability", "type": "object"} -observability: - # @schema {"name": "observability.enabled", "type": "boolean"} - enabled: true - # @schema {"name": "observability.otlp", "type": "object"} - otlp: - # @schema {"name": "observability.otlp.collector_endpoint", "type": "string"} - # Please confirm correct collector_endpoint - collector_endpoint: "opentelemetry-collector-collector.default.svc.cluster.local:4317" - trace: - # @schema {"name": "observability.trace.enabled", "type": "boolean"} - enabled: true -``` - -After apply it, the metrics can be shown on the Grafana dashboard like as below. - -Grafana dashboard showing benchmark metrics diff --git a/content/docs/v1.7/performance/tuning-search-performance.md b/content/docs/v1.7/performance/tuning-search-performance.md deleted file mode 100644 index edd39f71a..000000000 --- a/content/docs/v1.7/performance/tuning-search-performance.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: "Tuning Search Performance_v1.7/Performance" -date: 2024-02-15T17:12:28+09:00 -draft: false -weight: 0 -description: "" -menu: - performance: - parent: Performance ---- - -# Tuning Search Performance - -ANN is fast, but sometimes it can be improved more by tuning parameters. -This page shows the essence for improve ANN search by the Vald cluster. - -## Tuning Guideline - -When the search results do NOT satisfy the expected result, it can be improved by tuning parameters. - -First of all, we recommend tuning by following the steps below without doing it blindly. - -```mermaid -flowchart TD - A[Perform Linear Search API] - B{Is satisfies?} - C[Tuning Parameters to improve precision] - D[Tuning Embedding Models] - E[Perform Search API] - F[Tuning Parameters to improve latency] - A-->B - B-- Yes -->C - B-- No -->D - C--> E - E--> C - E--> F - F--> E -``` - -The best practice is: - -1. Measure [Linear Search](/docs/v1.7/docs/api/search#linearsearch-rpc) performance and use it as a baseline for Search API -1. Repeat tuning to improve precision and measure Search API until the conditions are met -1. Repeat tuning to improve latency and measure Search API until the conditions are met - -
-When the results are not good by Linear Search API, it may need to rethink the embedding model for vectorization. -
- -## Tuning parameters - -There are two viewpoints, client-side and cluster-side, for improving search performance. - -
-There is a trade-off between search speed and accuracy, so tuning accuracy at first is recommended. -
- -### Client side - -On the client side, parameters of [`Search.Config`](/docs/v1.7/docs/api/search#input) will affect the search result. - -| | description | how does it affect? | memo | -| :---------- | :---------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------ | -| radius | the search radius for NGT
[ref: search_radius](https://github.com/yahoojapan/NGT/tree/main/bin/ngt#search) | Define the search range when NGT searches the nearest neighbors | recommended value: `-1` | -| epsilon | the search coefficient for NGT
[ref: search_range_coefficient](https://github.com/yahoojapan/NGT/tree/main/bin/ngt#search) | Expansion factor of the NGT search range.
Search operation time increases when the epsilon is big. | recommended value range: `0.01 ~ 0.3`
default value: `0.1` | -| timeout(ns) | max time duration until receiving search results. | An error will be returned if the set `num` search results cannot be obtained within the set time.
By setting `min_num`, the search results will be returned if more than `min_num` can be searched within the time. | default value: `3,000,000,000ns` | - -### Cluster-side - -On the cluster side, these parameters can be set by `values.yaml`, affect the search result. - -| | description | how does it affect? | Memo | -| :--------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------ | -| agent.ngt.creation_edge_size | Number of nodes connected to one node
[ref: no_of_edges](https://github.com/yahoojapan/NGT/tree/main/bin/ngt#create) | It helps reduce unreachable edges.
The larger it is, the denser the graph structure will be, but the memory usage, search speed, and index construction time will increase accordingly. | default value: `20` | -| agent.ngt.search_edge_size | Number of nodes to search from the origin node when searching
[ref: no_of_edges_at_search_at](https://github.com/yahoojapan/NGT/tree/main/bin/ngt#create) | The number of nodes to search will increase.
Accuracy will be higher, but speed will be lower.
Adjust if adjusting the radius and epsilon does not improve the situation. | default value: `10` | diff --git a/content/docs/v1.7/release/changelog.md b/content/docs/v1.7/release/changelog.md index 74a9094e3..829eae358 100644 --- a/content/docs/v1.7/release/changelog.md +++ b/content/docs/v1.7/release/changelog.md @@ -1,6 +1,6 @@ --- title: "Changelog_v1.7/Release" -date: 2024-10-16T16:05:59+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 0 description: "" @@ -11,526 +11,6 @@ menu: # CHANGELOG -## v1.7.14 - -### Docker images - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
componentDocker pull
Agent NGT - docker pull vdaas/vald-agent-ngt:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-agent-ngt:v1.7.14 -
Agent sidecar - docker pull vdaas/vald-agent-sidecar:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-agent-sidecar:v1.7.14 -
Discoverers - docker pull vdaas/vald-discoverer-k8s:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-discoverer-k8s:v1.7.14 -
Gateways - docker pull vdaas/vald-lb-gateway:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-lb-gateway:v1.7.14
- docker pull vdaas/vald-filter-gateway:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-filter-gateway:v1.7.14 -
Index Manager - docker pull vdaas/vald-manager-index:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-manager-index:v1.7.14 -
Helm Operator - docker pull vdaas/vald-helm-operator:v1.7.14
- docker pull ghcr.io/vdaas/vald/vald-helm-operator:v1.7.14 -
- -### Documents - -- [GoDoc](https://pkg.go.dev/github.com/vdaas/vald@v1.7.14) -- [Helm Chart Reference](https://github.com/vdaas/vald/blob/v1.7.14/charts/vald/README.md) -- [Helm Operator Chart Reference](https://github.com/vdaas/vald/blob/v1.7.14/charts/vald-helm-operator/README.md) - -### Changes - -### :sparkles: New feature - -- Add String sorted topologicalSort [#2696](https://github.com/vdaas/vald/pull/2696) [#2698](https://github.com/vdaas/vald/pull/2698) -- Add CPU_INFO_FLAGS for Apple Silicon [#2694](https://github.com/vdaas/vald/pull/2694) [#2697](https://github.com/vdaas/vald/pull/2697) -- Add New gRPC Options and Add Reconnect Logic for connection Pool [#2685](https://github.com/vdaas/vald/pull/2685) [#2693](https://github.com/vdaas/vald/pull/2693) -- Add option to disable dns resolve [#2634](https://github.com/vdaas/vald/pull/2634) [#2641](https://github.com/vdaas/vald/pull/2641) -- Backport PR #2584 to release/v1.7 for Implement ngt property get API [#2588](https://github.com/vdaas/vald/pull/2588) -- add HTTP2 support for http.Client and Vald HTTP Server [#2572](https://github.com/vdaas/vald/pull/2572) [#2575](https://github.com/vdaas/vald/pull/2575) - -### :zap: Improve performance - -- Refactor grpc/status.withDetails function for performance [#2664](https://github.com/vdaas/vald/pull/2664) [#2668](https://github.com/vdaas/vald/pull/2668) - -### :recycle: Refactor - -- Refactor use Absolute path for Makefile [#2673](https://github.com/vdaas/vald/pull/2673) -- Refactor internal/net/grpc/client.go [#2675](https://github.com/vdaas/vald/pull/2675) -- modify ParseError to FromError for agent handler [#2667](https://github.com/vdaas/vald/pull/2667) [#2679](https://github.com/vdaas/vald/pull/2679) -- Backport PR #2674 to release/v1.7 for Refactor internal/net/grpc/client.go [#2675](https://github.com/vdaas/vald/pull/2675) -- Backport PR #2670 to release/v1.7 for Refactor use Absolute path for Makefile [#2673](https://github.com/vdaas/vald/pull/2673) -- Refactor grpc/status.withDetails function for performance [#2664](https://github.com/vdaas/vald/pull/2664) [#2668](https://github.com/vdaas/vald/pull/2668) -- Refactor for release v1.7.14 [#2639](https://github.com/vdaas/vald/pull/2639) [#2648](https://github.com/vdaas/vald/pull/2648) -- refactor(gateway): delete unused file [#2644](https://github.com/vdaas/vald/pull/2644) [#2646](https://github.com/vdaas/vald/pull/2646) -- Refactor test checkFunc condition [#2599](https://github.com/vdaas/vald/pull/2599) [#2602](https://github.com/vdaas/vald/pull/2602) -- Backport PR #2586 to release/v1.7 for modify rust package structure [#2590](https://github.com/vdaas/vald/pull/2590) -- Backport PR #2577 to release/v1.7 for refactor docker and change buildkit-syft-scanner reference to ghcr.io [#2578](https://github.com/vdaas/vald/pull/2578) - -### :bug: Bugfix - -- Fix gRPC error handling for gateway/filter handler [#2669](https://github.com/vdaas/vald/pull/2669) [#2689](https://github.com/vdaas/vald/pull/2689) -- fix: increase limit [#2683](https://github.com/vdaas/vald/pull/2683) [#2686](https://github.com/vdaas/vald/pull/2686) -- Fix gRPC error handling for mirror-gateway handler [#2665](https://github.com/vdaas/vald/pull/2665) [#2681](https://github.com/vdaas/vald/pull/2681) -- Fix gRPC error msg handling for lb-gateway handler [#2663](https://github.com/vdaas/vald/pull/2663) [#2682](https://github.com/vdaas/vald/pull/2682) -- Bugfix ingress route settings [#2636](https://github.com/vdaas/vald/pull/2636) [#2642](https://github.com/vdaas/vald/pull/2642) -- Fix broken links in the document files [#2611](https://github.com/vdaas/vald/pull/2611) [#2614](https://github.com/vdaas/vald/pull/2614) -- Fix: make command name [#2610](https://github.com/vdaas/vald/pull/2610) [#2612](https://github.com/vdaas/vald/pull/2612) -- Bugfix NGT flush logic [#2598](https://github.com/vdaas/vald/pull/2598) [#2606](https://github.com/vdaas/vald/pull/2606) - -### :pencil2: Document - -- Fix broken links in the document files [#2611](https://github.com/vdaas/vald/pull/2611) [#2614](https://github.com/vdaas/vald/pull/2614) - -### :white_check_mark: Testing - -- Refactor test checkFunc condition [#2599](https://github.com/vdaas/vald/pull/2599) [#2602](https://github.com/vdaas/vald/pull/2602) - -### :green_heart: CI - -- Buf CLI migrate to v2 [#2691](https://github.com/vdaas/vald/pull/2691) [#2695](https://github.com/vdaas/vald/pull/2695) -- [create-pull-request] automated change [#2677](https://github.com/vdaas/vald/pull/2677) [#2678](https://github.com/vdaas/vald/pull/2678) -- automatically generate workflows [#2595](https://github.com/vdaas/vald/pull/2595) [#2603](https://github.com/vdaas/vald/pull/2603) - -### :chart_with_upwards_trend: Metrics/Tracing - -- Introduce an observability crate using opentelemetry-rust [#2535](https://github.com/vdaas/vald/pull/2535) [#2609](https://github.com/vdaas/vald/pull/2609) - - - -## Summary by CodeRabbit - -- **New Features** - - - Added several new contributors to the project, enhancing community involvement. - - Introduced a new configuration file for spell checking, improving documentation quality. - - Expanded the project with new configuration files, documentation, and source code for enhanced functionality. - -- **Bug Fixes** - - - Updated version information in issue templates for accuracy. - -- **Documentation** - - - Improved clarity in the pull request template and updated version information. - -- **Chores** - - Modified GitHub Actions for better handling of Docker image tags. - - - -## v1.7.13 - -### Docker images - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
componentDocker pull
Agent NGT - docker pull vdaas/vald-agent-ngt:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-agent-ngt:v1.7.13 -
Agent sidecar - docker pull vdaas/vald-agent-sidecar:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-agent-sidecar:v1.7.13 -
Discoverers - docker pull vdaas/vald-discoverer-k8s:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-discoverer-k8s:v1.7.13 -
Gateways - docker pull vdaas/vald-lb-gateway:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-lb-gateway:v1.7.13
- docker pull vdaas/vald-filter-gateway:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-filter-gateway:v1.7.13 -
Index Manager - docker pull vdaas/vald-manager-index:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-manager-index:v1.7.13 -
Helm Operator - docker pull vdaas/vald-helm-operator:v1.7.13
- docker pull ghcr.io/vdaas/vald/vald-helm-operator:v1.7.13 -
- -### Documents - -- [GoDoc](https://pkg.go.dev/github.com/vdaas/vald@v1.7.13) -- [Helm Chart Reference](https://github.com/vdaas/vald/blob/v1.7.13/charts/vald/README.md) -- [Helm Operator Chart Reference](https://github.com/vdaas/vald/blob/v1.7.13/charts/vald-helm-operator/README.md) - -### Changes - -### [Bugfix] - -- **General Fixes** - - 1. Fix index correction process [#2565](https://github.com/vdaas/vald/pull/2565) ([#2566](https://github.com/vdaas/vald/pull/2566)) - 2. libquadmath is not available on ARM [#2559](https://github.com/vdaas/vald/pull/2559) - 3. fix: add checkout option [#2545](https://github.com/vdaas/vald/pull/2545) ([#2546](https://github.com/vdaas/vald/pull/2546)) - 4. fix: make format [#2534](https://github.com/vdaas/vald/pull/2534) ([#2540](https://github.com/vdaas/vald/pull/2540)) - 5. fix conflict bug [#2537](https://github.com/vdaas/vald/pull/2537) - 6. Bugfix that caused an error when argument has 3 or more nil arguments [#2517](https://github.com/vdaas/vald/pull/2517) ([#2520](https://github.com/vdaas/vald/pull/2520)) - 7. Bugfix recreate benchmark job when operator reboot [#2463](https://github.com/vdaas/vald/pull/2463) ([#2464](https://github.com/vdaas/vald/pull/2464)) - 8. Fix agent-faiss build failed [#2418](https://github.com/vdaas/vald/pull/2418) ([#2419](https://github.com/vdaas/vald/pull/2419)) - 9. Fix the logic to determine docker image [#2410](https://github.com/vdaas/vald/pull/2410) ([#2420](https://github.com/vdaas/vald/pull/2420)) - -- **Backport and Release-Related** - - 1. Fix workflow trigger for backport pr creation [#2471](https://github.com/vdaas/vald/pull/2471) ([#2472](https://github.com/vdaas/vald/pull/2472)) - 2. Fix output settings to determine-docker-image-tag action and release branch build tag name [#2423](https://github.com/vdaas/vald/pull/2423) ([#2425](https://github.com/vdaas/vald/pull/2425)) - -- **E2E and Index** - - 1. Fix e2e for read replica and add e2e for index operator [#2455](https://github.com/vdaas/vald/pull/2455) ([#2459](https://github.com/vdaas/vald/pull/2459)) - 2. Fix index job logic to pass DNS A record [#2438](https://github.com/vdaas/vald/pull/2438) ([#2448](https://github.com/vdaas/vald/pull/2448)) - -- **Documentation and Other** - 1. fix: typo of execution rule [#2426](https://github.com/vdaas/vald/pull/2426) ([#2427](https://github.com/vdaas/vald/pull/2427)) - 2. :pencil: Fix typo of file name [#2413](https://github.com/vdaas/vald/pull/2413) ([#2415](https://github.com/vdaas/vald/pull/2415)) - -### [Enhancement] - -- **General Improvements** - - 1. Update dependencies, C++ standard, and improve Dockerfiles for better build systems and localization [#2549](https://github.com/vdaas/vald/pull/2549) ([#2557](https://github.com/vdaas/vald/pull/2557)) - 2. Implement ngt Statistics API [#2539](https://github.com/vdaas/vald/pull/2539) ([#2547](https://github.com/vdaas/vald/pull/2547)) - 3. refactor index manager service add index service API to expose index informations [#2525](https://github.com/vdaas/vald/pull/2525) ([#2532](https://github.com/vdaas/vald/pull/2532)) - -- **API and Logic Changes** - - 1. Change default image tag from latest to nightly [#2516](https://github.com/vdaas/vald/pull/2516) ([#2518](https://github.com/vdaas/vald/pull/2518)) - 2. update large top-K ratio handling logic [#2509](https://github.com/vdaas/vald/pull/2509) ([#2511](https://github.com/vdaas/vald/pull/2511)) - 3. add inner product distance type for ngt [#2454](https://github.com/vdaas/vald/pull/2454) ([#2458](https://github.com/vdaas/vald/pull/2458)) - 4. Implement index operator logic for read replica rotation [#2444](https://github.com/vdaas/vald/pull/2444) ([#2456](https://github.com/vdaas/vald/pull/2456)) - -- **Performance and Optimization** - - 1. update deps & add validation for Flush API when agent is Read Only [#2433](https://github.com/vdaas/vald/pull/2433) ([#2436](https://github.com/vdaas/vald/pull/2436)) - 2. Add `index-operator` template implementation [#2375](https://github.com/vdaas/vald/pull/2375) ([#2424](https://github.com/vdaas/vald/pull/2424)) - -- **Testing and Metrics** - 1. Implement client metrics interceptor for continuous benchmark job [#2477](https://github.com/vdaas/vald/pull/2477) ([#2480](https://github.com/vdaas/vald/pull/2480)) - 2. Add tests for index information export [#2412](https://github.com/vdaas/vald/pull/2412) ([#2414](https://github.com/vdaas/vald/pull/2414)) - -### [CI] - -1. [create-pull-request] automated change [#2552](https://github.com/vdaas/vald/pull/2552) ([#2556](https://github.com/vdaas/vald/pull/2556)) -2. Add workflow to check git conflict for backport PR [#2548](https://github.com/vdaas/vald/pull/2548) ([#2550](https://github.com/vdaas/vald/pull/2550)) -3. [CI] Add workflow to synchronize ubuntu base image [#2526](https://github.com/vdaas/vald/pull/2526) ([#2527](https://github.com/vdaas/vald/pull/2527)) -4. Automatically add backport main label for release-pr [#2473](https://github.com/vdaas/vald/pull/2473) ([#2475](https://github.com/vdaas/vald/pull/2475)) -5. change external docker image reference to ghcr.io registry [#2567](https://github.com/vdaas/vald/pull/2567) ([#2568](https://github.com/vdaas/vald/pull/2568)) - -### [Backport] - -1. Backport PR #2542, #2538 to release/v1.7 [#2543](https://github.com/vdaas/vald/pull/2543) -2. Backport docs updates to release/v1.7 [#2521](https://github.com/vdaas/vald/pull/2521) -3. Backport Flush API [#2434](https://github.com/vdaas/vald/pull/2434) - -### [Documentation] - -1. capitalize faq [#2512](https://github.com/vdaas/vald/pull/2512) ([#2522](https://github.com/vdaas/vald/pull/2522)) -2. add faiss in values.yaml & valdrelease.yaml [#2514](https://github.com/vdaas/vald/pull/2514) ([#2519](https://github.com/vdaas/vald/pull/2519)) -3. add read replica and rotator docs [#2497](https://github.com/vdaas/vald/pull/2497) ([#2499](https://github.com/vdaas/vald/pull/2499)) -4. Update continuous benchmark docs [#2485](https://github.com/vdaas/vald/pull/2485) ([#2486](https://github.com/vdaas/vald/pull/2486)) -5. docs: add hrichiksite as a contributor for doc [#2441](https://github.com/vdaas/vald/pull/2441) ([#2442](https://github.com/vdaas/vald/pull/2442)) - -### [Other] - -1. Add base of benchmark operator dashboard [#2430](https://github.com/vdaas/vald/pull/2430) ([#2453](https://github.com/vdaas/vald/pull/2453)) -2. Add client metrics panels for continuous benchmark job [#2481](https://github.com/vdaas/vald/pull/2481) ([#2483](https://github.com/vdaas/vald/pull/2483)) -3. Add unit tests for index operator [#2460](https://github.com/vdaas/vald/pull/2460) ([#2461](https://github.com/vdaas/vald/pull/2461)) -4. add reviewer guideline [#2507](https://github.com/vdaas/vald/pull/2507) ([#2508](https://github.com/vdaas/vald/pull/2508)) -5. Sync release/v1.7 to main [#2495](https://github.com/vdaas/vald/pull/2495) -6. Add snapshot timestamp annotations to read replica agent [#2428](https://github.com/vdaas/vald/pull/2428) ([#2443](https://github.com/vdaas/vald/pull/2443)) -7. Update build rule for nightly image [#2421](https://github.com/vdaas/vald/pull/2421) ([#2422](https://github.com/vdaas/vald/pull/2422)) - -## v1.7.12 - -### Docker images - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
componentDocker pull
Agent NGT - docker pull vdaas/vald-agent-ngt:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-agent-ngt:v1.7.12 -
Agent sidecar - docker pull vdaas/vald-agent-sidecar:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-agent-sidecar:v1.7.12 -
Discoverers - docker pull vdaas/vald-discoverer-k8s:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-discoverer-k8s:v1.7.12 -
Gateways - docker pull vdaas/vald-lb-gateway:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-lb-gateway:v1.7.12
- docker pull vdaas/vald-filter-gateway:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-filter-gateway:v1.7.12 -
Index Manager - docker pull vdaas/vald-manager-index:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-manager-index:v1.7.12 -
Helm Operator - docker pull vdaas/vald-helm-operator:v1.7.12
- docker pull ghcr.io/vdaas/vald/vald-helm-operator:v1.7.12 -
- -### Documents - -- [GoDoc](https://pkg.go.dev/github.com/vdaas/vald@v1.7.12) -- [Helm Chart Reference](https://github.com/vdaas/vald/blob/v1.7.12/charts/vald/README.md) -- [Helm Operator Chart Reference](https://github.com/vdaas/vald/blob/v1.7.12/charts/vald-helm-operator/README.md) - -### Changes - -:sparkles: New feature - -- Add rotate-all option to rotator [#2305](https://github.com/vdaas/vald/pull/2305) [#2393](https://github.com/vdaas/vald/pull/2393) -- Make agent export index metrics to Pod k8s resource [#2319](https://github.com/vdaas/vald/pull/2319) [#2372](https://github.com/vdaas/vald/pull/2372) - -:recycle: Refactor - -- Delete unnecessary code for mirror [#2366](https://github.com/vdaas/vald/pull/2366) [#2391](https://github.com/vdaas/vald/pull/2391) - -:bug: Bugfix - -- Resolve kvs already closed before last saving [#2390](https://github.com/vdaas/vald/pull/2390) [#2394](https://github.com/vdaas/vald/pull/2394) - -:pencil2: Document - -- Create continous benchmark doc [#2352](https://github.com/vdaas/vald/pull/2352) [#2395](https://github.com/vdaas/vald/pull/2395) - -:white_check_mark: Testing - -- Fix: build error of internal kvs test [#2396](https://github.com/vdaas/vald/pull/2396) [#2398](https://github.com/vdaas/vald/pull/2398) - -:green_heart: CI - -- Fix: disable protobuf dispatch for client [#2401](https://github.com/vdaas/vald/pull/2401) [#2403](https://github.com/vdaas/vald/pull/2403) -- Add Con-Bench helm chart to the Vald charts [#2388](https://github.com/vdaas/vald/pull/2388) [#2389](https://github.com/vdaas/vald/pull/2389) -- Update workflow to release readreplica chart [#2383](https://github.com/vdaas/vald/pull/2383) [#2387](https://github.com/vdaas/vald/pull/2387) -- Backport ci deps others [#2386](https://github.com/vdaas/vald/pull/2386) -- Update docker build target platform selection rules [#2370](https://github.com/vdaas/vald/pull/2370) [#2374](https://github.com/vdaas/vald/pull/2374) -- Add commit hash build image [#2359](https://github.com/vdaas/vald/pull/2359) [#2371](https://github.com/vdaas/vald/pull/2371) -- Refactor code using golangci-lint [#2362](https://github.com/vdaas/vald/pull/2362) [#2365](https://github.com/vdaas/vald/pull/2365) -- Change docker scan timeout longer [#2363](https://github.com/vdaas/vald/pull/2363) [#2364](https://github.com/vdaas/vald/pull/2364) - -:arrow_up: Update dependencies - -- Update deps [#2404](https://github.com/vdaas/vald/pull/2404) [#2405](https://github.com/vdaas/vald/pull/2405) - -:lock: Security - -- Create SECURITY.md [#2367](https://github.com/vdaas/vald/pull/2367) [#2368](https://github.com/vdaas/vald/pull/2368) - -:art: Design - -- Change JP logo to EN logo [#2369](https://github.com/vdaas/vald/pull/2369) [#2392](https://github.com/vdaas/vald/pull/2392) - -## v1.7.11 - -### Docker images - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
componentDocker pull
Agent NGT - docker pull vdaas/vald-agent-ngt:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-agent-ngt:v1.7.11 -
Agent sidecar - docker pull vdaas/vald-agent-sidecar:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-agent-sidecar:v1.7.11 -
Discoverers - docker pull vdaas/vald-discoverer-k8s:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-discoverer-k8s:v1.7.11 -
Gateways - docker pull vdaas/vald-lb-gateway:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-lb-gateway:v1.7.11
- docker pull vdaas/vald-filter-gateway:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-filter-gateway:v1.7.11 -
Index Manager - docker pull vdaas/vald-manager-index:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-manager-index:v1.7.11 -
Helm Operator - docker pull vdaas/vald-helm-operator:v1.7.11
- docker pull ghcr.io/vdaas/vald/vald-helm-operator:v1.7.11 -
- -### Documents - -- [GoDoc](https://pkg.go.dev/github.com/vdaas/vald@v1.7.11) -- [Helm Chart Reference](https://github.com/vdaas/vald/blob/v1.7.11/charts/vald/README.md) -- [Helm Operator Chart Reference](https://github.com/vdaas/vald/blob/v1.7.11/charts/vald-helm-operator/README.md) - -### Changes - -:sparkles: New feature - -- Add HPA for read replica [#2307](https://github.com/vdaas/vald/pull/2307) -- Add internal KVS pogreb package [#2302](https://github.com/vdaas/vald/pull/2302) -- Two version deploy support [#2171](https://github.com/vdaas/vald/pull/2171) -- Add mirror gateway definitions and Implementations [#2262](https://github.com/vdaas/vald/pull/2262) -- Initialize dev env for Rust agent [#2293](https://github.com/vdaas/vald/pull/2293) -- Add new grafana dashboard for agent memory metrics [#2279](https://github.com/vdaas/vald/pull/2279) -- Implement continuous benchmark tool [#2216](https://github.com/vdaas/vald/pull/2216) - -:recycle: Refactor - -- Add newline between params to avoid false formatting [#2347](https://github.com/vdaas/vald/pull/2347) -- Fix golangci-lint config and apply tagalign [#2326](https://github.com/vdaas/vald/pull/2326) -- Refactor postAttachCommand [#2312](https://github.com/vdaas/vald/pull/2312) -- Refactor ignore rule [#2339](https://github.com/vdaas/vald/pull/2339) -- Fix NGT default params [#2332](https://github.com/vdaas/vald/pull/2332) -- Format yaml using google/yamlfmt & update go version and dependencies [#2322](https://github.com/vdaas/vald/pull/2322) -- Refactor update opentelemetry-go & faiss [#2303](https://github.com/vdaas/vald/pull/2303) -- Change discoverer client to broadcast to read replicas [#2276](https://github.com/vdaas/vald/pull/2276) -- Add stern and telepresence [#2320](https://github.com/vdaas/vald/pull/2320) -- Add issue metrics [#2308](https://github.com/vdaas/vald/pull/2308) -- Add dispatch workflow for update contents of vdaas/web repo [#2294](https://github.com/vdaas/vald/pull/2294) -- Fix: add release build for bench and mirror [#2300](https://github.com/vdaas/vald/pull/2300) -- Fix deeepsource errors [#2299](https://github.com/vdaas/vald/pull/2299) -- Add go cache for improvement docker build performance [#2297](https://github.com/vdaas/vald/pull/2297) -- Add detailed log for readreplica rotator [#2281](https://github.com/vdaas/vald/pull/2281) -- Add isSymlink function and test to gen license to avoid for symlink to become normal file. [#2290](https://github.com/vdaas/vald/pull/2290) -- Add owner reference to the resources made by rotator to delete them when read replica resources are deleted [#2287](https://github.com/vdaas/vald/pull/2287) -- Make vald-readreplica values.yaml to symbolic link [#2286](https://github.com/vdaas/vald/pull/2286) -- Separate readreplica chart [#2283](https://github.com/vdaas/vald/pull/2283) -- Happy New Year 2024 [#2284](https://github.com/vdaas/vald/pull/2284) - -:bug: Bugfix - -- Fix: disable arm64 [#2354](https://github.com/vdaas/vald/pull/2354) -- gcc environment for ARM [#2334](https://github.com/vdaas/vald/pull/2334) -- Revert dev Dockerfile to use official devcontainer image [#2335](https://github.com/vdaas/vald/pull/2335) -- Revert docker-image.yaml change [#2336](https://github.com/vdaas/vald/pull/2336) -- Fix release pr workflow [#2333](https://github.com/vdaas/vald/pull/2333) -- Fix e2e regressions [#2327](https://github.com/vdaas/vald/pull/2327) -- Bugfix grpc ip direct connection status check [#2316](https://github.com/vdaas/vald/pull/2316) -- Fix k3d connectivity error [#2317](https://github.com/vdaas/vald/pull/2317) -- Change lincense/gen/main.go to skip shebang [#2313](https://github.com/vdaas/vald/pull/2313) -- Stop using ENV ARCH and add --platform in Dockerfile [#2304](https://github.com/vdaas/vald/pull/2304) -- gRPC pool connection health check for DNS Addr may fail during VIP member disconnection [#2277](https://github.com/vdaas/vald/pull/2277) -- Fix isSymlink function to correctly check for symbolic links [#2292](https://github.com/vdaas/vald/pull/2292) -- Disable disconnection during non-IP-direct connection [#2291](https://github.com/vdaas/vald/pull/2291) -- Fix git add chart directory for release [#2356](https://github.com/vdaas/vald/pull/2356) - -:pencil2: Document - -- Add search optimization document [#2306](https://github.com/vdaas/vald/pull/2306) -- Update capacity planning doc [#2295](https://github.com/vdaas/vald/pull/2295) - -:green_heart: CI - -- Change to dynamically switch CI container image tag [#2310](https://github.com/vdaas/vald/pull/2310) -- Add E2E tests for read replica feature [#2298](https://github.com/vdaas/vald/pull/2298) -- CI, Docker EXTRA_ARGS not working problem [#2278](https://github.com/vdaas/vald/pull/2278) - ## v1.7.10 ### Docker images diff --git a/content/docs/v1.7/support/FAQ.md b/content/docs/v1.7/support/faq.md similarity index 96% rename from content/docs/v1.7/support/FAQ.md rename to content/docs/v1.7/support/faq.md index b6be8d6a6..dde5cc892 100644 --- a/content/docs/v1.7/support/FAQ.md +++ b/content/docs/v1.7/support/faq.md @@ -1,9 +1,9 @@ --- -title: "FAQ_v1.7/Support" -date: 2024-08-08T10:35:17+09:00 +title: "Faq_v1.7/Support" +date: 2024-10-28T06:58:08Z draft: false -weight: 0 -description: "" +weight: 200 +description: The major questions about using Vald menu: support: parent: Support diff --git a/content/docs/v1.7/troubleshooting/client-side.md b/content/docs/v1.7/troubleshooting/client-side.md index 9ef51c5e1..dfe8a435a 100644 --- a/content/docs/v1.7/troubleshooting/client-side.md +++ b/content/docs/v1.7/troubleshooting/client-side.md @@ -1,6 +1,6 @@ --- title: "Client Side_v1.7/Troubleshooting" -date: 2024-10-16T16:05:58+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 100 description: The popular troubleshooting for client side @@ -66,4 +66,4 @@ Please check your CPU information. - [Provisioning Troubleshooting](/docs/v1.7/troubleshooting/provisioning) - [API Status](/docs/v1.7/api/status) -- [FAQ](/docs/support/faq) +- [FAQ](/docs/v1.7/support/faq) diff --git a/content/docs/v1.7/troubleshooting/mirror-gateway.md b/content/docs/v1.7/troubleshooting/mirror-gateway.md deleted file mode 100644 index e0203d2c3..000000000 --- a/content/docs/v1.7/troubleshooting/mirror-gateway.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: "Mirror Gateway_v1.7/Troubleshooting" -date: 2024-02-15T17:12:29+09:00 -draft: false -weight: 0 -description: "" -menu: - troubleshooting: - parent: Troubleshooting ---- - -# Mirror Gateway Troubleshooting - -This page introduces the popular troubleshooting for Mirror Gateway. - -Additionally, if you encounter some errors when using API, the [API status code](/docs/v1.7/api/status) helps you, too. - -## Insert Operation - -Mirror Gateway sends an Update request to its host if some requests are `ALREADY_EXISTS`. - -Therefore, in addition to the [Insert API status code](/docs/v1.7/api/insert#status-code), the [Update API status code](/docs/v1.7/api/update#status-code) may also be returned to the user. - -Here are some common reasons of error. - -| name | common reason | -| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | -| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | -| INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | -| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| ALREADY_EXISTS | Request ID is already inserted. This status code is returned when all target hosts return `ALREADY_EXISTS`. | -| NOT_FOUND | Requested ID is NOT inserted. This is the status code of the Update request. | -| INTERNAL | Target Vald cluster or network route has some critical error. | - -`0 (OK)` is also returned when all target hosts return `OK` or `ALREADY_EXISTS`. - -## Update Operation - -Mirror Gateway sends an Update request to its host if some requests are `NOT_FOUND`. - -Therefore, in addition to the [Update API status code](/docs/v1.7/api/update#status-code), the [Insert API status code](/docs/v1.7/api/insert#status-code) may also be returned to the user. - -Here are some common reasons of error. - -| name | common reason | -| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | -| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | -| INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | -| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all target hosts return `NOT_FOUND`. | -| ALREADY_EXISTS | Request a pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | -| INTERNAL | Target Vald cluster or network route has some critical error. | - -`0 (OK)` is also returned when all target hosts return `OK` or `ALREADY_EXISTS`. - -## Upsert Operation - -The request process may not be completed when the response code is NOT `0 (OK)`. - -Here are some common reasons of error. - -| name | common reason | -| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | -| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | -| INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | -| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| ALREADY_EXISTS | Requested pair of ID and vector is already inserted. This status code is returned when all target hosts return `ALREADY_EXISTS`. | -| INTERNAL | Target Vald cluster or network route has some critical error. | - -`0 (OK)` is also returned when all target hosts return `OK` or `ALREADY_EXISTS`. - -## Remove Operation - -The request process may not be completed when the response code is NOT `0 (OK)`. - -Here are some common reasons of error. - -| name | common reason | -| :---------------- | :--------------------------------------------------------------------------------------------------- | -| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | -| INVALID_ARGUMENT | The Requested vector's ID is empty, or some request payload is invalid. | -| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all target hosts return `NOT_FOUND`. | -| INTERNAL | Target Vald cluster or network route has some critical error. | - -`0 (OK)` is also returned when all target hosts return `OK` or `NOT_FOUND`. diff --git a/content/docs/v1.7/tutorial/get-started-with-faiss-agent.md b/content/docs/v1.7/tutorial/get-started-with-faiss-agent.md deleted file mode 100644 index 79b05fa76..000000000 --- a/content/docs/v1.7/tutorial/get-started-with-faiss-agent.md +++ /dev/null @@ -1,405 +0,0 @@ ---- -title: "Get Started With Faiss Agent_v1.7/Tutorial" -date: 2024-10-16T16:05:59+09:00 -draft: false -weight: 200 -description: Running Vald cluster with faiss Agent on Kubernetes and execute client codes -menu: - tutorial: - parent: Tutorial ---- - -# Get Started - -This tutorial is for those who have already completed [Get Started](/docs/v1.7/tutorial/get-started). -Please refer to Prepare the Kubernetes Cluster and others there. - -## Deploy Vald on Kubernetes Cluster - -This chapter shows how to deploy Vald using Helm and run it on your Kubernetes cluster.
-In this tutorial, you will deploy the basic configuration of Vald that is consisted of vald-agent-faiss, vald-lb-gateway, vald-discoverer, and vald-manager-index.
- -1. Clone the repository - - ```bash - git clone https://github.com/vdaas/vald.git && \ - cd vald - ``` - -1. Confirm which cluster to deploy - - ```bash - kubectl cluster-info - ``` - -1. Edit Configurations - - Set the parameters for connecting to the vald-lb-gateway through Kubernetes ingress from the external network. - Please set these parameters. - - ```bash - vim example/helm/values.yaml - === - ## vald-lb-gateway settings - gateway: - lb: - ... - ingress: - enabled: true - # TODO: Set your ingress host. - host: localhost - # TODO: Set annotations which you have to set for your k8s cluster. - annotations: - ... - ## vald-agent-faiss settings - agent: - algorithm: faiss - image: - repository: vdaas/vald-agent-faiss - tag: latest - faiss: - auto_index_check_duration: 1m - auto_index_duration_limit: 24h - auto_index_length: 10 - auto_save_index_duration: 35m - dimension: 784 - enable_copy_on_write: false - enable_in_memory_mode: true - enable_proactive_gc: true - index_path: "" - initial_delay_max_duration: 3m - load_index_timeout_factor: 1ms - m: 8 # dimension % m == 0, train size >= 2^m(or nlist) * minPointsPerCentroid - max_load_index_timeout: 10m - method_type: "ivfpq" - metric_type: "innerproduct" - min_load_index_timeout: 3m - nbits_per_idx: 8 - nlist: 100 - ... - ``` - - Note:
- If you decided to use port-forward instead of ingress, please set `gateway.lb.ingress.enabled` to `false`. - -1. Deploy Vald using Helm - - Add vald repo into the helm repo. - - ```bash - helm repo add vald https://vald.vdaas.org/charts - ``` - - Deploy vald on your Kubernetes cluster. - - ```bash - helm install vald vald/vald --values example/helm/values.yaml - ``` - -1. Verify - - When finish deploying Vald, you can check the Vald's pods status following command. - - ```bash - kubectl get pods - ``` - -
Example output
- If the deployment is successful, all Vald components should be running. - - ```bash - NAME READY STATUS RESTARTS AGE - vald-agent-faiss-0 1/1 Running 0 7m12s - vald-agent-faiss-1 1/1 Running 0 7m12s - vald-agent-faiss-2 1/1 Running 0 7m12s - vald-agent-faiss-3 1/1 Running 0 7m12s - vald-agent-faiss-4 1/1 Running 0 7m12s - vald-discoverer-7f9f697dbb-q44qh 1/1 Running 0 7m11s - vald-lb-gateway-6b7b9f6948-4z5md 1/1 Running 0 7m12s - vald-lb-gateway-6b7b9f6948-68g94 1/1 Running 0 6m56s - vald-lb-gateway-6b7b9f6948-cvspq 1/1 Running 0 6m56s - vald-manager-index-74c7b5ddd6-jrnlw 1/1 Running 0 7m12s - ``` - -
- - ```bash - kubectl get ingress - ``` - -
Example output
- - ```bash - NAME CLASS HOSTS ADDRESS PORTS AGE - vald-lb-gateway-ingress localhost 192.168.16.2 80 7m43s - ``` - -
- - ```bash - kubectl get svc - ``` - -
Example output
- - ```bash - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - kubernetes ClusterIP 10.43.0.1 443/TCP 9m29s - vald-agent-faiss ClusterIP None 8081/TCP,3001/TCP 8m48s - vald-discoverer ClusterIP None 8081/TCP,3001/TCP 8m48s - vald-manager-index ClusterIP None 8081/TCP,3001/TCP 8m48s - vald-lb-gateway ClusterIP None 8081/TCP,3001/TCP 8m48s - ``` - -
- -## Run Example Code - -In this chapter, you will execute insert, search, and delete vectors to your Vald cluster using the example code.
-The [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) is used as a dataset for indexing and search query. - -The example code is implemented in Go and using [vald-client-go](https://github.com/vdaas/vald-client-go), one of the official Vald client libraries, for requesting to Vald cluster. -Vald provides multiple language client libraries such as Go, Java, Node.js, Python, etc. -If you are interested, please refer to [SDKs](/docs/v1.7/user-guides/sdks).
- -1. Port Forward(option) - - If you do not use Kubernetes Ingress, port-forward is required to make requests from your local environment. - - ```bash - kubectl port-forward deployment/vald-lb-gateway 8081:8081 - ``` - -1. Download dataset - - Download [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) that is used as a dataset for indexing and search query. - - Move to the working directory - - ```bash - cd example/client - ``` - - Download Fashion-MNIST testing dataset - - ```bash - wget http://ann-benchmarks.com/fashion-mnist-784-euclidean.hdf5 - ``` - -1. Run Example - - We use [`example/client/main.go`](https://github.com/vdaas/vald/blob/main/example/client/main.go) to run the example.
- This example will insert and index 400 vectors into the Vald from the Fashion-MNIST dataset via [gRPC](https://grpc.io/). - And then after waiting for indexing, it will request for searching the nearest vector 10 times. - You will get the 10 nearest neighbor vectors for each search query.
- Run example codes by executing the below command. - - ```bash - go run main.go - ``` - -
The detailed explanation of example code is here
- This will execute 6 steps. - - 1. init - - - Import packages -
example code
- - ```go - package main - - import ( - "context" - "encoding/json" - "flag" - "time" - - "github.com/kpango/fuid" - "github.com/kpango/glg" - "github.com/vdaas/vald-client-go/v1/payload" - "github.com/vdaas/vald-client-go/v1/vald" - - "gonum.org/v1/hdf5" - "google.golang.org/grpc" - ) - ``` - -
- - - Set variables - - - The constant number of training datasets and test datasets. -
example code
- - ```go - const ( - insertCount = 400 - testCount = 20 - ) - ``` - -
- - - The variables for configuration. -
example code
- - ```go - const ( - datasetPath string - grpcServerAddr string - indexingWaitSeconds uint - ) - ``` - -
- - - Recognition parameters. -
example code
- - ```go - func init() { - flag.StringVar(&datasetPath, "path", "fashion-mnist-784-euclidean.hdf5", "set dataset path") - flag.StringVar(&grpcServerAddr, "addr", "127.0.0.1:8081", "set gRPC server address") - flag.UintVar(&indexingWaitSeconds, "wait", 60, "set indexing wait seconds") - flag.Parse() - } - ``` - -
- - 1. load - - - Loading from Fashion-MNIST dataset and set id for each vector that is loaded. This step will return the training dataset, test dataset, and ids list of ids when loading is completed with success. -
example code
- - ```go - ids, train, test, err := load(datasetPath) - if err != nil { - glg.Fatal(err) - } - ``` - -
- - 1. Create the gRPC connection and Vald client with gRPC connection. - -
example code
- - ```go - ctx := context.Background() - - conn, err := grpc.NewClient(grpcServerAddr, grpc.WithInsecure()) - if err != nil { - glg.Fatal(err) - } - - client := vald.NewValdClient(conn) - ``` - -
- - 1. Insert and Index - - - Insert and Indexing 400 training datasets to the Vald agent. -
example code
- - ```go - for i := range ids [:insertCount] { - _, err := client.Insert(ctx, &payload.Insert_Request{ - Vector: &payload.Object_Vector{ - Id: ids[i], - Vector: train[i], - }, - Config: &payload.Insert_Config{ - SkipStrictExistCheck: true, - }, - }) - if err != nil { - glg.Fatal(err) - } - if i%10 == 0 { - glg.Infof("Inserted %d", i) - } - } - ``` - -
- - - Wait until indexing finish. -
example code
- - ```go - wt := time.Duration(indexingWaitSeconds) * time.Second - glg.Infof("Wait %s for indexing to finish", wt) - time.Sleep(wt) - ``` - -
- - 1. Search - - - Search 10 neighbor vectors for each 20 test datasets and return a list of the neighbor vectors. - - - When getting approximate vectors, the Vald client sends search config and vector to the server via gRPC. -
example code
- - ```go - glg.Infof("Start search %d times", testCount) - for i, vec := range test[:testCount] { - res, err := client.Search(ctx, &payload.Search_Request){ - Vector: vec, - Config: &payload.Search_Config{ - Num: 10, - Radius: -1, - Epsilon: 0.1, - Timeout: 100000000, - } - } - if err != nil { - glg.Fatal(err) - } - - b, _ := json.MarshalIndent(res.GetResults(), "", " ") - glg.Infof("%d - Results : %s\n\n", i+1, string(b)) - time.Sleep(1 * time.Second) - } - ``` - -
- - 1. Remove - - - Remove 400 indexed training datasets from the Vald agent. -
example code
- - ```go - for i := range ids [:insertCount] { - _, err := client.Remove(ctx, &payload.Remove_Request{ - Id: &payload.Object_ID{ - Id: ids[i], - }, - }) - if err != nil { - glg.Fatal(err) - } - if i%10 == 0 { - glg.Infof("Removed %d", i) - } - } - ``` - -
- -## Cleanup - -In the last, you can remove the deployed Vald Cluster by executing the below command. - -```bash -helm uninstall vald -``` - -## References - -- [Get Started with NGT agent by default](https://github.com/vdaas/vald/blob/main/docs/tutorial/get-started) -- [Faiss](https://github.com/facebookresearch/faiss) diff --git a/content/docs/v1.7/tutorial/get-started.md b/content/docs/v1.7/tutorial/get-started.md index 76f885e2d..c3f2fa73e 100644 --- a/content/docs/v1.7/tutorial/get-started.md +++ b/content/docs/v1.7/tutorial/get-started.md @@ -1,6 +1,6 @@ --- title: "Get Started_v1.7/Tutorial" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 100 description: Running Vald cluster with NGT Agent on Kubernetes and execute client codes @@ -46,7 +46,7 @@ If Helm or HDF5 is not installed, please install [Helm](https://helm.sh/docs/int
Installation command for Helm
```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ```
@@ -344,7 +344,7 @@ If you are interested, please refer to [SDKs](/docs/v1.7/user-guides/sdks).
```go ctx := context.Background() - conn, err := grpc.NewClient(grpcServerAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) + conn, err := grpc.DialContext(ctx, grpcServerAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { glg.Fatal(err) } @@ -425,11 +425,11 @@ If you are interested, please refer to [SDKs](/docs/v1.7/user-guides/sdks).
1. Remove - - Remove 200 indexed training datasets from the Vald agent. + - Remove 400 indexed training datasets from the Vald agent.
example code
```go - for i := range ids [:removeCount] { + for i := range ids [:insertCount] { _, err := client.Remove(ctx, &payload.Remove_Request{ Id: &payload.Object_ID{ Id: ids[i], @@ -446,20 +446,6 @@ If you are interested, please refer to [SDKs](/docs/v1.7/user-guides/sdks).
- 1. Flush - - - Remove all remaining training datasets from the Vald agent. -
example code
- - ```go - _, err := client.Flush(ctx, &payload.Flush_Request{}) - if err != nil { - glg.Fatal(err) - } - ``` - -
- ## Cleanup In the last, you can remove the deployed Vald Cluster by executing the below command. diff --git a/content/docs/v1.7/tutorial/vald-agent-standalone-on-k8s.md b/content/docs/v1.7/tutorial/vald-agent-standalone-on-k8s.md index 73e0a92a6..36177fbf9 100644 --- a/content/docs/v1.7/tutorial/vald-agent-standalone-on-k8s.md +++ b/content/docs/v1.7/tutorial/vald-agent-standalone-on-k8s.md @@ -1,6 +1,6 @@ --- title: "Vald Agent Standalone on K8s_v1.7/Tutorial" -date: 2024-10-16T16:06:00+09:00 +date: 2024-10-28T06:58:11Z draft: false weight: 300 description: Running only Vald Agent on Kubernetes and execute client codes @@ -47,7 +47,7 @@ If Helm or HDF5 is not installed, please install [Helm](https://helm.sh/docs/int
Installation command for Helm
```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ```
@@ -247,7 +247,7 @@ This chapter uses [NGT](https://github.com/yahoojapan/ngt) as Vald Agent to perf ```go ctx := context.Background() - conn, err := grpc.NewClient(grpcServerAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) + conn, err := grpc.DialContext(ctx, grpcServerAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { glg.Fatal(err) } diff --git a/content/docs/v1.7/tutorial/vald-multicluster-on-k8s.md b/content/docs/v1.7/tutorial/vald-multicluster-on-k8s.md deleted file mode 100644 index 97f6a9c2b..000000000 --- a/content/docs/v1.7/tutorial/vald-multicluster-on-k8s.md +++ /dev/null @@ -1,314 +0,0 @@ ---- -title: "Vald Multicluster on K8s_v1.7/Tutorial" -date: 2024-10-16T16:06:00+09:00 -draft: false -weight: 500 -description: Running Multi Vald Clusters with Mirror Gateway on Kubernetes and execute client codes -menu: - tutorial: - parent: Tutorial ---- - -# Multiple Vald Clusters using Vald Mirror Gateway - -This article shows how to deploy multiple Vald clusters on your Kubernetes cluster. - -## Overview - -Vald cluster consists of multiple microservices. - -In [Get Started](https://vald.vdaas.org/docs/tutorial/get-started), you may use 4 kinds of components to deploy the Vald cluster. - -In this tutorial, you will deploy multiple Vald clusters with Vald Mirror Gateway, which connects another Vald cluster. - -The below image shows the architecture image of this case. - - - -## Requirements - -- Vald: v1.8 ~ -- Kubernetes: v1.27 ~ -- Go: v1.20 ~ -- Helm: v3 ~ -- libhdf5 (_only required to get started_) - -Helm is used to deploy Vald cluster on your Kubernetes and HDF5 is used to decode the sample data file to run the example code. - -If Helm or HDF5 is not installed, please install [Helm](https://helm.sh/docs/intro/install) and [HDF5](https://www.hdfgroup.org/). - -
Installation command for Helm
- -```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -``` - -
- -
Installation command for HDF5
- -```bash -# yum -yum install -y hdf5-devel - -# apt -apt-get install libhdf5-serial-dev - -# homebrew -brew install hdf5 -``` - -
- -## Prepare the Kubernetes Cluster - -This tutorial requires the Kubernetes cluster. - -Vald cluster runs on public Cloud Kubernetes Services such as GKE, EKS. -In the sense of trying to `Get Started`, [k3d](https://k3d.io/) or [kind](https://kind.sigs.k8s.io/) are easy Kubernetes tools to use. - -This tutorial uses [Kubernetes Metrics Server](https://github.com/kubernetes-sigs/metrics-server) for running the Vald cluster. - -Please make sure these functions are available. - -The way to deploy Kubernetes Metrics Service is here: - -```bash -kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml && \ -kubectl wait -n kube-system --for=condition=ready pod -l k8s-app=metrics-server --timeout=600s -``` - -Please prepare three Namespaces on the Kubernetes cluster. - -```bash -kubectl create namespace vald-01 && \ -kubectl create namespace vald-02 && \ -kubectl create namespace vald-03 -``` - -## Deploy Vald Clusters on each Kubernetes Namespace - -This chapter shows how to deploy the multiple Vald clusters using Helm and run it on your Kubernetes cluster. -In this section, you will deploy three Vald clusters consisting of `vald-agent-ngt`, `vald-lb-gateway`, `vald-discoverer`, `vald-manager-index`, and `vald-mirror-gateway` using the basic configuration. - -1. Clone the repository - - To use the `deployment` YAML for deployment, let’s clone `[vdaas/vald](https://github.com/vdaas/vald.git)` repository. - - ```bash - git clone https://github.com/vdaas/vald.git &&　cd vald - ``` - -2. Deploy on the `vald-01` Namespace using [dev-vald-01.yaml](https://github.com/vdaas/vald/blob/main/charts/vald/values/multi-vald/dev-vald-01.yaml) and [values.yaml](https://github.com/vdaas/vald/blob/main/example/helm/values.yaml) - - ```bash - helm install vald-cluster-01 charts/vald \ - -f ./example/helm/values.yaml \ - -f ./charts/vald/values/multi-vald/dev-vald-01.yaml \ - -n vald-01 - ``` - -3. Deploy on the `vald-02` Namespace using [dev-vald-02.yaml](https://github.com/vdaas/vald/blob/main/charts/vald/values/multi-vald/dev-vald-02.yaml) and [values.yaml](https://github.com/vdaas/vald/blob/main/example/helm/values.yaml) - - ```bash - helm install vald-cluster-02 charts/vald \ - -f ./example/helm/values.yaml \ - -f ./charts/vald/values/multi-vald/dev-vald-02.yaml \ - -n vald-02 - ``` - -4. Deploy on the `vald-03` Namespace using [dev-vald-03.yaml](https://github.com/vdaas/vald/blob/main/charts/vald/values/multi-vald/dev-vald-03.yaml) and [values.yaml](https://github.com/vdaas/vald/blob/main/example/helm/values.yaml) - - ```bash - helm install vald-cluster-03 charts/vald \ - -f ./example/helm/values.yaml \ - -f ./charts/vald/values/multi-vald/dev-vald-03.yaml \ - -n vald-03 - ``` - -5. Verify - - If success deployment, the Vald cluster’s components should run on each Kubernetes Namespace. - -
vald-01 Namespace
- - ```bash - kubectl get pods -n vald-01 - NAME READY STATUS RESTARTS AGE - vald-agent-ngt-0 1/1 Running 0 2m41s - vald-agent-ngt-2 1/1 Running 0 2m41s - vald-agent-ngt-3 1/1 Running 0 2m41s - vald-agent-ngt-4 1/1 Running 0 2m41s - vald-agent-ngt-5 1/1 Running 0 2m41s - vald-agent-ngt-1 1/1 Running 0 2m41s - vald-discoverer-77967c9697-brbsp 1/1 Running 0 2m41s - vald-lb-gateway-587879d598-xmws7 1/1 Running 0 2m41s - vald-lb-gateway-587879d598-dzn9c 1/1 Running 0 2m41s - vald-manager-index-56d474c848-wkh6b 1/1 Running 0 2m41s - vald-lb-gateway-587879d598-9wb5q 1/1 Running 0 2m41s - vald-mirror-gateway-6df75cf7cf-gzcr4 1/1 Running 0 2m26s - vald-mirror-gateway-6df75cf7cf-vjbqx 1/1 Running 0 2m26s - vald-mirror-gateway-6df75cf7cf-c2g7t 1/1 Running 0 2m41s - ``` - -
- -
vald-02 Namespace
- - ```bash - kubectl get pods -n vald-02 - NAME READY STATUS RESTARTS AGE - vald-agent-ngt-0 1/1 Running 0 2m52s - vald-agent-ngt-1 1/1 Running 0 2m52s - vald-agent-ngt-2 1/1 Running 0 2m52s - vald-agent-ngt-4 1/1 Running 0 2m52s - vald-agent-ngt-5 1/1 Running 0 2m52s - vald-agent-ngt-3 1/1 Running 0 2m52s - vald-discoverer-8cfcff76-vlmpg 1/1 Running 0 2m52s - vald-lb-gateway-54896f9f49-wtlcv 1/1 Running 0 2m52s - vald-lb-gateway-54896f9f49-hbklj 1/1 Running 0 2m52s - vald-manager-index-676855f8d7-bb4wb 1/1 Running 0 2m52s - vald-lb-gateway-54896f9f49-kgrdf 1/1 Running 0 2m52s - vald-mirror-gateway-6598cf957-t2nz4 1/1 Running 0 2m37s - vald-mirror-gateway-6598cf957-wr448 1/1 Running 0 2m52s - vald-mirror-gateway-6598cf957-jdd6q 1/1 Running 0 2m37s - ``` - -
- -
vald-03 Namespace
- - ```bash - kubectl get pods -n vald-03 - NAME READY STATUS RESTARTS AGE - vald-agent-ngt-0 1/1 Running 0 2m46s - vald-agent-ngt-1 1/1 Running 0 2m46s - vald-agent-ngt-2 1/1 Running 0 2m46s - vald-agent-ngt-3 1/1 Running 0 2m46s - vald-agent-ngt-4 1/1 Running 0 2m46s - vald-agent-ngt-5 1/1 Running 0 2m46s - vald-discoverer-879867b44-8m59h 1/1 Running 0 2m46s - vald-lb-gateway-6c8c6b468d-ghlpx 1/1 Running 0 2m46s - vald-lb-gateway-6c8c6b468d-rt688 1/1 Running 0 2m46s - vald-lb-gateway-6c8c6b468d-jq7pl 1/1 Running 0 2m46s - vald-manager-index-5596f89644-xfv4t 1/1 Running 0 2m46s - vald-mirror-gateway-7b95956f8b-l57jz 1/1 Running 0 2m31s - vald-mirror-gateway-7b95956f8b-xd9n5 1/1 Running 0 2m46s - vald-mirror-gateway-7b95956f8b-dnxbb 1/1 Running 0 2m31s - ``` - -
- -## Deploy ValdMirrorTarget Resource (Custom Resource) - -It requires applying the `ValdMirrorTarget` Custom Resource to the one Namespace. - -When applied successfully, the destination information is automatically created on other Namespaces when interconnected with each `vald-mirror-gateway`. - -This tutorial will deploy the [ValdMirrorTarget](https://github.com/vdaas/vald/blob/main/charts/vald/values/multi-vald/mirror-target.yaml) Custom Resource to the `vald-03` Namespace with the following command. - -```bash -kubectl apply -f ./charts/vald/values/multi-vald/mirror-target.yaml -n vald-03 -``` - -The current connection status can be checked with the following command. - -
Example output
- -```bash -kubectl get vmt -A -o wide -NAMESPACE NAME HOST PORT STATUS LAST TRANSITION TIME AGE -vald-03 mirror-target-01 vald-mirror-gateway.vald-01.svc.cluster.local 8081 Connected 2023-05-22T02:07:51Z 19m -vald-03 mirror-target-02 vald-mirror-gateway.vald-02.svc.cluster.local 8081 Connected 2023-05-22T02:07:51Z 19m -vald-02 mirror-target-3296010438411762394 vald-mirror-gateway.vald-01.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -vald-02 mirror-target-12697587923462644654 vald-mirror-gateway.vald-03.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -vald-01 mirror-target-13698925675968803691 vald-mirror-gateway.vald-02.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -vald-01 mirror-target-17825710563723389324 vald-mirror-gateway.vald-03.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -``` - -
- -## Run Example Code - -In this chapter, you will execute insert, search, get, and delete vectors to your Vald clusters using the example code. - -The [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) is used as a dataset for indexing and search query. - -The example code is implemented in Go and uses [vald-client-go](https://github.com/vdaas/vald-client-go), one of the official Vald client libraries, for requesting to Vald cluster. - -Vald provides multiple language client libraries such as Go, Java, Node.js, Python, etc. - -If you are interested, please refer to [SDKs](https://vald.vdaas.org/docs/user-guides/sdks). - -1. Port Forward(option) - - If you do NOT use Kubernetes Ingress, port forwarding is required to make requests from your local environment. - - ```bash - kubectl port-forward svc/vald-mirror-gateway 8080:8081 -n vald-01 & \ - kubectl port-forward svc/vald-mirror-gateway 8081:8081 -n vald-02 & \ - kubectl port-forward svc/vald-mirror-gateway 8082:8081 -n vald-03 - ``` - -2. Download dataset - - Move to the working directory. - - ```bash - cd example/client/mirror - ``` - - Download [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist), which is used as a dataset for indexing and search query. - - ```bash - wget https://ann-benchmarks.com/fashion-mnist-784-euclidean.hdf5 - ``` - -3. Run Example - - We use [example/client/mirror/main.go](https://github.com/vdaas/vald/blob/main/example/client/mirror/main.go) to run the example. - - This example will insert and index 400 vectors into the Vald cluster from the Fashion-MNIST dataset via [gRPC](https://grpc.io/). - And then, after waiting for indexing, it will request to search the nearest vector 10 times to all Vald clusters. You will get the 10 nearest neighbor vectors for each search query. - And it will request to get vectors using inserted vector ID. - - Run example codes by executing the below command. - - ```bash - go run main.go - ``` - - See [GetStarted](https://vald.vdaas.org/docs/tutorial/get-started/) for an explanation of the example code. - -## Cleanup - -Last, you can remove the deployed Vald cluster by executing the below command. - -```bash -helm uninstall vald-cluster-01 -n vald-01 && \ -helm uninstall vald-cluster-02 -n vald-02 && \ -helm uninstall vald-cluster-03 -n vald-03 -``` - -You can remove all `ValdMirrorTarget` resources (Custom Resource) with the below command. - -```bash -kubectl delete --all ValdMirrorTargets -A -``` - -You can remove all created Namespaces with the below command. - -```bash -kubectl delete namespaces vald-01 vald-02 vald-03 -``` - -## Next Steps - -Congratulation! -You completely entered the World of multiple Vald clusters! - -For more information, we recommend you check the following: - -- [Configuration](https://vald.vdaas.org/docs/user-guides/configuration/) -- [Mirror Configuration](https://vald.vdaas.org/docs/user-guides/mirroring-configuration/) -- [Network Policy](https://vald.vdaas.org/docs/user-guides/network-policy/) diff --git a/content/docs/v1.7/user-guides/backup-configuration.md b/content/docs/v1.7/user-guides/backup-configuration.md index 060e1c609..54b9955b5 100644 --- a/content/docs/v1.7/user-guides/backup-configuration.md +++ b/content/docs/v1.7/user-guides/backup-configuration.md @@ -1,6 +1,6 @@ --- title: "Backup Configuration_v1.7/User Guides" -date: 2024-02-15T17:12:31+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 200 description: Applies backup feature for saving and restoring indexes @@ -40,7 +40,7 @@ Please refer to the following tables and decide which method fit for your case. This section shows the best practice for configuring backup features with PV, S3, or PV + S3. -Each sample configuration yaml is published on [here](https://github.com/vdaas/vald/tree/main/charts/vald/values). +Each sample configuration yaml is published on [here](https://github.com/vdaas/vald/tree/master/charts/vald/values). Please refer it for more details. ### General diff --git a/content/docs/v1.7/user-guides/capacity-planning.md b/content/docs/v1.7/user-guides/capacity-planning.md index 2e411f54e..b57a8dfdb 100644 --- a/content/docs/v1.7/user-guides/capacity-planning.md +++ b/content/docs/v1.7/user-guides/capacity-planning.md @@ -1,6 +1,6 @@ --- title: "Capacity Planning_v1.7/User Guides" -date: 2024-02-15T17:12:31+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 900 description: Estimates how Kubernetes resources will be required @@ -17,32 +17,21 @@ Capacity planning is essential before deploying the Vald cluster to the cloud se There are three viewpoints: Vald cluster view, Kubernetes view, and Component view. Let's see each view. -
-When introducing production, we recommend that you actually measure how many resources are required for verification. -
- ## Vald cluster view The essential point at the Vald cluster view is the hardware specification, especially RAM. The Vald cluster, especially Vald Agent components, requires much RAM capacity because the vector index is stored in memory. -The minimum required memory for each vector (bit) is: - -```bash -// minimum required bits of vector -{ oid (64bit) + timestamp (64bit) + uuid (user defined) } * 2 + { dimension * 64 } + { the creation edge size + the search edge size } * 8 -``` - -Considering the `index size` and `index_replica`, it is easy to figure out the minimum required RAM capacity by the following formula. +It is easy to figure out the minimum required RAM capacity by the following formula. ```bash -{ minimum required bits of vector } * { the index size } * { index_replica } +( { the dimension vector } × { bit number of vector } + { the bit of vectors ID string } ) × { the maximum number of the vector } × { the index replica } ``` -For example, you want to insert 1 million vectors with 900 dimensions with 32 byte (256 bit) UUID, the index replica is 3, `creation edge size` is 20, and `search edge size` is 10, the minimum required RAM capacity is: +For example, if you want to insert 1 million vectors with 900 dimensions and the object type is 32-bit with 32 byte (256 bit) ID, and the index replica is 3, the minimum required RAM capacity is: ```bash -{(64 + 64 + 256) × 2 + (900 × 64) + (20 + 10) × 8 } × 1,000,000 × 3 = 175,824,000,000 (bit) = 21.978 (GB) +(900 × 32 + 256 ) × 1,000,000 × 3 = 8,7168,000,000 (bit) = 10.896 (GB) ``` It is just the minimum required RAM for indexing. @@ -50,12 +39,12 @@ Considering the margin of RAM capacity, the minimum RAM capacity should be less Therefore, the actual minimum RAM capacity will be: ```bash -175,824,000,000 (bit) / 0.6 = 293,040,000,000 (bit) = 36.63 (GB) +8,7168,000,000 (bit) / 0.6 = 145,280,000,000 (bit) = 18.16 (GB) ```
In the production usage, memory usage may be not enough in the minimum required RAM.
-Because for example, there are a noisy problem, high memory usage for createIndex (indexing on memory), high traffic needs more memory, etc. +E.g., there are a noisy problem, high memory usage for createIndex (indexing on memory), high traffic needs more memory, etc.
## Kubernetes cluster view diff --git a/content/docs/v1.7/user-guides/client-api-config.md b/content/docs/v1.7/user-guides/client-api-config.md index 232612d8c..ffb1b6dc7 100644 --- a/content/docs/v1.7/user-guides/client-api-config.md +++ b/content/docs/v1.7/user-guides/client-api-config.md @@ -1,6 +1,6 @@ --- title: "Client Api Config_v1.7/User Guides" -date: 2024-10-16T16:05:59+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 1000 description: Shows more detailed explanation about APIs configuration @@ -57,7 +57,7 @@ func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() target := "localhost:8080" - conn, err := grpc.NewClient(target) + conn, err := grpc.DialContext(ctx, target) if err != nil { panic(err) } @@ -173,7 +173,7 @@ func example() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() target := "localhost:8080" - conn, err := grpc.NewClient(target) + conn, err := grpc.DialContext(ctx, target) if err != nil { panic(err) } @@ -297,7 +297,7 @@ func example() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() target := "localhost:8080" - conn, err := grpc.NewClient(target) + conn, err := grpc.DialContext(ctx, target) if err != nil { panic(err) } @@ -481,7 +481,7 @@ func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() target := "localhost:8080" - conn, err := grpc.NewClient(target) + conn, err := grpc.DialContext(ctx, target) if err != nil { panic(err) } @@ -664,7 +664,7 @@ func example() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() target := "localhost:8080" - conn, err := grpc.NewClient(target) + conn, err := grpc.DialContext(ctx, target) if err != nil { panic(err) } diff --git a/content/docs/v1.7/user-guides/cluster-role-binding.md b/content/docs/v1.7/user-guides/cluster-role-binding.md index b41fe5a82..2828a4108 100644 --- a/content/docs/v1.7/user-guides/cluster-role-binding.md +++ b/content/docs/v1.7/user-guides/cluster-role-binding.md @@ -1,6 +1,6 @@ --- title: "Cluster Role Binding_v1.7/User Guides" -date: 2024-10-16T16:05:59+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 500 description: Configures cluster role for Vald cluster @@ -144,105 +144,6 @@ If you disable these configurations, the Vald Discoverer will not work, and the If you want to modify or disable these configurations, you need to grant the [cluster role configuration](https://github.com/vdaas/vald/blob/main/k8s/discoverer/clusterrole.yaml) and bind it to the Vald Discoverer to retrieve required information to operate the Vald cluster. -## Configuration for Vald Mirror Gateway - -The Vald Mirror Gateway requires configuration on cluster role and cluster role binding to create/update/retrive [ValdMirrorTarget](https://vald.vdaas.org/docs/user-guides/mirroring-configuration/) resource from the Kubernetes Cluster. - -In this section, we will describe how to configure it and how to customize these configurations. - -### Cluster role configuration for Vald Mirror Gateway - -By looking at the [cluster role configuration](https://github.com/vdaas/vald/blob/main/k8s/gateway/mirror/clusterrole.yaml), the access right of the following resources are granted to the cluster role `gateway-mirror`. - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: gateway-mirror ---- -rules: - - apiGroups: - - vald.vdaas.org - resources: - - valdmirrortargets - verbs: - - create - - update - - delete - - get - - list - - watch - - patch - - apiGroups: - - vald.vdaas.org - resources: - - valdmirrortargets/status - verbs: - - create - - update - - get - - list - - patch - - apiGroups: - - vald.vdaas.org - resources: - - valdmirrortargets/finalizers - verbs: - - update -``` - -### Cluster role binding configuration for Vald Mirror Gateway - -The cluster role binding configuration binds the cluster role `gateway-mirror` described in the previous section to the service account `gateway-mirror` according to the [configuration file](https://github.com/vdaas/vald/blob/main/k8s/gateway/mirror/clusterrolebinding.yaml). - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: gateway-mirror - ... -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: gateway-mirror -subjects: - - kind: ServiceAccount - name: gateway-mirror - namespace: default -``` - -When the role binds to the service account, the access right of the role will be granted to the service account. -In this case, all the access rights of the role `gateway-mirror` will be granted to the service account `gateway-mirror`. - -### Customize cluster role and cluster role binding configuration on Helm chart for Vald Mirror Gateway - -To customize the cluster role configuration on the Helm chart for Vald Mirror Gateway, you may need to change the `gateway.mirror.clusterRole` configuration on the Helm chart file. The cluster role configurations are enabled by default. - -```yaml -gateway: - mirror: - --- - clusterRole: - # gateway.mirror.clusterRole.enabled -- creates clusterRole resource - enabled: true - # gateway.mirror.clusterRole.name -- name of clusterRole - name: gateway-mirror - clusterRoleBinding: - # gateway.mirror.clusterRoleBinding.enabled -- creates clusterRoleBinding resource - enabled: true - # gateway.mirror..clusterRoleBinding.name -- name of clusterRoleBinding - name: gateway-mirror - serviceAccount: - # gateway.mirror.serviceAccount.enabled -- creates service account - enabled: true - # gateway.mirror.serviceAccount.name -- name of service account - name: gateway-mirror -``` - -
-If you disable these configurations, the Vald Mirror Gateway will not work properly. -
- ## Customize cluster role configuration on Cloud Providers Please refer to the official guidelines to configure cluster role configuration for your cloud provider, and configure the service account name for Vald Discoverer. diff --git a/content/docs/v1.7/user-guides/configuration.md b/content/docs/v1.7/user-guides/configuration.md index 4de6aadb4..9d853a939 100644 --- a/content/docs/v1.7/user-guides/configuration.md +++ b/content/docs/v1.7/user-guides/configuration.md @@ -1,6 +1,6 @@ --- title: "Configuration_v1.7/User Guides" -date: 2024-08-08T10:35:17+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 100 description: Applies basic configuration for running Vald cluster @@ -186,42 +186,6 @@ When the setting parameter of Vald Agent NGT is shorter than the setting value o If this happens, the Index Manager may not function properly. -#### Faiss - -Vald Agent Faiss uses [facebookresearch/faiss][faiss] as a core library for searching vectors. -The behaviors of Faiss can be configured by setting `agent.faiss` field object. - -The important parameters are the followings: - -- `agent.faiss.dimension` -- `agent.faiss.distance_type` -- `agent.faiss.m` -- `agent.faiss.metric_type` -- `agent.faiss.nbits_per_idx` -- `agent.faiss.nlist` - -Users should configure these parameters first to fit their use case. -For further details, please read [the Faiss wiki][faiss-wiki]. - -Vald Agent Faiss has a feature to start indexing automatically. -The behavior of this feature can be configured with these parameters: - -- `agent.faiss.auto_index_duration_limit` -- `agent.faiss.auto_index_check_duration` -- `agent.faiss.auto_index_length` - -
-While the Vald Agent Faiss is in the process of creating indexes, it will ignore all search requests to the target pods. -
- -
-When deploying Vald Index Manager, the above parameters should be set much longer than the Vald Index Manager settings (Please refer to the Vald Index Manager section) or minus value.
-E.g., set agent.faiss.auto_index_duration_limit to "720h" or "-1h" and agent.faiss.auto_index_check_duration to "24h" or "-1h".
-This is because the Vald Index Manager accurately grasps the index information of each Vald Agent Faiss and controls the execution timing of indexing.

-When the setting parameter of Vald Agent Faiss is shorter than the setting value of Vald Index Manager, Vald Agent Faiss may start indexing by itself without the execution command from Vald Index Manager. -If this happens, the Index Manager may not function properly. -
- #### Resource requests and limits, Pod priorities Because the Vald Agent pod places indexes on memory, termination of agent pods causes loss of indexes. @@ -405,7 +369,7 @@ For further details, there are references to the Helm values in the Vald GitHub - [README of Vald Helm Chart][vald-helm-chart] - [README of Vald-Helm-Operator Chart][vald-helm-operator-chart] - + [vald-helm-chart]: https://github.com/vdaas/vald/tree/main/charts/vald [vald-helm-operator-chart]: https://github.com/vdaas/vald/tree/main/charts/vald-helm-operator @@ -418,5 +382,3 @@ For further details, there are references to the Helm values in the Vald GitHub [kubernetes-topology-spread-constraints]: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ [yj-ngt]: https://github.com/yahoojapan/NGT [yj-ngt-wiki]: https://github.com/yahoojapan/NGT/wiki -[faiss]: https://github.com/facebookresearch/faiss -[faiss-wiki]: https://github.com/facebookresearch/faiss/wiki diff --git a/content/docs/v1.7/user-guides/deployment.md b/content/docs/v1.7/user-guides/deployment.md index 43c478b0a..165588bf2 100644 --- a/content/docs/v1.7/user-guides/deployment.md +++ b/content/docs/v1.7/user-guides/deployment.md @@ -1,6 +1,6 @@ --- title: "Deployment_v1.7/User Guides" -date: 2024-08-08T10:35:17+09:00 +date: 2024-10-28T06:58:09Z draft: false weight: 600 description: How to launch Vald cluster on your Kubernetes cluster @@ -38,7 +38,7 @@ If Helm is not installed, please install [Helm](https://helm.sh/docs/intro/insta
Installation command for Helm
```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ```
@@ -149,7 +149,7 @@ After create `values.yaml`, you can deploy by the following steps. When you need to update the configuration, you can update by following command with your new `values.yaml`. ```bash -helm upgrade vald vald/vald --values +helm upgrate vald vald/vald --values ``` ### Cleanup diff --git a/content/docs/v1.7/user-guides/filtering-configuration.md b/content/docs/v1.7/user-guides/filtering-configuration.md index 62ca46958..743f19431 100644 --- a/content/docs/v1.7/user-guides/filtering-configuration.md +++ b/content/docs/v1.7/user-guides/filtering-configuration.md @@ -1,6 +1,6 @@ --- title: "Filtering Configuration_v1.7/User Guides" -date: 2024-10-16T16:05:59+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 300 description: Applies filtering feature to the Vald cluster @@ -164,7 +164,7 @@ func main() { // connect to the Vald cluster ctx := context.Background() - conn, err := grpc.NewClient(grpcServerAddr, grpc.WithInsecure()) + conn, err := grpc.DialContext(ctx, grpcServerAddr, grpc.WithInsecure()) if err != nil { log.Error(err) return diff --git a/content/docs/v1.7/user-guides/mirroring-configuration.md b/content/docs/v1.7/user-guides/mirroring-configuration.md deleted file mode 100644 index eed9fc518..000000000 --- a/content/docs/v1.7/user-guides/mirroring-configuration.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -title: "Mirroring Configuration_v1.7/User Guides" -date: 2024-10-16T16:05:59+09:00 -draft: false -weight: 400 -description: Applies mirror gateway for running multi Vald clusters -menu: - userguides: - parent: User Guides ---- - -# Mirror Configuration - -This page describes how to enable mirroring features on the Vald cluster. - -Before you use the mirroring functions, please look at [the Vald Mirror Gateway document](/docs/v1.7/overview/component/mirror-gateway) for what you can do. - -## Requirement - -- Vald version: v1.8 -- The number of Vald clusters: 2~ - -## Configuration - -This chapter shows how to configure values.yaml to enable Vald Mirror Gateway and how to interconnect Vald Mirror Gateways. - -The setting points are the followings: - -1. Enable the Vald Mirror Gateway using Helm values configuration -2. Interconnect Vald Mirror Gateways using the Custom Resource configuration - -### Helm Values Configuration - -The Helm values configuration is required for each Vald cluster to be deployed. - -It is easy to enable the mirroring feature. - -```yaml ---- -gateway: - mirror: - enabled: true -``` - -If you want to make more detailed settings, please set the following parameters. - -```yaml -gateway: - mirror: - ... - gateway_config: - ... - # gRPC client configuration (overrides defaults.grpc.client) - client: {} - # The duration to register other Mirror Gateways. - register_duration: "1s" - # The target namespace to discover ValdMirrorTarget (CR) resource. - # The default value is its own namespace. - namespace: "vald" - # The group name of the Mirror Gateways (optional). - # It is used to discover ValdMirrorTarget resources (CR) with the same group name. - # The default value is empty. - group: "group1" - # The duration to discover other mirror gateways in the same group. - discovery_duration: 1s - # The colocation name of the data center (optional). - colocation: "dc1" -``` - -The cluster role configuration is required when you deploy Vald clusters with Vald Mirror Gateway on multiple Namespaces in the Kubernetes cluster. - -Please refer to [Cluster Role Configuration](/docs/v1.7/user-guides/cluster-role-binding) about cluster role settings for Mirror Gateway. - -### Custom Resource Configuration - -The Mirror Gateway is not connected to other mirror gateways when deployed. - -The Vald Mirror Gateway connects to another Mirror Gateway component specified in the `ValdMirrorTarget` resource (Custom Resource). - -Based on this resource, if the connection succeeds, the Mirror Gateway will interconnect with another. - -```yaml -apiVersion: vald.vdaas.org/v1 -kind: ValdMirrorTarget -metadata: - name: mirror-target-01 - namespace: vald-03 - labels: - # The group name of the Mirror Gateways. - group: mirror-group-01 -spec: - # Colocation name. (optional) - colocation: dc1 - # The connection target to another mirror gateway. - target: - # The hostname. - host: vald-mirror-gateway.vald-01.svc.cluster.local - # The port number - port: 8081 -``` diff --git a/content/docs/v1.7/user-guides/observability-configuration.md b/content/docs/v1.7/user-guides/observability-configuration.md index d288ec4b7..d23f37bd2 100644 --- a/content/docs/v1.7/user-guides/observability-configuration.md +++ b/content/docs/v1.7/user-guides/observability-configuration.md @@ -1,6 +1,6 @@ --- title: "Observability Configuration_v1.7/User Guides" -date: 2024-10-16T16:05:59+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 1100 description: Monitors performance on metrics dashboards @@ -16,14 +16,14 @@ By enabling observability, you can monitor and visualize the number of indexes, This page shows the best practice for applying observing features by constructing an observability environment and setting the Vald Helm chart. -## Architecture +## Arhitecture -Vald conforms [OpenTelemetry Protocol](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and does NOT depend on any commercial data format. +Vald conforms [OpenTeremetry Protocol](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) and does NOT depend on any commercial data format. OpenTelemetry Collector handles receiving / processing / exporting the telemetry data, which does NOT depend on the vendors' format. -All Vald components can send OTLP-compliant telemetry data, like metrics, traces, or logs, to the OpenTelemetry Collector. +All Vald components can send OTLP-compliant telemetry data, like metrics, traces, or logs, to the OpenTelemerty Collector. The telemetry data is used to monitor or visualize with the observing tools. ## Build an observing environment @@ -34,7 +34,7 @@ The recommended observability environment consists of the following: - Jaeger Operator and Jaeger - Prometheus Operator and Prometheus - Grafana -- OpenTelemetry Operator and OpenTelemetry Collector +- OpenTelemetry Operator and OpenTelemtry Collector Vald provides [the default manifests](https://github.com/vdaas/vald/tree/main/k8s/metrics) and [the make commands](https://github.com/vdaas/vald/blob/main/Makefile.d/k8s.mk) for deploying those components. Please refer to the following sections to deploy each component. @@ -78,7 +78,7 @@ make k8s/metrics/grafana/deploy ### OpenTelemetry Operator and OpenTelemetry Collector Vald uses OpenTelemetry Collector to get the telemetry data and export it to monitor backend components. -The following command deploys OpenTelemetry Collector via OpenTelemetry Collector. +The following command deploys OpenTelemtry Collector via OpenTelemerty Collector.
Before executing the following command, please ensure the Prometheus operator runs healthy. @@ -143,7 +143,7 @@ defaults: - "go_version" - "go_os" - "go_arch" - - "algorithm_info" + - "ngt_version" # enable memory metrics enable_memory: true # enable goroutine metrics @@ -187,7 +187,7 @@ defaults: #### Specify the Telemetry attribute -You can add the component information to the attribute of telemetry data by editing `defaults.observability.otlp.attribute`. +You can add the component information to the attribute of telemetry data by editing `defaults.observability.otlp.attirbute`. E.g., when setting `vald-agent-ngt-0` as `agent.observability.otlp.attribute.pod_name`, `target_pod: vald-agent-ngt-0` will be added to the attribute. These attributes are set auto by the environment values, so Vald recommends using default values unless there is a specific reason. diff --git a/content/docs/v1.7/user-guides/read-replica-and-rotator.md b/content/docs/v1.7/user-guides/read-replica-and-rotator.md deleted file mode 100644 index f053d92cd..000000000 --- a/content/docs/v1.7/user-guides/read-replica-and-rotator.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: "Read Replica and Rotator_v1.7/User Guides" -date: 2024-08-08T10:35:17+09:00 -draft: false -weight: 1500 -description: How to improve search request speed -menu: - userguides: - parent: User Guides ---- - -# Read Replica and Rotator - -Read replica enhances the search QPS (Queries Per Second) of the Vald cluster by deploying read-only agents in addition to the regular agents and distributing the requests among them. Read replica is deployed as Kubernetes deployments, and depending on the number of replicas (N), QPS increases by approximately 1.7 to 1.8 times \* N. - -
-The increase in QPS is possible with sufficient infrastructure (see Important notes). -
- -## How to deploy read replica - -The read replica is managed with a separate chart from the Vald cluster and is deployed as an addon to the Vald cluster. The Vald cluster should be deployed first, followed by the deployment of the read replica. - -> The reason Vald and Vald-readreplica are in separate charts is to avoid conflicts between the read replica's restart and the Helm operator's processes when Vald is managed by a helm operator. Therefore, the read replica will always be deployed using Helm commands. - -### When you deploy Vald with Helm command - -1. Edit `values.yaml` like below (Please refer to [deployment](/docs/v1.7/user-guides/deployment) for other fields.) - - ```yaml - agent: - ngt: - export_index_info_to_k8s: true - readreplica: - enabled: true - minReplicas: 1 # if you don't use hpa, this will be the replicas of the Deployment - maxReplicas: 3 - hpa: - enabled: true # if you prefer to use hpa - targetCPUUtilizationPercentage: 80 - manager: - index: - operator: - enabled: true - rotation_job_concurrency: 2 - ``` - -1. Deploy Vald cluster - - ```bash - helm install vald vald/vald --values values.yaml - ``` - -1. Deploy `vald-readreplica` with the same `values.yaml` - - ```bash - helm install vald-readreplica vald/vald-readreplica --values values.yaml - ``` - -### When you deploy Vald cluster with `vald-helm-operator` - -1. Edit `valdrelease.yaml` with the same fields as above - -1. Deploy Vald cluster - - ```bash - helm install vald-helm-operator-release vald/vald-helm-operator - kubectl apply -f valdrelease.yaml - ``` - -1. Deploy `vald-readreplica` - - ```bash - helm install vald-readreplica vald/vald-readreplica --values - ``` - -## Architecture - -Read replica mainly consists of the following four parts. - -Read Replica Architecture - -### Read replica deployment - -The deployment that generates Pods where the actual processing of read replica takes place. Read replica accepts read requests (search) and reads the index from the read replica PVC. - -### Read replica PVC - -The PVC for read replica Pods is used to read the index. It is generated based on the latest snapshot from the PVC of the regular agent. Unlike the agent PVC, it is generated as ROX, allowing it to be read from multiple Pods. - -### Index operator - -The operator handles the following processes: - -1. Monitoring the time when the agent saved the index to the PVC and when the read replica performed index rotation -1. Generating [Read replica rotator](#read-replica-rotator) job when an index save occurs after the most recent rotation - -> The Index operator also manages the timing of index create/save operations other than those mentioned above. Please refer to another document for details. - -### Read replica rotator - -The Kubernetes job handles the following processes: - -1. Creating a snapshot from the agent's PVC -1. Generating a PVC for read replica from the snapshot -1. Rolling update of the read replica deployment to launch a group of read replica pods with the latest index - -## Important notes - -Result consistency is guaranteed - -There is a time lag between index insertion, agent save, and the completion of read replica rotation. During this time, there may be inconsistencies between the index in the agent itself and the index in the read replica. - -- Sufficient infrastructure is required for QPS scaling - - Even if read replicas are deployed, QPS will not scale if sufficient resources are not available in the Kubernetes cluster. Specifically, agent resources and read replica resources should be deployed on separate nodes. Vald sets `podAntiAffinity` to ensure that agent resources and read replica resources are deployed on separate nodes as much as possible. diff --git a/content/docs/v1.7/user-guides/upgrade-cluster.md b/content/docs/v1.7/user-guides/upgrade-cluster.md index 88acd41ee..214b89114 100644 --- a/content/docs/v1.7/user-guides/upgrade-cluster.md +++ b/content/docs/v1.7/user-guides/upgrade-cluster.md @@ -1,6 +1,6 @@ --- title: "Upgrade Cluster_v1.7/User Guides" -date: 2024-08-08T10:35:17+09:00 +date: 2024-10-28T06:58:10Z draft: false weight: 800 description: Describes hot to upgrade Vald version for the running Vald cluster @@ -38,7 +38,7 @@ You can edit your `values.yaml` and update your Vald cluster by following steps: ```bash helm upgrade vald/vald --values - # e.g., helm upgrade vald vald/vald --values values.yaml + # e.g., helm upgrade vald vald/vald --values valeus.yaml ``` ### Using vald-helm-operator @@ -118,7 +118,7 @@ If there is major or minor upgrading, the chart structure may have changed. ```bash helm upgrade vald/vald --values - # e.g., helm upgrade vald vald/vald --values values.yaml + # e.g., helm upgrade vald vald/vald --values valeus.yaml ``` ### Using vald-helm-operator diff --git a/preview b/preview index f9ead8eb1..9dc503098 160000 --- a/preview +++ b/preview @@ -1 +1 @@ -Subproject commit f9ead8eb1e3135b3223314959268c51109598078 +Subproject commit 9dc5030983b2047fe97a12383a53bb441ca22070