From a38573ccb1bdcefd76624d8f6cc5b271d627be4b Mon Sep 17 00:00:00 2001 From: hlts2 Date: Thu, 12 Oct 2023 17:44:17 +0900 Subject: [PATCH 01/19] fix: update mirror document Signed-off-by: hlts2 --- docs/api/mirror-gateway.md | 78 +-------------------- docs/overview/component/mirror-gateway.md | 46 ++++++------ docs/user-guides/mirroring-configuration.md | 4 +- 3 files changed, 25 insertions(+), 103 deletions(-) diff --git a/docs/api/mirror-gateway.md b/docs/api/mirror-gateway.md index 69e9de3cf6..4b05dad5c2 100644 --- a/docs/api/mirror-gateway.md +++ b/docs/api/mirror-gateway.md @@ -2,13 +2,11 @@ ## Overview -Mirror Service is responsible for providing the `Register` and `Advertise` interface for the Vald Mirror Gateway. +Mirror Service is responsible for providing the `Register` interface for the Vald Mirror Gateway. ```rpc service Mirror { rpc Register(payload.v1.Mirror.Targets) returns (payload.v1.Mirror.Targets) {} - - rpc Advertise(payload.v1.Mirror.Targets) returns (payload.v1.Mirror.Targets) {} } ``` @@ -85,77 +83,3 @@ Register RPC is the method to register other Vald Mirror Gateway targets. | 3 | INVALID_ARGUMENT | | 4 | DEADLINE_EXCEEDED | | 13 | INTERNAL | - -## Advertise RPC - -Advertise RPC is the method to advertise Vald Mirror Gateway targets. - -### Input - -- the scheme of `payload.v1.Mirror.Targets`. - - ```rpc - message Mirror { - message Target { - string host = 1; - uint32 port = 2; - } - - message Targets { - repeated Target targets = 1; - } - } - ``` - - - Mirror.Targets - - | field | type | label | required | desc. | - | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | - | targets | Mirror.Target | repeated(Array[Mirror.Target]) | \* | The multiple target information. | - - - Mirror.Target - - | field | type | label | required | desc. | - | :---: | :----- | :---- | :------: | :------------------- | - | host | string | | \* | The target hostname. | - | port | uint32 | | \* | The target port. | - -### Output - -- the scheme of `payload.v1.Mirror.Targets`. - -```rpc -message Mirror { - message Target { - string host = 1; - uint32 port = 2; - } - - message Targets { - repeated Target targets = 1; - } -} -``` - -- Mirror.Targets - - | field | type | label | required | desc. | - | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | - | targets | Mirror.Target | repeated(Array[Mirror.Target]) | | The multiple target information. | - -- Mirror.Target - - | field | type | label | required | desc. | - | :---: | :----- | :---- | :------: | :------------------- | - | host | string | | \* | The target hostname. | - | port | uint32 | | \* | The target port. | - -### Status Code - -| code | desc. | -| :--: | :---------------- | -| 0 | OK | -| 1 | CANCELLED | -| 3 | INVALID_ARGUMENT | -| 4 | DEADLINE_EXCEEDED | -| 13 | INTERNAL | diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 8f40016a03..d862a8c5fb 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -44,35 +44,33 @@ Then, while forwarding the user request, the Vald Mirror Gateway bypasses the in On the other hand, if the incoming user request is an [Object API](https://vald.vdaas.org/docs/api/object/) or [Search API](https://vald.vdaas.org/docs/api/search/), it is bypassed to only a Vald LB Gateway in its own cluster without forwarding it to other Vald Mirror Gateways. -### Automatic rollback on failure +### Continuous processing on failure The request may fail at the forwarding destination or the bypass destination. -If some requests fail, the vector data will not be consistent across Vald clusters. +If some of the requests fails, the processing continues based on their status code. -To keep index state consistency, the Vald Mirror Gateway will send the rollback request for the failed request. After the rollback request succeeds, the index state will be the same as before requesting. - -The following is the list of rollback types. +The following is an overview of the process for each request. - Insert Request - - Rollback Condition/Trigger - - Status code other than `ALREADY_EXISTS` exists - - Rollback request to the successful request - - REMOVE request -- Remove Request - - Rollback Condition/Trigger - - Status code other than `NOT_FOUND` exists - - Rollback request to the successful request - - UPSERT Request with old vector + + - If the target host returns a status of `ALREADY_EXISTS`, the Update request is sent to this host. + - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. + - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, returns `OK` to the user without continuous processing. + - Update Request - - Rollback Condition/Trigger - - Status code other than `ALREADY_EXISTS` exists - - Rollback request to the successful request - - REMOVE Request if there is no old vector data - - UPDATE Request if there is old vector data + + - If the target host returns a status of `NOT_FOUND`, the Insert request is sent to this host. + - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. + - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, returns `OK` to the user without continuous processing. + - Upsert Request - - Rollback Condition/Trigger - - Status code other than `ALREADY_EXISTS` exists - - Rollback request to the successful request - - REMOVE Request if there is no old vector data - - UPDATE Request if there is old vector data + + - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, returns `OK` to the user without continuous processing. + - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. + +- Remove Request + - If all target hosts returns a status of `OK` or `NOT_FOUND`, returns `OK` to the user without continuous processing. + - If all target hosts returns a status of `NOT_FOUND`, returns `NOT_FOUND` to the user without continuous processing. + +For more information, please refer to [Mirror Gateway Troubleshooting](https://vald.vdaas.org/docs/troubleshooting/mirror-gateway/). diff --git a/docs/user-guides/mirroring-configuration.md b/docs/user-guides/mirroring-configuration.md index 4b30e04b2c..3bfecd31fb 100644 --- a/docs/user-guides/mirroring-configuration.md +++ b/docs/user-guides/mirroring-configuration.md @@ -41,8 +41,8 @@ gateway: ... # gRPC client configuration (overrides defaults.grpc.client) client: {} - # The interval to advertise addresses of Mirror Gateway to other Mirror Gateway. - advertise_interval: "1s" + # The duration to register other Mirror Gateways. + register_duration: "1s" # The target namespace to discover ValdMirrorTarget (CR) resource. # The default value is its own namespace. namespace: "vald" From c16545dbd3a98c7dc24e8d9c25515d0437794abf Mon Sep 17 00:00:00 2001 From: hlts2 Date: Thu, 12 Oct 2023 17:48:17 +0900 Subject: [PATCH 02/19] fix: format Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index d862a8c5fb..d363d24260 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -70,6 +70,7 @@ The following is an overview of the process for each request. - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. - Remove Request + - If all target hosts returns a status of `OK` or `NOT_FOUND`, returns `OK` to the user without continuous processing. - If all target hosts returns a status of `NOT_FOUND`, returns `NOT_FOUND` to the user without continuous processing. From f7fb3ab39f39062ad34ca014b8e62516f2e7822c Mon Sep 17 00:00:00 2001 From: hlts2 Date: Thu, 12 Oct 2023 18:03:33 +0900 Subject: [PATCH 03/19] feat: add removeByTimestamp section Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index d363d24260..5031def552 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -74,4 +74,8 @@ The following is an overview of the process for each request. - If all target hosts returns a status of `OK` or `NOT_FOUND`, returns `OK` to the user without continuous processing. - If all target hosts returns a status of `NOT_FOUND`, returns `NOT_FOUND` to the user without continuous processing. +- RemoveByTimestamp Request + + - Same as `Remove Request`. + For more information, please refer to [Mirror Gateway Troubleshooting](https://vald.vdaas.org/docs/troubleshooting/mirror-gateway/). From 0350329dc860ae21b88484a9044685ece065f0cc Mon Sep 17 00:00:00 2001 From: hlts2 Date: Thu, 12 Oct 2023 18:07:34 +0900 Subject: [PATCH 04/19] fix: grammar warning Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 5031def552..d3f990302a 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -55,24 +55,24 @@ The following is an overview of the process for each request. - Insert Request - If the target host returns a status of `ALREADY_EXISTS`, the Update request is sent to this host. - - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. - - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, returns `OK` to the user without continuous processing. + - If all target hosts returns a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS` to the user without continuous processing. + - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK` to the user without continuous processing. - Update Request - If the target host returns a status of `NOT_FOUND`, the Insert request is sent to this host. - - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. - - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, returns `OK` to the user without continuous processing. + - If all target hosts returns a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS` to the user without continuous processing. + - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK` to the user without continuous processing. - Upsert Request - - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, returns `OK` to the user without continuous processing. - - If all target hosts returns a status of `ALREADY_EXISTS`, returns `ALREADY_EXISTS` to the user without continuous processing. + - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK` to the user without continuous processing. + - If all target hosts returns a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS` to the user without continuous processing. - Remove Request - - If all target hosts returns a status of `OK` or `NOT_FOUND`, returns `OK` to the user without continuous processing. - - If all target hosts returns a status of `NOT_FOUND`, returns `NOT_FOUND` to the user without continuous processing. + - If all target hosts returns a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK` to the user without continuous processing. + - If all target hosts returns a status of `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND` to the user without continuous processing. - RemoveByTimestamp Request From a5a7e7b28c32f3bf8bea8536825f1556c781eec0 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 11:11:56 +0900 Subject: [PATCH 05/19] fix: grammar warning and refactor document Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index d3f990302a..482fd2b141 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -54,25 +54,25 @@ The following is an overview of the process for each request. - Insert Request - - If the target host returns a status of `ALREADY_EXISTS`, the Update request is sent to this host. - - If all target hosts returns a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS` to the user without continuous processing. - - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK` to the user without continuous processing. + - If the target host return a status of `ALREADY_EXISTS`, the Update request is sent to this host. + - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. + - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Update Request - - If the target host returns a status of `NOT_FOUND`, the Insert request is sent to this host. - - If all target hosts returns a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS` to the user without continuous processing. - - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK` to the user without continuous processing. + - If the target host return a status of `NOT_FOUND`, the Insert request is sent to this host. + - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. + - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Upsert Request - - If all target hosts returns a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK` to the user without continuous processing. - - If all target hosts returns a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS` to the user without continuous processing. + - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. + - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Remove Request - - If all target hosts returns a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK` to the user without continuous processing. - - If all target hosts returns a status of `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND` to the user without continuous processing. + - If all target hosts return a status of `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND`. + - If all target hosts return a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. - RemoveByTimestamp Request From caf353b306afa8c986db303d510b80443183f2b8 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 11:14:36 +0900 Subject: [PATCH 06/19] fix: grammar warning Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 482fd2b141..36df765983 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -54,13 +54,13 @@ The following is an overview of the process for each request. - Insert Request - - If the target host return a status of `ALREADY_EXISTS`, the Update request is sent to this host. + - If the target host returns a status of `ALREADY_EXISTS`, the Update request is sent to this host. - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Update Request - - If the target host return a status of `NOT_FOUND`, the Insert request is sent to this host. + - If the target host returns a status of `NOT_FOUND`, the Insert request is sent to this host. - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. From 2c492f722de2c5a473ce240c5a7d6b7e876f6c71 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 11:18:52 +0900 Subject: [PATCH 07/19] feat: add mirror gateway troubleshooting docs Signed-off-by: hlts2 --- docs/troubleshooting/mirror-gateway.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 docs/troubleshooting/mirror-gateway.md diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md new file mode 100644 index 0000000000..1cd0192974 --- /dev/null +++ b/docs/troubleshooting/mirror-gateway.md @@ -0,0 +1,14 @@ +# Mirror Gateway Troubleshooting + +This page introduces the popular troubleshooting for Mirror Gateway. + +Additionally, if you encounter some errors when using API, the [API status code](../api/status.md) helps you, too. + +## Insert Operation + +## Update Operation + +## Remove Operation + +## RemoveByTimestamp Operation + From e314d817ea9f2326c5dd9381b991512680de1ed2 Mon Sep 17 00:00:00 2001 From: "deepsource-autofix[bot]" <62050782+deepsource-autofix[bot]@users.noreply.github.com> Date: Mon, 16 Oct 2023 02:19:17 +0000 Subject: [PATCH 08/19] style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 2c492f7 according to the output from Gofumpt and Prettier. Details: https://github.com/vdaas/vald/pull/2207 --- docs/troubleshooting/mirror-gateway.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index 1cd0192974..040a3e8d1b 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -11,4 +11,3 @@ Additionally, if you encounter some errors when using API, the [API status code] ## Remove Operation ## RemoveByTimestamp Operation - From 54f89ed166cf94943df42cd75a1876b5b0f0f33b Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 12:39:47 +0900 Subject: [PATCH 09/19] feat: add contents of troubleshooting document Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 1 + docs/troubleshooting/mirror-gateway.md | 81 +++++++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 36df765983..0da42f5698 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -63,6 +63,7 @@ The following is an overview of the process for each request. - If the target host returns a status of `NOT_FOUND`, the Insert request is sent to this host. - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. + - If all target hosts return a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. - Upsert Request diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index 040a3e8d1b..2a02aaf692 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -6,8 +6,89 @@ Additionally, if you encounter some errors when using API, the [API status code] ## Insert Operation +Mirror Gateway sends an Update request to its host if some requests are `ALREADY_EXISTS`. + +Therefore, in addition to the [Insert API status code](../api/insert.md#status-code), the [Update API status code](../api/update.md#status-code) may also be returned to the user. + +Here are some common reasons of error. + +| name | common reason | +| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | +| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | +| INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | +| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | +| ALREADY_EXISTS | Request ID is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | +| NOT_FOUND | Requested ID is NOT inserted. This is the status code of the Update request. | +| INTERNAL | Target Vald cluster or network route has some critical error. | + +`0 (OK)` is also returned when all hosts return `OK` and `ALREADY_EXISTS`. + ## Update Operation +Mirror Gateway sends an Update request to its host if some requests are `NOT_FOUND`. + +Therefore, in addition to the [Update API status code](../api/update.md#status-code), the [Insert API status code](../api/insert.md#status-code) may also be returned to the user. + +Here are some common reasons of error. + +| name | common reason | +| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | +| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | +| INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | +| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | +| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | +| ALREADY_EXISTS | Request pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | +| INTERNAL | Target Vald cluster or network route has some critical error. | + +`0 (OK)` is also returned in the following cases. + +- All hosts return `OK` and `NOT_FOUND`. +- All hosts return `OK` and `ALREADY_EXISTS`. + +## Upsert Operation + +The request process may not be completed when the response code is NOT `0 (OK)`. + +Here are some common reasons of error. + +| name | common reason | +| :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | +| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | +| INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | +| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | +| ALREADY_EXISTS | Requested pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | +| INTERNAL | Target Vald cluster or network route has some critical error. | + +`0 (OK)` is also returned when all hosts return `OK` and `ALREADY_EXISTS`. + ## Remove Operation +The request process may not be completed when the response code is NOT `0 (OK)`. + +Here are some common reasons of error. + +| name | common reason | +| :---------------- | :---------------------------------------------------------------------------------------------- | +| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | +| INVALID_ARGUMENT | The Requested vector's ID is empty, or some request payload is invalid. | +| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | +| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | +| INTERNAL | Target Vald cluster or network route has some critical error. | + +`0 (OK)` is also returned when all hosts return `OK` and `NOT_FOUND`. + ## RemoveByTimestamp Operation + +The request process may not be completed when the response code is NOT `0 (OK)`. + +Here are some common reasons of error. + +| name | common reason | +| :---------------- | :---------------------------------------------------------------------------------------------- | +| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | +| INVALID_ARGUMENT | The Requested vector's ID is empty, or some request payload is invalid. | +| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | +| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | +| INTERNAL | Target Vald cluster or network route has some critical error. | + +`0 (OK)` is also returned when all hosts return `OK` and `NOT_FOUND`. From 974997cb4fc976c46bbea93d4ff64d67458d6cb1 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 12:48:46 +0900 Subject: [PATCH 10/19] fix: grammar warning Signed-off-by: hlts2 --- docs/troubleshooting/mirror-gateway.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index 2a02aaf692..d5af09999e 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -37,10 +37,10 @@ Here are some common reasons of error. | INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | | DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | | NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | -| ALREADY_EXISTS | Request pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | +| ALREADY_EXISTS | Request a pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned in the following cases. +`0 (OK)` is also returned in the following cases: - All hosts return `OK` and `NOT_FOUND`. - All hosts return `OK` and `ALREADY_EXISTS`. From 065a5e4de22defd7f843b147751cddcc59659cb5 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 12:51:03 +0900 Subject: [PATCH 11/19] fix: grammar Signed-off-by: hlts2 --- docs/troubleshooting/mirror-gateway.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index d5af09999e..159545aab8 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -40,7 +40,7 @@ Here are some common reasons of error. | ALREADY_EXISTS | Request a pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned in the following cases: +`0 (OK)` is also returned to the following cases: - All hosts return `OK` and `NOT_FOUND`. - All hosts return `OK` and `ALREADY_EXISTS`. From 8fabe34136f58c13cd07aae142fdf0864797205f Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 14:04:03 +0900 Subject: [PATCH 12/19] fix: link path Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 0da42f5698..ab2cde27d2 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -10,7 +10,7 @@ This component makes it possible to enhance availability during a cluster failur Vald Mirror Gateway is responsible for the followings: -- Forward user requests ([Insert](https://vald.vdaas.org/docs/api/insert/) / [Upsert](https://vald.vdaas.org/docs/api/upsert/) / [Update](https://vald.vdaas.org/docs/api/update/) / [Remove](https://vald.vdaas.org/docs/api/remove/)) to the other Vald Mirror Gateways in the same group. +- Forward user requests ([Insert](../../api/insert.md) / [Upsert](../../api/upsert.md) / [Update](../../api/update.md) / [Remove](../../api/remove.md)) to the other Vald Mirror Gateways in the same group. - Manages the state of indexes stored in all clusters to ensure they are consistent. ## Features @@ -33,16 +33,16 @@ The `ValdMirrorTarget` is a Custom Resource related to the connection destinatio When two Vald clusters contain Vald Mirror Gateways, Vald Mirror Gateways can send the request to each other by applying `ValdMirrorTarget`. -For more information about `ValdMirrorTarget` configuration, please refer to [Custom Resource Configuration](https://vald.vdaas.org/docs/user-guides/mirroring-configuration/). +For more information about `ValdMirrorTarget` configuration, please refer to [Custom Resource Configuration](../../user-guides/mirroring-configuration.md). ### Request forwarding -The Vald Mirror Gateway forwards the incoming user request ([Insert](https://vald.vdaas.org/docs/api/insert/) / [Upsert](https://vald.vdaas.org/docs/api/upsert/) / [Update](https://vald.vdaas.org/docs/api/update/) / [Remove](https://vald.vdaas.org/docs/api/remove/)) to other Vald Mirror Gateways. +The Vald Mirror Gateway forwards the incoming user request ([Insert](../../api/insert.md) / [Upsert](../../api/upsert.md) / [Update](../../api/update.md) / [Remove](../../api/remove.md)) to other Vald Mirror Gateways. Then, while forwarding the user request, the Vald Mirror Gateway bypasses the incoming user request to Vald LB Gateway in its own cluster. -On the other hand, if the incoming user request is an [Object API](https://vald.vdaas.org/docs/api/object/) or [Search API](https://vald.vdaas.org/docs/api/search/), it is bypassed to only a Vald LB Gateway in its own cluster without forwarding it to other Vald Mirror Gateways. +On the other hand, if the incoming user request is an [Object API](../../api/object.md) or [Search API](../../api/search.md), it is bypassed to only a Vald LB Gateway in its own cluster without forwarding it to other Vald Mirror Gateways. ### Continuous processing on failure @@ -79,4 +79,4 @@ The following is an overview of the process for each request. - Same as `Remove Request`. -For more information, please refer to [Mirror Gateway Troubleshooting](https://vald.vdaas.org/docs/troubleshooting/mirror-gateway/). +For more information, please refer to [Mirror Gateway Troubleshooting](../../troubleshooting/mirror-gateway.md). From adb3877b949e4d7bbe216f291614e67fc629bc2a Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 14:45:35 +0900 Subject: [PATCH 13/19] fix: invalid link path Signed-off-by: hlts2 --- docs/user-guides/mirroring-configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guides/mirroring-configuration.md b/docs/user-guides/mirroring-configuration.md index 3bfecd31fb..6926f53985 100644 --- a/docs/user-guides/mirroring-configuration.md +++ b/docs/user-guides/mirroring-configuration.md @@ -2,7 +2,7 @@ This page describes how to enable mirroring features on the Vald cluster. -Before you use the mirroring functions, please look at [the Vald Mirror Gateway document](../../docs/overview/component/mirror-gateway.md) for what you can do. +Before you use the mirroring functions, please look at [the Vald Mirror Gateway document](../overview/component/mirror-gateway.md) for what you can do. ## Requirement @@ -58,7 +58,7 @@ gateway: The cluster role configuration is required when you deploy Vald clusters with Vald Mirror Gateway on multiple Namespaces in the Kubernetes cluster. -Please refer to [Cluster Role Configuration](https://vald.vdaas.org/docs/user-guides/cluster-role-binding/) about cluster role settings for Mirror Gateway. +Please refer to [Cluster Role Configuration](./cluster-role-binding.md) about cluster role settings for Mirror Gateway. ### Custom Resource Configuration From bbf8bbf1d712455bedf507d77eda7457fe4fdfca Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 15:36:05 +0900 Subject: [PATCH 14/19] fix: refactor docs Signed-off-by: hlts2 --- docs/troubleshooting/mirror-gateway.md | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-) diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index 159545aab8..0e4aeb2706 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -21,7 +21,7 @@ Here are some common reasons of error. | NOT_FOUND | Requested ID is NOT inserted. This is the status code of the Update request. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` and `ALREADY_EXISTS`. +`0 (OK)` is also returned when all hosts return `OK` or `ALREADY_EXISTS`. ## Update Operation @@ -42,8 +42,8 @@ Here are some common reasons of error. `0 (OK)` is also returned to the following cases: -- All hosts return `OK` and `NOT_FOUND`. -- All hosts return `OK` and `ALREADY_EXISTS`. +- All hosts return `OK` or `NOT_FOUND`. +- All hosts return `OK` or `ALREADY_EXISTS`. ## Upsert Operation @@ -59,7 +59,7 @@ Here are some common reasons of error. | ALREADY_EXISTS | Requested pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` and `ALREADY_EXISTS`. +`0 (OK)` is also returned when all hosts return `OK` or `ALREADY_EXISTS`. ## Remove Operation @@ -75,20 +75,4 @@ Here are some common reasons of error. | NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` and `NOT_FOUND`. - -## RemoveByTimestamp Operation - -The request process may not be completed when the response code is NOT `0 (OK)`. - -Here are some common reasons of error. - -| name | common reason | -| :---------------- | :---------------------------------------------------------------------------------------------- | -| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | -| INVALID_ARGUMENT | The Requested vector's ID is empty, or some request payload is invalid. | -| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | -| INTERNAL | Target Vald cluster or network route has some critical error. | - -`0 (OK)` is also returned when all hosts return `OK` and `NOT_FOUND`. +`0 (OK)` is also returned when all hosts return `OK` or `NOT_FOUND`. From a1c8a33355cb94e5973a9216b14e4d62b90fde25 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 15:38:46 +0900 Subject: [PATCH 15/19] fix: deleted unnecessary contents Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index ab2cde27d2..00af63c19d 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -70,13 +70,9 @@ The following is an overview of the process for each request. - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. -- Remove Request +- Remove/RemoveByTimestamp Request - If all target hosts return a status of `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND`. - If all target hosts return a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. -- RemoveByTimestamp Request - - - Same as `Remove Request`. - For more information, please refer to [Mirror Gateway Troubleshooting](../../troubleshooting/mirror-gateway.md). From c76a10a5d15d1d4628e62257ea218efc407e0032 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 16:29:00 +0900 Subject: [PATCH 16/19] fix: status handling logic document Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 27 +++++++++++++---------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 00af63c19d..9b00ca99c8 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -50,29 +50,32 @@ The request may fail at the forwarding destination or the bypass destination. If some of the requests fails, the processing continues based on their status code. -The following is an overview of the process for each request. +Here's an overview of how the Mirror Gateway handles failures for each type of request: - Insert Request - - If the target host returns a status of `ALREADY_EXISTS`, the Update request is sent to this host. - - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. + - If the target host returns a status code of `ALREADY_EXISTS`, the Update request is sent to this host. + - If the target host returns a status code other than `OK`, `ALREADY_EXISTS`, the Mirror Gateway returns that status code without continuous processing. + - If all target hosts return a status code `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. + - If all target hosts return a status code `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Update Request - - If the target host returns a status of `NOT_FOUND`, the Insert request is sent to this host. - - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - - If all target hosts return a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. + - If the target host returns a status code `NOT_FOUND`, the Insert request is sent to this host. + - If the target host returns a status code other than `OK`, `ALREADY_EXISTS`, the Mirror Gateway returns that status code without continuous processing. + - If all target hosts return a status code `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. + - If all target hosts return a status code `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Upsert Request - - If all target hosts return a status of `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. - - If all target hosts return a status of `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. + - If all target hosts return a status code `ALREADY_EXISTS`, the Mirror Gateway returns `ALREADY_EXISTS`. + - If the target host returns a status code other than `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns that status code without continuous processing. + - If all target hosts return a status code `OK` or `ALREADY_EXISTS`, the Mirror Gateway returns `OK`. - Remove/RemoveByTimestamp Request - - If all target hosts return a status of `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND`. - - If all target hosts return a status of `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. + - If all target hosts return a status code `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND`. + - If the target host returns a status code other than `OK` or `NOT_FOUND`, the Mirror Gateway returns that status code without continuous processing. + - If all target hosts return a status code `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. For more information, please refer to [Mirror Gateway Troubleshooting](../../troubleshooting/mirror-gateway.md). From 48290a34f3d0cbd893ca331927d069ce8b142585 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 16:36:43 +0900 Subject: [PATCH 17/19] fix: document refactor Signed-off-by: hlts2 --- docs/overview/component/mirror-gateway.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/overview/component/mirror-gateway.md b/docs/overview/component/mirror-gateway.md index 9b00ca99c8..845ec1ed0b 100644 --- a/docs/overview/component/mirror-gateway.md +++ b/docs/overview/component/mirror-gateway.md @@ -50,7 +50,9 @@ The request may fail at the forwarding destination or the bypass destination. If some of the requests fails, the processing continues based on their status code. -Here's an overview of how the Mirror Gateway handles failures for each type of request: +Here's an overview of how the Mirror Gateway handles failures for each type of request. + +For more information about status code, please refer to [Mirror Gateway Troubleshooting](../../troubleshooting/mirror-gateway.md). - Insert Request @@ -77,5 +79,3 @@ Here's an overview of how the Mirror Gateway handles failures for each type of r - If all target hosts return a status code `NOT_FOUND`, the Mirror Gateway returns `NOT_FOUND`. - If the target host returns a status code other than `OK` or `NOT_FOUND`, the Mirror Gateway returns that status code without continuous processing. - If all target hosts return a status code `OK` or `NOT_FOUND`, the Mirror Gateway returns `OK`. - -For more information, please refer to [Mirror Gateway Troubleshooting](../../troubleshooting/mirror-gateway.md). From 94e35d4e8faaa118d0f500a520203866a6c27988 Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 16:41:33 +0900 Subject: [PATCH 18/19] fix: bugfix status handling document Signed-off-by: hlts2 --- docs/troubleshooting/mirror-gateway.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index 0e4aeb2706..ec34b5255e 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -40,10 +40,7 @@ Here are some common reasons of error. | ALREADY_EXISTS | Request a pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned to the following cases: - -- All hosts return `OK` or `NOT_FOUND`. -- All hosts return `OK` or `ALREADY_EXISTS`. +`0 (OK)` is also returned when all hosts return `OK` or `ALREADY_EXISTS`. ## Upsert Operation From ba053d7659bc676ca6fff24acc52b3c320c2b57e Mon Sep 17 00:00:00 2001 From: hlts2 Date: Mon, 16 Oct 2023 16:58:01 +0900 Subject: [PATCH 19/19] fix: refactor sentence Signed-off-by: hlts2 --- docs/troubleshooting/mirror-gateway.md | 28 +++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/troubleshooting/mirror-gateway.md b/docs/troubleshooting/mirror-gateway.md index ec34b5255e..eab467e302 100644 --- a/docs/troubleshooting/mirror-gateway.md +++ b/docs/troubleshooting/mirror-gateway.md @@ -17,11 +17,11 @@ Here are some common reasons of error. | CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | | INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | | DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| ALREADY_EXISTS | Request ID is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | +| ALREADY_EXISTS | Request ID is already inserted. This status code is returned when all target hosts return `ALREADY_EXISTS`. | | NOT_FOUND | Requested ID is NOT inserted. This is the status code of the Update request. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` or `ALREADY_EXISTS`. +`0 (OK)` is also returned when all target hosts return `OK` or `ALREADY_EXISTS`. ## Update Operation @@ -36,11 +36,11 @@ Here are some common reasons of error. | CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | | INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | | DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | +| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all target hosts return `NOT_FOUND`. | | ALREADY_EXISTS | Request a pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` or `ALREADY_EXISTS`. +`0 (OK)` is also returned when all target hosts return `OK` or `ALREADY_EXISTS`. ## Upsert Operation @@ -53,10 +53,10 @@ Here are some common reasons of error. | CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | | INVALID_ARGUMENT | The Dimension of the request vector is NOT the same as Vald Agent's config, the requested vector's ID is empty, or some request payload is invalid. | | DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| ALREADY_EXISTS | Requested pair of ID and vector is already inserted. This status code is returned when all hosts return `ALREADY_EXISTS`. | +| ALREADY_EXISTS | Requested pair of ID and vector is already inserted. This status code is returned when all target hosts return `ALREADY_EXISTS`. | | INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` or `ALREADY_EXISTS`. +`0 (OK)` is also returned when all target hosts return `OK` or `ALREADY_EXISTS`. ## Remove Operation @@ -64,12 +64,12 @@ The request process may not be completed when the response code is NOT `0 (OK)`. Here are some common reasons of error. -| name | common reason | -| :---------------- | :---------------------------------------------------------------------------------------------- | -| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | -| INVALID_ARGUMENT | The Requested vector's ID is empty, or some request payload is invalid. | -| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | -| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all hosts return `NOT_FOUND`. | -| INTERNAL | Target Vald cluster or network route has some critical error. | +| name | common reason | +| :---------------- | :--------------------------------------------------------------------------------------------------- | +| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | +| INVALID_ARGUMENT | The Requested vector's ID is empty, or some request payload is invalid. | +| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | +| NOT_FOUND | Requested ID is NOT inserted. This status code is returned when all target hosts return `NOT_FOUND`. | +| INTERNAL | Target Vald cluster or network route has some critical error. | -`0 (OK)` is also returned when all hosts return `OK` or `NOT_FOUND`. +`0 (OK)` is also returned when all target hosts return `OK` or `NOT_FOUND`.