Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add document for RemoveByTimestamp RPC #2238

Merged
merged 10 commits into from
Nov 24, 2023
Merged

Add document for RemoveByTimestamp RPC #2238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f7ae704
feat: add document for RemoveByTimestamp rpc
hlts2 Nov 21, 2023
64eefeb
fix: add comment to enum value
hlts2 Nov 21, 2023
9d94c99
fix: refactor document
hlts2 Nov 21, 2023
29b94da
fix: make proto/all format
hlts2 Nov 21, 2023
a9b7e9b
style: format code with Gofumpt and Prettier
deepsource-autofix[bot] Nov 21, 2023
f901b62
Merge branch 'main' into documentation/docs/add-rpc
hlts2 Nov 21, 2023
cd02e4f
feat: add notice section to removeByTimestamp arpc
hlts2 Nov 22, 2023
017e768
style: format code with Gofumpt and Prettier
deepsource-autofix[bot] Nov 22, 2023
b4d1db7
Update docs/api/remove.md
hlts2 Nov 24, 2023
3996a08
Merge branch 'main' into documentation/docs/add-rpc
hlts2 Nov 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions apis/docs/v1/docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@
- [Upsert.MultiRequest](#payload-v1-Upsert-MultiRequest)
- [Upsert.ObjectRequest](#payload-v1-Upsert-ObjectRequest)
- [Upsert.Request](#payload-v1-Upsert-Request)

- [Remove.Timestamp.Operator](#payload-v1-Remove-Timestamp-Operator)
- [Search.AggregationAlgorithm](#payload-v1-Search-AggregationAlgorithm)

Expand Down Expand Up @@ -971,6 +972,8 @@ Represent the remove request.

### Remove.Timestamp

Represent the timestamp comparison.

| Field | Type | Label | Description |
| --------- | ------------------------------------------------------------------ | ----- | ------------------------- |
| timestamp | [int64](#int64) | | The timestamp. |
Expand All @@ -982,9 +985,9 @@ Represent the remove request.

Represent the remove request based on timestamp.

| Field | Type | Label | Description |
| ---------- | ------------------------------------------------ | -------- | ------------------------------------------------------------------ |
| timestamps | [Remove.Timestamp](#payload-v1-Remove-Timestamp) | repeated | Represent the multiple remove request contents based on timestamp. |
| Field | Type | Label | Description |
| ---------- | ------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------ |
| timestamps | [Remove.Timestamp](#payload-v1-Remove-Timestamp) | repeated | The timestamp comparison list. If more than one is specified, the `AND` search is applied. |

<a name="payload-v1-Search"></a>

Expand Down Expand Up @@ -1236,14 +1239,14 @@ Represent the upsert request.

Operator is enum of each conditional operator.

| Name | Number | Description |
| ---- | ------ | ----------- |
| Eq | 0 | |
| Ne | 1 | |
| Ge | 2 | |
| Gt | 3 | |
| Le | 4 | |
| Lt | 5 | |
| Name | Number | Description |
| ---- | ------ | ----------------------------------------------------------------------------- |
| Eq | 0 | The timestamp is equal to the specified value in the request. |
| Ne | 1 | The timestamp is not equal to the specified value in the request. |
| Ge | 2 | The timestamp is greater than or equal to the specified value in the request. |
| Gt | 3 | The timestamp is greater than the specified value in the request. |
| Le | 4 | The timestamp is less than or equal to the specified value in the request. |
| Lt | 5 | The timestamp is less than the specified value in the request. |

<a name="payload-v1-Search-AggregationAlgorithm"></a>

Expand Down
9 changes: 8 additions & 1 deletion apis/grpc/v1/payload/payload.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion apis/proto/v1/payload/payload.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,18 +300,25 @@ message Remove {

// Represent the remove request based on timestamp.
message TimestampRequest {
// Represent the multiple remove request contents based on timestamp.
// The timestamp comparison list. If more than one is specified, the `AND` search is applied.
repeated Timestamp timestamps = 1;
}

// Represent the timestamp comparison.
message Timestamp {
// Operator is enum of each conditional operator.
enum Operator {
// The timestamp is equal to the specified value in the request.
Eq = 0;
// The timestamp is not equal to the specified value in the request.
Ne = 1;
// The timestamp is greater than or equal to the specified value in the request.
Ge = 2;
// The timestamp is greater than the specified value in the request.
Gt = 3;
// The timestamp is less than or equal to the specified value in the request.
Le = 4;
// The timestamp is less than the specified value in the request.
Lt = 5;
}
// The timestamp.
Expand Down
7 changes: 4 additions & 3 deletions apis/swagger/v1/vald/apis/proto/v1/vald/remove.swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@
"items": {
"$ref": "#/definitions/v1RemoveTimestamp"
},
"description": "Represent the multiple remove request contents based on timestamp."
"description": "The timestamp comparison list. If more than one is specified, the `AND` search is applied."
}
},
"description": "Represent the remove request based on timestamp."
Expand All @@ -154,7 +154,7 @@
"type": "string",
"enum": ["Eq", "Ne", "Ge", "Gt", "Le", "Lt"],
"default": "Eq",
"description": "Operator is enum of each conditional operator."
"description": "Operator is enum of each conditional operator.\n\n - Eq: The timestamp is equal to the specified value in the request.\n - Ne: The timestamp is not equal to the specified value in the request.\n - Ge: The timestamp is greater than or equal to the specified value in the request.\n - Gt: The timestamp is greater than the specified value in the request.\n - Le: The timestamp is less than or equal to the specified value in the request.\n - Lt: The timestamp is less than the specified value in the request."
},
"protobufAny": {
"type": "object",
Expand Down Expand Up @@ -314,7 +314,8 @@
"$ref": "#/definitions/TimestampOperator",
"description": "The conditional operator."
}
}
},
"description": "Represent the timestamp comparison."
}
}
}
122 changes: 122 additions & 0 deletions docs/api/remove.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ service Remove {

rpc Remove(payload.v1.Remove.Request) returns (payload.v1.Object.Location) {}

rpc RemoveByTimestamp(payload.v1.Remove.TimestampRequest) returns (payload.v1.Object.Locations) {}
hlts2 marked this conversation as resolved.
Show resolved Hide resolved

rpc StreamRemove(stream payload.v1.Remove.Request)
returns (stream payload.v1.Object.StreamLocation) {}

Expand Down Expand Up @@ -114,6 +116,126 @@ Here are some common reasons and how to resolve each error.
| NOT_FOUND | Requested ID is NOT inserted. | Send a request with an ID that is already inserted. |
| INTERNAL | Target Vald cluster or network route has some critical error. | Check target Vald cluster first and check network route including ingress as second. |

## RemoveByTimestamp RPC

RemoveByTimestamp RPC is the method to remove vectors based on timestamp.

### Input

- the scheme of `payload.v1.Remove.TimestampRequest`

```rpc
message Remove {
message TimestampRequest {
repeated Timestamp timestamps = 1;
}

message Timestamp {
enum Operator {
Eq = 0;
Ne = 1;
Ge = 2;
Gt = 3;
Le = 4;
Lt = 5;
}
int64 timestamp = 1;
Operator operator = 2;
hlts2 marked this conversation as resolved.
Show resolved Hide resolved
}
}

message Object {
message ID {
string id = 1 [ (validate.rules).string.min_len = 1 ];
}
}
```

- Remove.TimestampRequest

| field | type | label | required | description |
| :--------: | :--------------- | :-------------------------------- | :------: | :-------------------------------------------------------------------------------------------- |
| timestamps | Remove.Timestamp | repeated(Array[Remove.Timestamp]) | \* | The timestamp comparison list.<br>If more than one is specified, the `AND` search is applied. |

- Remove.Timestamp

| field | type | label | required | description |
| :-------: | :------------------------ | :---- | :------: | :------------------------------------------------ |
| timestamp | int64 | | \* | The timestamp. |
| operator | Remove.Timestamp.Operator | | | The conditionl operator. (default value is `Eq`). |

- Remove.Timestamp.Operator

| value | description |
| :---: | :--------------------- |
| Eq | Equal. |
| Ne | Not Equal. |
| Ge | Greater than or Equal. |
| Gt | Greater than. |
| Le | Less than or Equal. |
| Lt | Less than. |

<div class="notice">
In the TimestampRequest message, the 'timestamps' field is repeated, allowing the inclusion of multiple Timestamp.<br>
When multiple Timestamps are provided, it results in an `AND` condition, enabling the realization of deletions with specified ranges.<br>
This design allows for versatile deletion operations, facilitating tasks such as removing data within a specific time range.
</div>

### Output

- the scheme of `payload.v1.Object.Locations`.

```rpc
message Object {
message Locations { repeated Location locations = 1; }

message Location {
string name = 1;
string uuid = 2;
repeated string ips = 3;
}
}
```

- Object.Locations

| field | type | label | description |
| :------: | :-------------- | :------------------------------- | :----------------------------- |
| location | Object.Location | repeated(Array[Object.Location]) | The list of `Object.Location`. |

- Object.Location

| field | type | label | description |
| :---: | :----- | :---------------------- | :-------------------------------------------------------------------- |
| name | string | | The name of vald agent pod where the request vector is removed. |
| uuid | string | | The ID of the removed vector. It is the same as an `Object.ID`. |
| ips | string | repeated(Array[string]) | The IP list of `vald-agent` pods where the request vector is removed. |

### Status Code

| code | name |
| :--: | :---------------- |
| 0 | OK |
| 1 | CANCELLED |
| 4 | DEADLINE_EXCEEDED |
| 5 | NOT_FOUND |
| 13 | INTERNAL |

Please refer to [Response Status Code](./status.md) for more details.

### Troubleshooting

The request process may not be completed when the response code is NOT `0 (OK)`.

Here are some common reasons and how to resolve each error.

| name | common reason | how to resolve |
| :---------------- | :---------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- |
| CANCELLED | Executed cancel() of rpc from client/server-side or network problems between client and server. | Check the code, especially around timeout and connection management, and fix if needed. |
| DEADLINE_EXCEEDED | The RPC timeout setting is too short on the client/server side. | Check the gRPC timeout setting on both the client and server sides and fix it if needed. |
| NOT_FOUND | No vectors in the system match the specified timestamp conditions. | Check whether vectors matching the specified timestamp conditions exist in the system, and fix conditions if needed. |
| INTERNAL | Target Vald cluster or network route has some critical error. | Check target Vald cluster first and check network route including ingress as second. |

## StreamRemove RPC

StreamRemove RPC is the method to remove multiple vectors using the [bidirectional streaming RPC](https://grpc.io/docs/what-is-grpc/core-concepts/#bidirectional-streaming-rpc).<br>
Expand Down
Loading