Skip to content

Commit

Permalink
Move refresh interval and retention info to the data refresh main page (
Browse files Browse the repository at this point in the history
  • Loading branch information
sgrebnov authored May 15, 2024
1 parent 83a4173 commit 0a7622b
Show file tree
Hide file tree
Showing 2 changed files with 43 additions and 37 deletions.
42 changes: 42 additions & 0 deletions spiceaidocs/docs/data-accelerators/data-refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,3 +146,45 @@ In this example a query against `accelerated_dataset` within Spice like `SELECT
:::warning
It is possible that even though the accelerated table returns some results, it may not contain all the data that would be returned by the federated table. `on_zero_results` only controls the behavior in the simple case where no data is returned by the acceleration for a given query.
:::

## Refresh Interval

For accelerated datasets in `full` mode, the [`refresh_check_interval`](/reference/spicepod/datasets#accelerationrefresh_check_interval) parameter controls how often the accelerated dataset is refreshed.

Example:

```yaml
datasets:
- from: spice.ai/eth.recent_blocks
name: eth_recent_blocks
acceleration:
enabled: true
refresh_mode: full
refresh_check_interval: 10s
```

This configuration will refresh `eth.recent_blocks` data every 10 seconds.

Accelerated datasets can also be refreshed on-demand via the `refresh` CLI command or `POST /v1/datasets/:name/acceleration/refresh` API endpoint.

An example using cURL:

```bash
curl -i -XPOST 127.0.0.1:3000/v1/datasets/eth_recent_blocks/refresh
```

And response

```bash
HTTP/1.1 201 Created
content-type: application/json
content-length: 55
date: Thu, 11 Apr 2024 20:11:18 GMT
{"message":"Dataset refresh triggered for eth_recent_blocks."}
```
## Retention Policy

A retention policy automatically removes data from accelerated datasets with a temporal column that exceeds the defined retention period, optimizing resource utilization.

The policy is set using the [`acceleration.retention_check_enabled`](/reference/spicepod/datasets#accelerationretention_check_enabled), [`acceleration.retention_period`](/reference/spicepod/datasets#accelerationretention_period) and [`acceleration.retention_check_interval`](/reference/spicepod/datasets#accelerationretention_check_interval) parameters, along with the [`time_column`](/reference/spicepod/datasets#time_column) and [`time_format`](/reference/spicepod/datasets#time_format) dataset parameters.
38 changes: 1 addition & 37 deletions spiceaidocs/docs/features/local-acceleration/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,40 +23,6 @@ Data Storage: Ensure that the local storage has enough capacity to store the acc

Data Security: Assess data sensitivity and secure network connections between edge and data connector when replicating data for further usage. Assess the security of any Data Accelerator that is external to the Spice runtime and connected to the Spice runtime. Implement encryption, access controls, and secure protocols.

## Refresh Modes

Dataset acceleration can be configured in `full` (the entire dataset is refreshed) or `append` (new data from a dataset source is appended) modes.

## Refresh Interval

For accelerated datasets in `full` mode, the [`refresh_check_interval`](/reference/spicepod/datasets#accelerationrefresh_check_interval) parameter controls how often the accelerated dataset is refreshed.

Accelerated datasets can also be refreshed on-demand via the `refresh` CLI command or `POST /v1/datasets/:name/acceleration/refresh` API endpoint.

An example using cURL:

```bash
curl -i -XPOST 127.0.0.1:3000/v1/datasets/eth_recent_blocks/refresh
```

And response

```bash
HTTP/1.1 201 Created
content-type: application/json
content-length: 55
date: Thu, 11 Apr 2024 20:11:18 GMT

{"message":"Dataset refresh triggered for eth_recent_blocks."}
```

## Retention Policy

A retention policy automatically removes data from accelerated datasets with a temporal column that exceeds the defined retention period, optimizing resource utilization.

The policy is set using the [`acceleration.retention_check_enabled`](/reference/spicepod/datasets#accelerationretention_check_enabled), [`acceleration.retention_period`](/reference/spicepod/datasets#accelerationretention_period) and [`acceleration.retention_check_interval`](/reference/spicepod/datasets#accelerationretention_check_interval) parameters, along with the [`time_column`](/reference/spicepod/datasets#time_column) and [`time_format`](/reference/spicepod/datasets#time_format) dataset parameters.


## Example

### Locally Accelerating eth.recent_blocks
Expand Down Expand Up @@ -89,6 +55,4 @@ spice sql
select * from eth_recent_blocks
```

## Limitations

- Currently, only official [Data Accelerators](../data-accelerators) are supported for local acceleration.
[Learn more about Data Accelerators](/data-accelerators) for faster access.

0 comments on commit 0a7622b

Please sign in to comment.