From 5626e9e7a0bb339ded592ab6d30e3d6c6b27a4ae Mon Sep 17 00:00:00 2001 From: Phillip LeBlanc Date: Tue, 19 Nov 2024 01:51:04 +0900 Subject: [PATCH] Move `ready_state` to the dataset level (#637) --- .../data-accelerators/data-refresh.md | 4 +-- .../docs/reference/spicepod/datasets.md | 34 +++++++++---------- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/spiceaidocs/docs/components/data-accelerators/data-refresh.md b/spiceaidocs/docs/components/data-accelerators/data-refresh.md index 71f0f68b..184c382f 100644 --- a/spiceaidocs/docs/components/data-accelerators/data-refresh.md +++ b/spiceaidocs/docs/components/data-accelerators/data-refresh.md @@ -54,7 +54,7 @@ Datasets configured with acceleration `refresh_mode: changes` require a [Change By default, Spice will return an error for queries against an accelerated dataset that is still loading its initial data. The endpoint [`/v1/ready`](/api/http/ready) is used in production deployments to control when queries are sent to the Spice runtime. -The ready state for an accelerated dataset can be configured using the [`ready_state`](/reference/spicepod/datasets#accelerationready_state) parameter in the acceleration configuration. +The ready state for an accelerated dataset can be configured using the [`ready_state`](/reference/spicepod/datasets#ready_state) parameter in the dataset configuration. - `ready_state: on_load`: Default. The dataset is considered ready after the initial load of the accelerated data. For file-based accelerated datasets that have existing data, this will be ready immediately. Queries against this dataset before the data is loaded will return an error. - `ready_state: on_registration`: The dataset is considered ready when the dataset is registered in Spice, even before the initial data is loaded. Queries against this dataset before the data is loaded will automatically fallback to the federated source. Once the data is loaded, queries will be served from the acceleration. @@ -65,9 +65,9 @@ Example: datasets: - from: s3://my_bucket/my_dataset name: my_dataset + ready_state: on_load # or on_registration acceleration: enabled: true - ready_state: on_load # or on_registration ``` ## Filtered Refresh diff --git a/spiceaidocs/docs/reference/spicepod/datasets.md b/spiceaidocs/docs/reference/spicepod/datasets.md index 72c037ef..28ca68e7 100644 --- a/spiceaidocs/docs/reference/spicepod/datasets.md +++ b/spiceaidocs/docs/reference/spicepod/datasets.md @@ -167,6 +167,23 @@ Not all connectors support specifying an `invalid_type_action`. When specified o ::: +## `ready_state` + +Supports one of two values: + +- `on_registration`: Mark the dataset as ready immediately, and queries on this table will fall back to the underlying source directly until the initial acceleration is complete +- `on_load`: Mark the dataset as ready only after the initial acceleration. Queries against the dataset will return an error before the load has been completed. + +```yaml +datasets: + - from: s3://my_bucket/my_dataset/ + name: my_dataset + ready_state: on_registration # or on_load + params: ... + acceleration: + enabled: true +``` + ## `acceleration` Optional. Accelerate queries to the dataset by caching data locally. @@ -248,23 +265,6 @@ Retries utilize a [Fibonacci backoff strategy](https://en.wikipedia.org/wiki/Fib Optional. Defines the maximum number of retry attempts when refresh retries are enabled. The default is undefined, allowing for unlimited attempts. -## `acceleration.ready_state` - -Supports one of two values: - -- `on_registration`: Mark the dataset as ready immediately, and queries on this table will fall back to the underlying source directly until the initial acceleration is complete -- `on_load`: Mark the dataset as ready only after the initial acceleration. Queries against the dataset will return an error before the load has been completed. - -```yaml -datasets: - - from: s3://my_bucket/my_dataset/ - name: my_dataset - params: ... - acceleration: - enabled: true - ready_state: on_registration # or on_load -``` - ## `acceleration.params` Optional. Parameters to pass to the acceleration engine. The parameters are specific to the acceleration engine used.