From 7ab31211e9298e2072d70ea91d1f7d692d91f807 Mon Sep 17 00:00:00 2001 From: peasee <98815791+peasee@users.noreply.github.com> Date: Tue, 12 Nov 2024 11:47:37 +1000 Subject: [PATCH 1/2] docs: Release DuckDB RC --- .../docs/components/data-connectors/duckdb.md | 2 ++ .../docs/components/data-connectors/index.md | 2 +- .../docs/reference/spicepod/datasets.md | 22 +++++++++++++++++-- 3 files changed, 23 insertions(+), 3 deletions(-) diff --git a/spiceaidocs/docs/components/data-connectors/duckdb.md b/spiceaidocs/docs/components/data-connectors/duckdb.md index 26f9a1b1..cfb6166e 100644 --- a/spiceaidocs/docs/components/data-connectors/duckdb.md +++ b/spiceaidocs/docs/components/data-connectors/duckdb.md @@ -24,6 +24,8 @@ The DuckDB data connector can be configured by providing the following `params`: Configuration `params` are provided either in the top level `dataset` for a dataset source, or in the `acceleration` section for a data store. +The DuckDB data connector supports specifying an [`invalid_type_action` dataset parameter](../../reference/spicepod/datasets.md#invalid_type_action), modifying the behavior of the Runtime when a data type the connector does not support is encountered. + A generic example of DuckDB data connector configuration. ```yaml diff --git a/spiceaidocs/docs/components/data-connectors/index.md b/spiceaidocs/docs/components/data-connectors/index.md index 64a8bdfc..b631482d 100644 --- a/spiceaidocs/docs/components/data-connectors/index.md +++ b/spiceaidocs/docs/components/data-connectors/index.md @@ -13,6 +13,7 @@ Currently supported Data Connectors include: | Name | Description | Status | Protocol/Format | Refresh Modes | Supports [Ingestion][ingestion] | Supports Documents | | --------------- | ------------------------- | ----------------- | ----------------------------------- | --------------------------- | ------------------------------- | ------------------ | +| `duckdb` | DuckDB | Release Candidate | | `append`, `full` | ❌ | ❌ | | `github` | GitHub | Release Candidate | GraphQL, REST | `append`, `full` | ❌ | ❌ | | `mysql` | MySQL | Release Candidate | | `append`, `full` | Roadmap | ❌ | | `postgres` | PostgreSQL | Release Candidate | | `append`, `full` | Roadmap | ❌ | @@ -26,7 +27,6 @@ Currently supported Data Connectors include: | `clickhouse` | Clickhouse | Alpha | | `append`, `full` | ❌ | ❌ | | `debezium` | Debezium | Alpha | CDC, Kafka | `append`, `full`, `changes` | ❌ | ❌ | | `dremio` | Dremio | Alpha | Arrow Flight SQL | `append`, `full` | ❌ | ❌ | -| `duckdb` | DuckDB | Alpha | | `append`, `full` | ❌ | ❌ | | `file` | File | Alpha | Parquet, CSV | `append`, `full` | Roadmap | ✅ | | `ftp`, `sftp` | FTP/SFTP | Alpha | Parquet, CSV | `append`, `full` | ❌ | ✅ | | `graphql` | GraphQL | Alpha | GraphQL | `append`, `full` | ❌ | ❌ | diff --git a/spiceaidocs/docs/reference/spicepod/datasets.md b/spiceaidocs/docs/reference/spicepod/datasets.md index 3d61485a..32b94409 100644 --- a/spiceaidocs/docs/reference/spicepod/datasets.md +++ b/spiceaidocs/docs/reference/spicepod/datasets.md @@ -144,6 +144,23 @@ Spice emits a warning if the `time_column` from the data source is incompatible :::warning[Limitations] - String-based columns are assumed to be ISO8601 format. + +::: + +## `invalid_type_action` + +Optional. Specifies the action to take when a data type that is not supported by the data connector is encountered. + +The following values are supported: + +- `error` - Default. Return an error when an unsupported data type is encountered. +- `warn` - Log a warning and ignore the column containing the unsupported data type. +- `ignore` - Log nothing and ignore the column containing the unsupported data type. + +:::warning[Limitations] + +Not all connectors support specifying an `invalid_type_action`. When specified on a connector that does not support the option, the connector will fail to register. + ::: ## `acceleration` @@ -196,6 +213,7 @@ Must be of the form `SELECT * FROM {name} WHERE {refresh_filter}`. `{name}` is t - The refresh SQL only supports filtering data from the current dataset - joining across other datasets is not supported. - Selecting a subset of columns isn't supported - the refresh SQL needs to start with `SELECT * FROM {name}`. - Queries for data that have been filtered out will not fall back to querying against the federated table. + ::: ## `acceleration.refresh_data_window` @@ -230,8 +248,8 @@ Optional. Defines the maximum number of retry attempts when refresh retries are Supports one of two values: -* `on_registration`: Mark the dataset as ready immediately, and queries on this table will fall back to the underlying source directly until the initial acceleration is complete -* `on_load`: Mark the dataset as ready only after the initial acceleration. Queries against the dataset will return an error before the load has been completed. +- `on_registration`: Mark the dataset as ready immediately, and queries on this table will fall back to the underlying source directly until the initial acceleration is complete +- `on_load`: Mark the dataset as ready only after the initial acceleration. Queries against the dataset will return an error before the load has been completed. ```yaml datasets: From e7e9920192d933b583ace51491c218bc7297a9cd Mon Sep 17 00:00:00 2001 From: peasee <98815791+peasee@users.noreply.github.com> Date: Tue, 12 Nov 2024 12:28:40 +1000 Subject: [PATCH 2/2] docs: Include list of supported invalid_type_action connectors --- spiceaidocs/docs/reference/spicepod/datasets.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/spiceaidocs/docs/reference/spicepod/datasets.md b/spiceaidocs/docs/reference/spicepod/datasets.md index 32b94409..8e18e6cb 100644 --- a/spiceaidocs/docs/reference/spicepod/datasets.md +++ b/spiceaidocs/docs/reference/spicepod/datasets.md @@ -159,7 +159,9 @@ The following values are supported: :::warning[Limitations] -Not all connectors support specifying an `invalid_type_action`. When specified on a connector that does not support the option, the connector will fail to register. +Not all connectors support specifying an `invalid_type_action`. When specified on a connector that does not support the option, the connector will fail to register. The following connectors support `invalid_type_action`: + +- [DuckDB](../../components/data-connectors/duckdb.md) :::