From d06d9ed88fd9812a6bc41f3affcd2e48dc7b6969 Mon Sep 17 00:00:00 2001 From: Kirill Khramkov Date: Tue, 14 May 2024 14:47:37 +0400 Subject: [PATCH 1/4] Add timeout to params --- spiceaidocs/docs/data-connectors/s3.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/spiceaidocs/docs/data-connectors/s3.md b/spiceaidocs/docs/data-connectors/s3.md index 25df2e2d..57656c2f 100644 --- a/spiceaidocs/docs/data-connectors/s3.md +++ b/spiceaidocs/docs/data-connectors/s3.md @@ -31,13 +31,14 @@ Example: `name: cool_dataset` - `endpoint`: The S3 endpoint, or equivalent (e.g. MinIO endpoint), for the S3-compatible storage. E.g. `endpoint: https://my.minio.server` - `region`: Region of the S3 bucket, if region specific. E.g. `region: us-east-1` +- `timeout`: Timeout for S3 operations. E.g. `timeout: 60s` ## Auth Not required for public endpoints. - `key`: The access key (e.g. `AWS_ACCESS_KEY_ID` for AWS) -- `secret`The secret key (e.g. `AWS_SECRET_ACCESS_KEY` for AWS) +- `secret`: The secret key (e.g. `AWS_SECRET_ACCESS_KEY` for AWS) For endpoints protected by access keys, `key` and `secret` are required and must be passed using a [Secrets Store](/secret-stores) or via `spice s3 login`. By default S3 connector will look for a secret named `s3` with keys `key` and `secret`. @@ -67,7 +68,7 @@ Support for dataset specific authentication is on the roadmap. secrets: store: env - + # <...> ``` @@ -88,7 +89,7 @@ Support for dataset specific authentication is on the roadmap. secrets: store: kubernetes - + # <...> ``` @@ -111,7 +112,7 @@ Support for dataset specific authentication is on the roadmap. secrets: store: keyring - + # <...> ``` From ab35a89bcd39a620c8ff4c43198cf645e91301ff Mon Sep 17 00:00:00 2001 From: Kirill Khramkov Date: Fri, 17 May 2024 15:04:21 +0400 Subject: [PATCH 2/4] Specify default values --- spiceaidocs/docs/data-connectors/s3.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/spiceaidocs/docs/data-connectors/s3.md b/spiceaidocs/docs/data-connectors/s3.md index 57656c2f..701cb458 100644 --- a/spiceaidocs/docs/data-connectors/s3.md +++ b/spiceaidocs/docs/data-connectors/s3.md @@ -29,9 +29,9 @@ Example: `name: cool_dataset` ### `params` (optional) -- `endpoint`: The S3 endpoint, or equivalent (e.g. MinIO endpoint), for the S3-compatible storage. E.g. `endpoint: https://my.minio.server` -- `region`: Region of the S3 bucket, if region specific. E.g. `region: us-east-1` -- `timeout`: Timeout for S3 operations. E.g. `timeout: 60s` +- `endpoint`: The S3 endpoint, or equivalent (e.g. MinIO endpoint), for the S3-compatible storage. Defaults to region endpoint. E.g. `endpoint: https://my.minio.server` +- `region`: Region of the S3 bucket, if region specific. Default value is `us-east-1` E.g. `region: us-east-1` +- `timeout`: Specifies timeout for S3 operations. Default value is `30s` E.g. `timeout: 60s` ## Auth From b264e9d94651441a03af8a83354b4e380a1872fb Mon Sep 17 00:00:00 2001 From: Kirill Khramkov Date: Fri, 17 May 2024 15:06:42 +0400 Subject: [PATCH 3/4] Add file_format paramter --- spiceaidocs/docs/data-connectors/s3.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/spiceaidocs/docs/data-connectors/s3.md b/spiceaidocs/docs/data-connectors/s3.md index 701cb458..9a731ec7 100644 --- a/spiceaidocs/docs/data-connectors/s3.md +++ b/spiceaidocs/docs/data-connectors/s3.md @@ -29,6 +29,9 @@ Example: `name: cool_dataset` ### `params` (optional) +- `file_format`: Specifies the requested file format. Default is `parquet`. + - `parquet`: (default) Parquet file format. + - `csv`: CSV file format. - `endpoint`: The S3 endpoint, or equivalent (e.g. MinIO endpoint), for the S3-compatible storage. Defaults to region endpoint. E.g. `endpoint: https://my.minio.server` - `region`: Region of the S3 bucket, if region specific. Default value is `us-east-1` E.g. `region: us-east-1` - `timeout`: Specifies timeout for S3 operations. Default value is `30s` E.g. `timeout: 60s` From 1da699bcef8e9ef698c09d1351b67467fadf8e85 Mon Sep 17 00:00:00 2001 From: Kirill Khramkov Date: Fri, 17 May 2024 15:07:43 +0400 Subject: [PATCH 4/4] Update table --- spiceaidocs/docs/data-connectors/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spiceaidocs/docs/data-connectors/index.md b/spiceaidocs/docs/data-connectors/index.md index 2794b71c..0b2c9d86 100644 --- a/spiceaidocs/docs/data-connectors/index.md +++ b/spiceaidocs/docs/data-connectors/index.md @@ -16,7 +16,7 @@ Currently supported Data Connectors include: | `databricks` | Databricks | Alpha | Spark Connect
S3 / Delta Lake | `full` | ❌ | | `postgres` | PostgreSQL | Alpha | | `full` | ❌ | | `spiceai` | Spice.ai | Alpha | Arrow Flight | `append`, `full` | ✅ | -| `s3` | S3 | Alpha | Parquet | `full` | ❌ | +| `s3` | S3 | Alpha | Parquet, CSV | `full` | ❌ | | `dremio` | Dremio | Alpha | Arrow Flight SQL | `full` | ❌ | | `flightsql` | FlightSQL | Alpha | Arrow Flight SQL | `full` | ❌ | | `snowflake` | Snowflake | Alpha | Arrow | `full` | ❌ |