diff --git a/spiceaidocs/content/en/reference/Connectors/_index.md b/spiceaidocs/content/en/reference/Connectors/_index.md new file mode 100644 index 00000000..5f827595 --- /dev/null +++ b/spiceaidocs/content/en/reference/Connectors/_index.md @@ -0,0 +1,9 @@ +--- +type: docs +title: "Data Connectors" +linkTitle: "Data Connectors" +description: '' +weight: 70 +--- + + diff --git a/spiceaidocs/content/en/reference/Connectors/s3.md b/spiceaidocs/content/en/reference/Connectors/s3.md new file mode 100644 index 00000000..2e3408ba --- /dev/null +++ b/spiceaidocs/content/en/reference/Connectors/s3.md @@ -0,0 +1,42 @@ +--- +type: docs +title: "S3 Data Connector" +linkTitle: "S3 Data Connector" +description: 'S3 Data Connector YAML reference' +weight: 80 +--- + +S3 as a connector for federated SQL query across Parquet files stored in S3, or S3-compatible storage solutions (e.g. Minio, Cloudflare R2). + +## `params` + +- `endpoint`: The S3 endpoint, or equivalent (e.g. Minio endpoint), for the S3-compatible storage. +- `region`: Region of the S3 bucket, if region specific. + +## `auth` + +Check [Secrets]({{}}). + +Required attribbutes: +- `key`: The access key authorised to access the S3 data (e.g. `AWS_ACCESS_KEY_ID` for AWS) +- `secret`The secret key authorised to access the S3 data (e.g. `AWS_SECRET_ACCESS_KEY` for AWS) + + +## Example +### Minio +```yaml +- from: s3://s3-bucket-name/path/to/parquet/cool_dataset.parquet + name: cool_dataset + params: + endpoint: https://my.minio.server + region: "us-east-1" # Best practice for Minio +``` + +#### S3 +```yaml +- from: s3://my-startups-data/path/to/parquet/cool_dataset.parquet + name: cool_dataset + params: + endpoint: http://my-startups-data.s3.amazonaws.com + region: "ap-southeast-2" +``` \ No newline at end of file