Documentation for HTTP(s) connector. (#302)

* add http docs * link HTTP doc * Update https.md * update HTTP params * PR comments
spiceai · Jul 4, 2024 · 8c70431 · 8c70431
1 parent 055e3ff
commit 8c70431
Show file tree

Hide file tree

Showing 3 changed files with 60 additions and 15 deletions.
diff --git a/spiceaidocs/docs/data-connectors/https.md b/spiceaidocs/docs/data-connectors/https.md
@@ -0,0 +1,43 @@
+---
+title: 'HTTP(s) Data Connector'
+sidebar_label: 'HTTP(s) Data Connector'
+description: 'HTTP(s) Data Connector Documentation'
+pagination_prev: null
+---
+
+The HTTP(s) Data Connector enables federated SQL query against a variety of tabular formatted (e.g. Parquet/CSV) files stored at a HTTP endpoint.
+
+The connector supports HTTP authentication via either `param` and/or `secrets`.
+
+### Parameters
+ - `http_port`: Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.
+ - `http_username`: Optional. Username to provide connection for HTTP basic authentication. Default: None.
+ - `http_password`: Optional. Password to provide connection for HTTP basic authentication. Default: None.
+ - `http_password_key`: Key of the secret that contains the value to use for `http_password`. Default: None.
+
+### Examples
+```yaml
+datasets:
+  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
+    name: laion_freesound
+
+  - from: http://static_username@localhost:3001/report.csv
+    name: local_report
+    params:
+      http_password: BadPa$5w0rd
+```
+
+To use a secret for the HTTP password, for example, by env variable
+```yaml
+datasets:
+  - from: http://static_username@localhost/report.csv
+    name: local_report
+    params:
+      http_password_key: local_password
+      http_port: 3001
+```
+
+With the associated secret set, for example:
+```shell
+export SPICE_SECRET_HTTP_LOCAL_PASSWORD="BadPa$5w0rd"
+```
diff --git a/spiceaidocs/docs/data-connectors/index.md b/spiceaidocs/docs/data-connectors/index.md
@@ -11,21 +11,22 @@ Data Connectors provide connections to databases, data warehouses, and data lake
 
 Currently supported Data Connectors include:
 
-| Name         | Description | Status       | Protocol/Format                     | Refresh Modes    | Supports Inserts |
-| ------------ | ----------- | ------------ | ----------------------------------- | ---------------- | ---------------- |
-| `databricks` | Databricks  | Alpha        | Spark Connect <br/> S3 / Delta Lake | `full`           | ❌               |
-| `postgres`   | PostgreSQL  | Alpha        |                                     | `full`           | ❌               |
-| `spiceai`    | Spice.ai    | Alpha        | Arrow Flight                        | `append`, `full` | ✅               |
-| `s3`         | S3          | Alpha        | Parquet, CSV                        | `full`           | ❌               |
-| `dremio`     | Dremio      | Alpha        | Arrow Flight SQL                    | `full`           | ❌               |
-| `flightsql`  | FlightSQL   | Alpha        | Arrow Flight SQL                    | `full`           | ❌               |
-| `snowflake`  | Snowflake   | Alpha        | Arrow                               | `full`           | ❌               |
-| `mysql`      | MySQL       | Alpha        |                                     | `full`           | ❌               |
-| `clickhouse` | Clickhouse  | Alpha        |                                     | `full`           | ❌               |
-| `spark`      | Spark       | Alpha        | Spark Connect                       | `full`           | ❌               |
-| `ftp`, `sftp`| FTP/SFTP    | Alpha        | Parquet, CSV                        | `full`           | ❌               |
-| `graphql`    | GraphQL     | Alpha        | GraphQL                             | `full`           | ❌               |
-| `odbc`       | ODBC        | Alpha        | ODBC                                | `full`           | ❌               |
+| Name            | Description | Status       | Protocol/Format                     | Refresh Modes    | Supports Inserts |
+| --------------- | ----------- | ------------ | ----------------------------------- | ---------------- | ---------------- |
+| `databricks`    | Databricks  | Alpha        | Spark Connect <br/> S3 / Delta Lake | `full`           | ❌               |
+| `postgres`      | PostgreSQL  | Alpha        |                                     | `full`           | ❌               |
+| `spiceai`       | Spice.ai    | Alpha        | Arrow Flight                        | `append`, `full` | ✅               |
+| `s3`            | S3          | Alpha        | Parquet, CSV                        | `full`           | ❌               |
+| `dremio`        | Dremio      | Alpha        | Arrow Flight SQL                    | `full`           | ❌               |
+| `flightsql`     | FlightSQL   | Alpha        | Arrow Flight SQL                    | `full`           | ❌               |
+| `snowflake`     | Snowflake   | Alpha        | Arrow                               | `full`           | ❌               |
+| `mysql`         | MySQL       | Alpha        |                                     | `full`           | ❌               |
+| `clickhouse`    | Clickhouse  | Alpha        |                                     | `full`           | ❌               |
+| `spark`         | Spark       | Alpha        | Spark Connect                       | `full`           | ❌               |
+| `ftp`, `sftp`   | FTP/SFTP    | Alpha        | Parquet, CSV                        | `full`           | ❌               |
+| `graphql`       | GraphQL     | Alpha        | GraphQL                             | `full`           | ❌               |
+| `odbc`          | ODBC        | Alpha        | ODBC                                | `full`           | ❌               |
+| `http`, `https` | HTTP(s)     | Alpha        | Parquet, CSV                        | `full`           | ❌               |
 
 ## Data Connector Docs
 

diff --git a/spiceaidocs/docs/reference/spicepod/datasets.md b/spiceaidocs/docs/reference/spicepod/datasets.md
@@ -86,6 +86,7 @@ Where:
   - [`flightsql`](../../data-connectors/flightsql.md)
   - [`snowflake`](../../data-connectors/snowflake.md)
   - [`ftp`, `sftp`](../../data-connectors/ftp.md)
+  - [`http`, `https`](../../data-connectors/https.md)
   - [`clickhouse`](../../data-connectors/clickhouse.md)
   - [`graphql`](../../data-connectors/graphql.md)