Skip to content

Commit

Permalink
Add documentation for spice login abfs (#675)
Browse files Browse the repository at this point in the history 
* Add documentation for `spice login abfs`

* fix

* fix
  • Loading branch information
@Sevenannn
Sevenannn authored Dec 13, 2024
1 parent 3f8da51 commit 866a209
Show file tree
Hide file tree
Showing 2 changed files with 62 additions and 32 deletions.
26 changes: 14 additions & 12 deletions spiceaidocs/docs/cli/reference/login.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "login"
sidebar_label: "login"
title: 'login'
sidebar_label: 'login'
pagination_prev: null
pagination_next: null
---
Expand All @@ -15,18 +15,20 @@ spice login [command] [flags]

### Flags

- `-h`, `--help` Print this help message
- `-k`, `--key` string API key (for spice.ai)
- `-h`, `--help` Print this help message
- `-k`, `--key` string API key (for spice.ai)

#### Available Commands
- `databricks` Login to a Databricks instance
- `delta_lake` Configure credentials to access a Delta Lake table
- `dremio` Login to a Dremio instance
- `postgres` Login to a Postgres instance
- `s3` Login to an s3 storage
- `sharepoint` Login to a Microsoft 365 sharepoint account
- `snowflake` Login to a Snowflake warehouse
- `spark` Login to a Spark Connect remote

- `abfs` Login to a Azure Storage Account
- `databricks` Login to a Databricks instance
- `delta_lake` Configure credentials to access a Delta Lake table
- `dremio` Login to a Dremio instance
- `postgres` Login to a Postgres instance
- `s3` Login to an s3 storage
- `sharepoint` Login to a Microsoft 365 sharepoint account
- `snowflake` Login to a Snowflake warehouse
- `spark` Login to a Spark Connect remote

#### Examples

Expand Down
68 changes: 48 additions & 20 deletions spiceaidocs/docs/components/data-connectors/abfs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,20 @@ datasets:

Defines the ABFS-compatible URI to a folder or object:

- `from: abfs://<container>/<path>` with the account name configured using `abfs_account` parameter, or
- `from: abfs://<container>/<path>` with the account name configured using `abfs_account` parameter, or
- `from: abfs://<container>@<account_name>.dfs.core.windows.net/<path>`

### `name`

Defines the dataset name, which is used as the table name within Spice.

Example:

```yaml
datasets:
- from: abfs://foocontainer/taxi_sample.csv
name: cool_dataset
params:
...
params: ...
```

```sql
Expand Down Expand Up @@ -73,29 +73,28 @@ SELECT COUNT(*) FROM cool_dataset;
| `abfs_disable_tagging` | Disable tagging objects. Use this if your backing store doesn't support tags |
| `hive_partitioning_enabled` | Enable partitioning using hive-style partitioning from the folder structure. Defaults to `false` |


#### Authentication parameters

The following parameters are used when authenticating with Azure. Only one of these parameters can be used at a time:

* `abfs_access_key`
* `abfs_bearer_token`
* `abfs_client_secret`
* `abfs_skip_signature`
- `abfs_access_key`
- `abfs_bearer_token`
- `abfs_client_secret`
- `abfs_skip_signature`

If none of these are set the connector will default to using a [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)

| Parameter name | Description |
| --------------------------- | ----------------------------------------------------------- |
| `abfs_access_key` | Secret access key |
| `abfs_bearer_token` | `BEARER` token |
| `abfs_client_id` | Client ID for client authentication flow |
| `abfs_client_secret` | Client Secret to use for client authentication flow |
| `abfs_tenant_id` | Tenant ID to use for client authentication flow |
| `abfs_skip_signature` | Skip credentials and request signing for public containers |
| `abfs_msi_endpoint` | Endpoint for managed identity tokens |
| `abfs_federated_token_file` | File path for federated identity token in Kubernetes |
| `abfs_use_cli` | Set to `true` to use the Azure CLI to acquire access tokens |
| Parameter name | Description |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `abfs_access_key` | Secret access key |
| `abfs_bearer_token` | `BEARER` access token for user authentication. The token can be obtained from the OAuth2 flow (see [access token authentication](#access-token-authentication)). |
| `abfs_client_id` | Client ID for client authentication flow |
| `abfs_client_secret` | Client Secret to use for client authentication flow |
| `abfs_tenant_id` | Tenant ID to use for client authentication flow |
| `abfs_skip_signature` | Skip credentials and request signing for public containers |
| `abfs_msi_endpoint` | Endpoint for managed identity tokens |
| `abfs_federated_token_file` | File path for federated identity token in Kubernetes |
| `abfs_use_cli` | Set to `true` to use the Azure CLI to acquire access tokens |

#### Retry parameters

Expand All @@ -107,6 +106,35 @@ If none of these are set the connector will default to using a [managed identity
| `abfs_backoff_max_duration` | Maximum retry delay (e.g., `1m`) |
| `abfs_backoff_base` | Exponential backoff base (e.g., `0.1`) |

## Authentication

ABFS connector supports three types of authentication, as detailed in the [authentication parameters](#authentication-parameters)

### Service principal authentication

Configure service principal authentication by setting the `abfs_client_secret` parameter.

1. Create a new Azure AD application in the [Azure portal](https://portal.azure.com/#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Overview) and generate a `client secret` under `Certificates & secrets`.
2. Grant the Azure AD application read access to the storage account under `Access Control (IAM)`, this can typically be done using the `Storage Blob Data Reader` built-in role.

### Access key authentication

Configure service principal authentication by setting the `abfs_access_key` parameter to [Azure Storage Account Access Key](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal)

### Access token authentication

Configure access token authentication by setting the `abfs_bearer_token` parameter, typically obtained through the following the OAuth2 flow with `spice login abfs`.

1. Create a new Azure AD application in the [Azure portal](https://portal.azure.com/#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Overview).
2. Under the application's `API permissions`, add the permission: `Azure Storage - user_impersonation`.
3. Under the applications's `Authentication`, add `http://localhost` as Mobile and desktop applications redirect URI.
4. Grant the user read access to the storage account under `Access Control (IAM)`, this can typically be done using the `Storage Blob Data Reader` built-in role.
5. Obtain the `abfs_bearer_token` using the following command. The `abfs_bearer_token`, `abfs_client_id`, `abfs_tenant_id` will be automatically filled in environment secret after login. Refere to [`spice login`](/cli/reference/login) documentation for more details.

```shell
spice login abfs --tenant-id $TENANT_ID --client-id $CLIENT_ID
```

## Supported file formats

Specify the file format using `file_format` parameter. More details in [Object Store File Formats](/components/data-connectors/index.md#object-store-file-formats).
Expand Down Expand Up @@ -173,4 +201,4 @@ datasets:

## Secrets

Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the [secret stores documentation](/components/secret-stores). Additionally, learn how to use referenced secrets in component parameters by visiting the [using referenced secrets guide](/components/secret-stores#using-secrets).
Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the [secret stores documentation](/components/secret-stores). Additionally, learn how to use referenced secrets in component parameters by visiting the [using referenced secrets guide](/components/secret-stores#using-secrets).

0 comments on commit 866a209

Please sign in to comment.