Skip to content

Commit

Permalink
Add Arrow Data Accelerator documentation (#443)
Browse files Browse the repository at this point in the history
* Add Arrow Data Accelerator documentation

Fixes #223

Add documentation for the In-Memory Arrow Data Accelerator.
  • Loading branch information
lukekim authored Oct 7, 2024
1 parent 89537d5 commit 0be0405
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 4 deletions.
33 changes: 33 additions & 0 deletions spiceaidocs/docs/components/data-accelerators/arrow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: 'In-Memory Arrow Data Accelerator'
sidebar_label: 'In-Memory Arrow Data Accelerator'
description: 'In-Memory Arrow Data Accelerator Documentation'
sidebar_position: 1
---

The In-Memory Arrow Data Accelerator is the default data accelerator in Spice. It uses Apache Arrow to store data in-memory for fast access and query performance.

## Configuration

To use the In-Memory Arrow Data Accelerator, specify `arrow` as the `engine` for acceleration.

```yaml
datasets:
- from: spice.ai:path.to.my_dataset
name: my_dataset
acceleration:
engine: arrow
```
## Limitations
- The In-Memory Arrow Data Accelerator does not support persistent storage. Data is stored in-memory and will be lost when the Spice runtime is stopped.
- The In-Memory Arrow Data Accelerator does not support `Decimal256` (76 digits), as it exceeds Arrow's maximum Decimal width of 38 digits.

:::warning[Memory Considerations]

When accelerating a dataset using the In-Memory Arrow Data Accelerator, some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.

In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by [`duckdb`](./duckdb.md) and [`sqlite`](./sqlite.md) accelerators by specifying `mode: file`.

:::
8 changes: 4 additions & 4 deletions spiceaidocs/docs/components/data-accelerators/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@ Supported Data Accelerators include:
| Engine Name | Description | Status | Engine Modes |
| --------------------------------- | ----------------------- | ------ | ---------------- |
| `arrow` | In-Memory Arrow Records | Alpha | `memory` |
| [`duckdb`](./duckdb.md) | Embedded DuckDB | Alpha | `memory`, `file` |
| [`sqlite`](./sqlite.md) | Embedded SQLite | Alpha | `memory`, `file` |
| [`postgres`](./postgres/index.md) | Attached PostgreSQL | Alpha | |
| [`arrow`](./arrow.md) | In-Memory Arrow Records | Beta | `memory` |
| [`duckdb`](./duckdb.md) | Embedded DuckDB | Beta | `memory`, `file` |
| [`sqlite`](./sqlite.md) | Embedded SQLite | Beta | `memory`, `file` |
| [`postgres`](./postgres/index.md) | Attached PostgreSQL | Beta | |

## Data Types

Expand Down

0 comments on commit 0be0405

Please sign in to comment.