Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add spill to disk setting doc #280

Merged
merged 1 commit into from
Nov 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/content/docs/cypher/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ configuration **cannot** be used with other query clauses, such as `RETURN`.
| `PROGRESS_BAR_TIME` | show progress bar after time in ms | 1000 |
| `CHECKPOINT_THRESHOLD` | the WAL size threshold in bytes at which to automatically trigger a checkpoint | 16777216 (16MB) |
| `WARNING_LIMIT` | The maximum number of warnings that can be stored in a single connection | 8192 |
| `SPILL_TO_DISK_TMP_FILE` | The location of the temporary file to use to store data if there is not enough memory during a copy | `copy.tmp` inside the database directory |

### Database configuration
| Option | Description | Default |
Expand Down Expand Up @@ -74,3 +75,10 @@ CALL checkpoint_threshold=16777216;
```cypher
CALL warning_limit=1024;
```

#### Configure Spill to disk temporary file
```cypher
CALL spill_to_disk_tmp_file="/path/to/tmp/file";
# Disables spilling to disk
CALL spill_to_disk_tmp_file="";
```
12 changes: 12 additions & 0 deletions src/content/docs/import/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,15 @@ The above approach only works when the input data source contains *fewer* column
node table being copied into. If the node table had fewer columns than the data source, it would
lead to a Runtime error because of a column number mismatch.
:::

## Importing large REL tables

Importing large REL tables can use significant amounts of memory. If the memory used to prepare the
imported REL table data comes close to exceeding the buffer pool size,
Kùzu will spill some of this data to a `copy.tmp` file in the database directory.
The location of this file can be customized using the
[`spill_to_disk_tmp_file`](/cypher/configuration#configure-spill-to-disk-temporary-file) setting.
You can also disable spilling to disk by setting `spill_to_disk_tmp_file` to an empty string.

This feature is disabled by default for in-memory databases and databases using file system extensions,
but can be enabled for these cases by setting a custom `spill_to_disk_tmp_file` path.