From 8cd0ea597e0e3ac361f9e67734564a33a916723d Mon Sep 17 00:00:00 2001 From: Benjamin Winger Date: Thu, 3 Oct 2024 17:37:14 +0100 Subject: [PATCH] Add spill to disk setting doc --- src/content/docs/cypher/configuration.md | 8 ++++++++ src/content/docs/import/index.mdx | 12 ++++++++++++ 2 files changed, 20 insertions(+) diff --git a/src/content/docs/cypher/configuration.md b/src/content/docs/cypher/configuration.md index 3fe3f6f5..1011a6cf 100644 --- a/src/content/docs/cypher/configuration.md +++ b/src/content/docs/cypher/configuration.md @@ -20,6 +20,7 @@ configuration **cannot** be used with other query clauses, such as `RETURN`. | `PROGRESS_BAR_TIME` | show progress bar after time in ms | 1000 | | `CHECKPOINT_THRESHOLD` | the WAL size threshold in bytes at which to automatically trigger a checkpoint | 16777216 (16MB) | | `WARNING_LIMIT` | The maximum number of warnings that can be stored in a single connection | 8192 | +| `SPILL_TO_DISK_TMP_FILE` | The location of the temporary file to use to store data if there is not enough memory during a copy | `copy.tmp` inside the database directory | ### Database configuration | Option | Description | Default | @@ -74,3 +75,10 @@ CALL checkpoint_threshold=16777216; ```cypher CALL warning_limit=1024; ``` + +#### Configure Spill to disk temporary file +```cypher +CALL spill_to_disk_tmp_file="/path/to/tmp/file"; +# Disables spilling to disk +CALL spill_to_disk_tmp_file=""; +``` diff --git a/src/content/docs/import/index.mdx b/src/content/docs/import/index.mdx index 6148f3c3..ada57757 100644 --- a/src/content/docs/import/index.mdx +++ b/src/content/docs/import/index.mdx @@ -112,3 +112,15 @@ The above approach only works when the input data source contains *fewer* column node table being copied into. If the node table had fewer columns than the data source, it would lead to a Runtime error because of a column number mismatch. ::: + +## Importing large REL tables + +Importing large REL tables can use significant amounts of memory. If the memory used to prepare the +imported REL table data comes close to exceeding the buffer pool size, +Kùzu will spill some of this data to a `copy.tmp` file in the database directory. +The location of this file can be customized using the +[`spill_to_disk_tmp_file`](/cypher/configuration#configure-spill-to-disk-temporary-file) setting. +You can also disable spilling to disk by setting `spill_to_disk_tmp_file` to an empty string. + +This feature is disabled by default for in-memory databases and databases using file system extensions, +but can be enabled for these cases by setting a custom `spill_to_disk_tmp_file` path.