Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import duckdb-replicator #6067

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions runtime/pkg/rduckdb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# rduckdb

## Motivation
1. As an embedded database, DuckDB does not inherently provide the same isolation for ETL and serving workloads that other OLAP databases offer.
2. We have observed significant degradation in query performance during data ingestion.
3. In a Kubernetes environment, it is recommended to use local disks instead of network disks, necessitating separate local disk backups.

## Features
1. Utilizes separate DuckDB handles for reading and writing, each with distinct CPU and memory resources.
2. Automatically backs up writes to GCS in real-time.
3. Automatically restores from backups when starting with an empty local disk.

## Examples
1. Refer to `examples/main.go` for a usage example.

## Future Work
1. Enable writes and reads to be executed on separate machines.
2. Limit read operations to specific tables to support ephemeral tables (intermediate tables required only for writes).
Loading
Loading