Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: external location for fuse engine #7297

Closed
Tracked by #7592
BohuTANG opened this issue Aug 25, 2022 · 2 comments · Fixed by #7789
Closed
Tracked by #7592

feat: external location for fuse engine #7297

BohuTANG opened this issue Aug 25, 2022 · 2 comments · Fixed by #7789
Assignees
Labels
C-feature Category: feature

Comments

@BohuTANG
Copy link
Member

Summary

If we create table ... engine=fuseand write some data, the progress like:

  1. Put the table schema to meta service
  2. Read config from storage.s3 and create a storage operator
  3. Write parquet file to storage
  4. Commit the snapshot to meta service

This is how databend fuse engine work now: fuse engine data will write to the storage configured in the databend-query.toml.

If we support(The syntax only for proposal):

create table ... engine=fuse <externalLocation>

externalLocation ::=
  's3://<bucket>[<path>]'
  CONNECTION = (
        ENDPOINT_URL = 's3[gcp]://<endpoint-URL>'
        ACCESS_KEY_ID = '<your-access-key-ID>'
        SECRET_ACCESS_KEY = '<your-secret-access-key>'
        REGION = '<region-name>'
        ENABLE_VIRTUAL_HOST_STYLE = true|false
  )

Things are changed: we can write fuse engine data to an external location that is controlled by the user, and the user can read the parquet file created by the fuse engine since they are all parquet format!

@BohuTANG BohuTANG added the C-feature Category: feature label Aug 25, 2022
@Xuanwo Xuanwo moved this to 🔨 In Progress in Xuanwo's Work Aug 25, 2022
@Xuanwo Xuanwo moved this from 🔨 In Progress to 📋 Backlog in Xuanwo's Work Aug 25, 2022
@Xuanwo Xuanwo removed this from Xuanwo's Work Sep 14, 2022
@BohuTANG
Copy link
Member Author

Cc @soyeric128 for documentation. Thanks.

@soyeric128
Copy link
Collaborator

Cc @soyeric128 for documentation. Thanks.

So this option stores the table's snapshots, segments, and blocks to an external s3 location other than the default one configured in the [storage.s3]?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants