Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: fuse block file cache #8647

Open
Tracked by #7823
BohuTANG opened this issue Nov 5, 2022 · 2 comments
Open
Tracked by #7823

feat: fuse block file cache #8647

BohuTANG opened this issue Nov 5, 2022 · 2 comments
Labels
C-feature Category: feature

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Nov 5, 2022

Summary

Now, Databend will cache these in memory:

  1. snapshot file
  2. segment file
  3. index file

Because all of these items' sizes are not big and fit the memory well.

To understand more about how the Databend cache works, for example:
Q1:

select id, name, age, city from t1 where age > 20 and age < 30;

The progress likes:

  1. Read the latest snapshot file -- cached in memory
  2. Pruning the segment file and reading it from the s3 -- cached in memory
  3. Read the id&name&age&city column(In Databend , it is called block and in Parquet format) from the s3 -- no cached

So if we run another SQL, Q2:

select id, name, age, city from t1 where age > 25 and age < 30;

The progress is as follows:

  1. Read the latest snapshot file from memory
  2. Pruning the segment file and reading from memory
  3. Read the id&name&age&city column(In Databend , it is called block and in Parquet format) from the s3 -- no cached

If we cache the column parquet files in step 3, the Q2 will avoid file reads from s3, faster!

Update:
Each column file(AKA block file) in Databend is a parquet file with one RowGroup, and the range reader is the RowGroup data, which is the entire parquet file except the footer.

Question

To avoid writing through blocks to disk(which would affect reads), we should use a Memory + Disk LRU cache, such as memory size: 1GB, and Disk size: 10GB, and make the block write-back async.

At last, I'd like to show the performance gains Snowflake has made thanks to caching for the hits dataset test:
Run Q1: SELECT COUNT() FROM hits.public.hits2;
Then Q2: SELECT COUNT(
) FROM hits.public.hits2 WHERE AdvEngineID <> 0;
image

@BohuTANG BohuTANG added the C-feature Category: feature label Nov 5, 2022
@BohuTANG
Copy link
Member Author

BohuTANG commented Nov 5, 2022

cc @Xuanwo @dantengsky

@Xuanwo
Copy link
Member

Xuanwo commented Nov 18, 2022

Addressed by #8830

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

2 participants