Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parquet): read in parallel. #7903

Merged
merged 8 commits into from
Sep 27, 2022
Merged

feat(parquet): read in parallel. #7903

merged 8 commits into from
Sep 27, 2022

Conversation

youngsofun
Copy link
Member

@youngsofun youngsofun commented Sep 26, 2022

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR

part of #7732

Fixes #7231 (big parquet oom)

@vercel
Copy link

vercel bot commented Sep 26, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
databend ⬜️ Ignored (Inspect) Sep 27, 2022 at 1:12AM (UTC)

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Sep 26, 2022
@youngsofun youngsofun marked this pull request as draft September 26, 2022 12:55
@youngsofun
Copy link
Member Author

youngsofun commented Sep 26, 2022

@dantengsky I often see errors like this when I test copy into ontime200 .. manually , even on a new table just created (with table name never used before)

 2022-09-26T12:53:10.803063Z ERROR common_meta_api::schema_api_impl: error=TableVersionMismatched: 160 expect `== 163` but `168`  while `update_table_meta

I deleted local ./databend and testbucket/admin/1 on minio

@dantengsky
Copy link
Member

@dantengsky I often see errors like this when I test copy into ontime200 .. manually , even on a new table just created (with table name never used before)

 2022-09-26T12:53:10.803063Z ERROR common_meta_api::schema_api_impl: error=TableVersionMismatched: 160 expect `== 163` but `168`  while `update_table_meta

I deleted local ./databend and testbucket/admin/1 on minio

yes ,I'have saw the same errors occasionally, in some unexpected situations(where I think there should be no TableVersionMismatched exception). unfortunately, I forget the details : (

but in my case, it is not related to the deletion of data from storage.

is there a way to reproduce this error?

@youngsofun
Copy link
Member Author

youngsofun commented Sep 26, 2022

interpreters::interpreter_table_recluster::test_alter_recluster_interpreter
fail again @zhyass

#7849


all tests are passed, except this not related flaky one. just need to rerun.

@youngsofun
Copy link
Member Author

@dantengsky

mysql> copy into ontime4 from 's3://testbucket/admin/data/ontime_200.csv' CONNECTION=(aws_key_id='minioadmin' aws_secret_key='minioadmin' endpoint_url='http://127.0.0.1:9900/') FILE_FORMAT = (type = 'csv' skip_header=1 ) force=true; 


@youngsofun youngsofun mentioned this pull request Sep 26, 2022
58 tasks
@youngsofun youngsofun marked this pull request as ready for review September 26, 2022 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: copy from parquet file OOM
4 participants