Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runtime: Use Arrow to ingest data from BigQuery into DuckDB #2953

Open
begelundmuller opened this issue Aug 22, 2023 · 2 comments
Open

Runtime: Use Arrow to ingest data from BigQuery into DuckDB #2953

begelundmuller opened this issue Aug 22, 2023 · 2 comments
Labels
Team:Platform Platform Working Group

Comments

@begelundmuller
Copy link
Contributor

No description provided.

@begelundmuller begelundmuller mentioned this issue Aug 22, 2023
Closed
@begelundmuller begelundmuller added the Team:Platform Platform Working Group label Aug 22, 2023
@k-anshul
Copy link
Member

depends on following external PRs:

  1. feat(bigquery): expose Apache Arrow data googleapis/google-cloud-go#8500
  2. [Go] Arrow to parquet conversion fails for arrow.DECIMAL256 type apache/arrow#37419
  3. C-API: Fix duckdb_arrow_scan API  duckdb/duckdb#8653

@begelundmuller begelundmuller changed the title Runtime: Use Arrow for ingesting data from BigQuery Runtime: Use Arrow to ingest data from BigQuery into DuckDB Sep 12, 2023
@begelundmuller
Copy link
Contributor Author

For now, we implemented a BigQuery -> Arrow -> local Parquet -> DuckDB pipeline, which leans into stable Parquet writers and readers in both Go and DuckDB. In the future, it would be great to ingest Arrow data directly into DuckDB, but it currently doesn't have good type support and feels unstable. Moving this item to the backlog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Platform Platform Working Group
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@begelundmuller @k-anshul