Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single file deliveries #8

Open
farski opened this issue May 27, 2022 · 0 comments
Open

Single file deliveries #8

farski opened this issue May 27, 2022 · 0 comments

Comments

@farski
Copy link
Member

farski commented May 27, 2022

Currently all extract jobs are being run with a * wildcard name, meaning BigQuery can create multiple files at its own discretion. For large queries or partitioned tables, this will pretty much always result in multiple files, even if not strictly necessary based on file size limits.

Two possibilities to consider:

  1. An option or default to try one file first, and fail over to wildcard if BigQuery indicates the export was over the limit.
  2. Add the ability for us to concatenate down to a single file before delivery to the final destination.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant