Allow PARQUET format for uploading data. #609
Conversation
…age depends on R > 4.0.0
Co-authored-by: Hadley Wickham <[email protected]>
Co-authored-by: Hadley Wickham <[email protected]>
|
Hi @hadley, Thank you for your review. I have taken into account all of your comments, and I hope I haven't missed any. Regards, |
NEWS.md
Outdated
| @@ -1,5 +1,14 @@ | |||
| # bigrquery (development version) | |||
|
|
|||
| ## Significant improvements | |||
There was a problem hiding this comment.
Can you leave these headings off? We add them as part of the final release process?
DESCRIPTION
Outdated
| @@ -29,7 +29,8 @@ Imports: | |||
| prettyunits, | |||
| rlang (>= 1.1.0), | |||
| tibble | |||
| Suggests: | |||
| Suggests: | |||
| arrow, | |||
There was a problem hiding this comment.
Would you consider trying nanoparquet instead? It's very new but has no dependencies, so we could use it from imports.
There was a problem hiding this comment.
I agree that nanoparquet is a good alternative because it has no dependencies. The only downside I see is that you would have to write to disk to read the raw data, as it lacks an output stream buffer implementation.
If you think the advantages outweigh the disadvantages, I could start testing and adapting the code.
There was a problem hiding this comment.
Ah I didn't think about that, but I suspect it's still worthwhile given the lighter dependencies. Do you mind filing a nanoparquet issue to add a stream buffer output?
There was a problem hiding this comment.
I will begin implementing nanoparquet. If BufferedOutputStream is added in the future, an update will be necessary.
|
Hi @hadley, The implementation of Nanoparquet to replace Arrow has been completed. After several data loading tests, I believe it works very well. I look forward to your comments. Regards, |
|
Thanks so much for working on this! |
Hi @hadley,
This pull request contains the implementation for allowing the user to decide in which format they want to transmit the data (JSON or PARQUET) to BigQuery (For large amounts of data, loading data in JSON format is very time-consuming due to the size of the data that needs to be transmitted. To address this problem, BigQuery accepts other file formats, including Parquet). The most significant change enabling PARQUET data transmission is that the uploadType is no longer multipart; it is now resumable.
https://cloud.google.com/bigquery/docs/reference/api-uploads
Regards,