feat(c/driver/postgresql): Accept bulk ingest of dictionary-encoded strings/binary #1275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lidavidm
approved these changes
Nov 9, 2023
| @@ -1283,6 +1279,26 @@ class PostgresCopyBinaryFieldWriter : public PostgresCopyFieldWriter { | |||
| } | |||
| }; | |||
|
|
|||
| class PostgresCopyBinaryDictFieldWriter : public PostgresCopyFieldWriter { | |||
There was a problem hiding this comment.
It seems like eventually, we'd want to be able to parametrize this writer on the sub-writer, but we can defer that for now
vleslief-ms
pushed a commit
to vleslief-ms/arrow-adbc
that referenced
this pull request
Nov 9, 2023
…trings/binary (apache#1275) This PR adds the ability for the Postgres driver to ingest dictionary-encoded arrays. This shows up in R because factors are relatively common and encode by default in Arrow to dictionary-encoded string for performance reasons. Reprex in R: ``` r library(adbcdrivermanager) con <- adbcpostgresql::adbcpostgresql() |> adbc_database_init(uri = Sys.getenv("ADBC_POSTGRESQL_TEST_URI")) |> adbc_connection_init() df <- data.frame(x = letters, y = factor(letters)) write_adbc(df, con, "some_table") #> Error in adbc_statement_execute_query(stmt): [libpq] Failed to create table: ERROR: relation "some_table" already exists #> #> Query was: CREATE TABLE "public" . "some_table" ("x" TEXT, "y" TEXT) read_adbc(con, "SELECT * from some_table") |> as.data.frame() |> str() #> 'data.frame': 26 obs. of 2 variables: #> $ x: chr "a" "b" "c" "d" ... #> $ y: chr "a" "b" "c" "d" ... ``` <sup>Created on 2023-11-09 with [reprex v2.0.2](https://reprex.tidyverse.org)</sup> There is probably some opportunity to consolidate some of the code that currently lives in the `BindStream` into the `PostgresType` and/or `PostgresTypeResolver`...I'm happy to poke away at that at some point but in the meantime it seemed like it wasn't too onerous to tack on dictionary support here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the ability for the Postgres driver to ingest dictionary-encoded arrays. This shows up in R because factors are relatively common and encode by default in Arrow to dictionary-encoded string for performance reasons.
Reprex in R:
Created on 2023-11-09 with reprex v2.0.2
There is probably some opportunity to consolidate some of the code that currently lives in the
BindStreaminto thePostgresTypeand/orPostgresTypeResolver...I'm happy to poke away at that at some point but in the meantime it seemed like it wasn't too onerous to tack on dictionary support here.