ARROW-15168: [R] Add S3 generics to create main Arrow objects #12817

paleolimbot · 2022-04-06T19:46:46Z

No description provided.

github-actions · 2022-04-06T19:47:07Z

https://issues.apache.org/jira/browse/ARROW-15168

github-actions · 2022-04-06T19:47:08Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-04-14T18:43:40Z

Ok! This is ready for a review. The main motivation here is to allow other packages to define conversions to Arrow objects. This is most useful for Table objects, since we currently convert to Table before write_parquet/ipc_stream/feather/csv_arrow() and because converting an R vector, and for Array objects, because the R Vector -> Array conversion is at the heart of the default conversion to Table. A motivating project for these methods is 'geoarrow', which implements an ExtensionType to store geometry columns (e.g., sf::st_sfc()) and table-level metadata for table-level objects (e.g., sf::st_sf()).

Other methods I added here are important for lower-level packages like the work-in-progress 'substrait' and 'narrow', where there are analogues for DataType (e.g., the narrow_schema() or substrait.Type), Schema (e.g., the narrow_schema() or substrait.NamedStruct) , and RecordBatchReader (e.g., the narrow_array_stream()). In particular, I think that as_record_batch_reader() will be very useful (more flexible than Table for things like database results that can be streamed into the writers).

Adding the S3 methods was reasonably straightforward; however, actually using them was quite complicated because R Vector -> Array conversion is highly optimized and mostly done in C++. The approach I took was to keep the existing code, changing as little as possible, for R objects that we handle internally. For other objects, the C++ code calls type() and as_arrow_array(). As is apparent based on the number of files changed by this PR, that "simple" approach lead to a whack-a-mole of test failures that I think I was able to solve with a minimal footprint; however, another option is to close this PR and tackle the changes in reverse order with smaller PRs.

In a nutshell, before, users were stuck with the built-in behaviour for objects with a custom class. This was usually OK, but occasionally sub-optimal, non-functional, or corupting. I've picked a probably rare example because record-style vectors are new, but the current set of conversions assumes that they're list-like (as opposed to data-frame like):

# install.packages("arrow")
library(arrow, warn.conflicts = FALSE)

tbl <- tibble::tibble(
  x = 1:5,
  points = wk::xy(x = 1:5, y = 6:10)
)

# wk::xy() is a record-style vector, like POSIXlt, with a vctrs implementation
str(unclass(tbl$points))
#> List of 2
#>  $ x: num [1:5] 1 2 3 4 5
#>  $ y: num [1:5] 6 7 8 9 10

# in the release version this this fails
Table$create(tbl)
#> Error: Invalid: All columns must have the same length

# ...or generates bogus output
as.data.frame(Table$create(x = 1:2, points = tbl$points))
#> Warning: Invalid metadata$r
#> # A tibble: 2 × 2
#>       x         points
#>   <int> <list<double>>
#> 1     1            [5]
#> 2     2            [5]

After this PR you can define type() and as_arrow_array() and the conversion should "just work". This is particularly useful in conjunction with the new extension type support, which can handle most (non-list-based) vctr classes (e.g., this PR removes the internal conversions for POSIXlt and haven types because the vctrs extension array handles them out of the box).

# remotes::install_github("apache/arrow/r#12817")
library(arrow, warn.conflicts = FALSE)

tbl <- tibble::tibble(
  x = 1:5,
  points = wk::xy(x = 1:5, y = 6:10)
)

# wk::xy() is a record-style vector, like POSIXlt, with a vctrs implementation
str(unclass(tbl$points))
#> List of 2
#>  $ x: num [1:5] 1 2 3 4 5
#>  $ y: num [1:5] 6 7 8 9 10

# this now fails:
tf <- tempfile()
write_feather(tbl, tf)
#> Error:
#> ! Can't infer Arrow data type from object inheriting from wk_xy / wk_rcrd

# until...
type.wk_xy <- function(x, ...) {
  vctrs_extension_type(vctrs::vec_ptype(x))
}

as_arrow_array.wk_xy <- function(x, ...) {
  vctrs_extension_array(x)
}

# now works!
write_feather(tbl, tf)
read_feather(tf)
#> # A tibble: 5 × 2
#>       x points 
#>   <int> <wk_xy>
#> 1     1 (1  6) 
#> 2     2 (2  7) 
#> 3     3 (3  8) 
#> 4     4 (4  9) 
#> 5     5 (5 10)

# if for some reason the extension type is not loaded, we get the storage type
# with no warning (maybe not ideal?)
arrow::unregister_extension_type("arrow.r.vctrs")
read_feather(tf)
#> # A tibble: 5 × 2
#>       x points$x    $y
#>   <int>    <dbl> <dbl>
#> 1     1        1     6
#> 2     2        2     7
#> 3     3        3     8
#> 4     4        4     9
#> 5     5        5    10

^{Created on 2022-04-14 by the reprex package (v2.0.1)}

wjones127

This looks great Dewey. I had a few small comments and questions. Seems like a great step forward in supporting third-party types.

r/R/record-batch-reader.R

r/R/type.R

wjones127 · 2022-04-14T21:33:11Z

r/R/util.R

-    " must be an object of class 'data.frame', 'RecordBatch', or 'Table', not '",
-    class(env[[deparse(call$x)]])[[1]],
-    "'."
+as_writable_table <- function(x, arg_name = "x") {


If we are passing down the arg_name, do we want to also take the call object to pass in? That would make it so the function calling this helper would be shown instead of as_writable_table in the error chain.

I opted to just simplify this a bit...it only gets used in a few places! I'll send you a message about the use of sys.call() vs caller_env()...I did some digging and it makes sense to me now why the docs say to use caller_env().

r/R/type.R

wjones127 · 2022-04-14T22:04:17Z

r/src/table.cpp

+  // preserving R type information
+  for (R_xlen_t i = 0; i < schema->num_fields(); i++) {
+    if (schema->field(i)->type()->id() == Type::EXTENSION) {
+      metadata_columns[i] = R_NilValue;


Does this handle nested ExtensionType columns? is that a worry at all?

This was a really good catch! It led me down a bit of a rabbit hole but it now works!

# remotes::install_github("apache/arrow/r#12817") library(arrow, warn.conflicts = FALSE) nested <- tibble::tibble(x = vctrs::new_vctr(1:5, class = "custom_vctr")) infer_type(nested) #> StructType #> struct<x: <custom_vctr[0]>> as_arrow_array(nested) #> StructArray #> <struct<x: <custom_vctr[0]>>> #> -- is_valid: all not null #> -- child 0 type: <custom_vctr[0]> #> [ #> 1, #> 2, #> 3, #> 4, #> 5 #> ]

^{Created on 2022-04-19 by the reprex package (v2.0.1)}

r/tests/testthat/test-metadata.R

nealrichardson

A few notes, didn't get all the way through the PR, just the changes in r/R

nealrichardson · 2022-04-18T13:58:23Z

r/R/array.R

+
+#' @export
+as_arrow_array.POSIXlt <- function(x, ..., type = NULL) {
+  as_arrow_array.vctrs_vctr(x, ..., type = type)


Is there a risk of infinite loop if you have an object with class c("vctrs_vctr", "POSIXlt")? Or do we trust NextMethod() to do sane things?

I hadn't considered that! It doesn't look like that example causes problems and I haven't historically had problems with NextMethod(). Is there a related case I should be checking?

# remotes::install_github("apache/arrow/r#12817") library(arrow, warn.conflicts = FALSE) x <- as.POSIXlt(Sys.Date()) class(x) <- c("vctrs_vctr", "POSIXlt") array <- as_arrow_array(x) x2 <- as.vector(array) class(x2) #> [1] "vctrs_vctr" "POSIXlt"

No, I was just seeing the vctrs_vctr method doing NextMethod() and POSIXlt calling vctrs_vctr so thought they would bounce back to each other, and maybe they do but NextMethod handles it.

You were totally right...I was using NextMethod() to re-use the error generation code, so you could trigger an infinite loop with an invalid type and an oddly formed POSIXlt. A better approach was to pull the error code into its own function (this also came up in the data.frame method that was needed to make nested extension types work).

nealrichardson · 2022-04-18T13:59:22Z

r/R/array.R

+
+  # If from_constructor is FALSE, we use the built-in logic exposed by
+  # Array$create(). If there is no built-in conversion, C++ will call back
+  # here with from_constructor = TRUE to generate a nice error message.


Why does C++ have to call back here to create an error message?

I agree that it's weird. It's because array creation occurs from both R and C++ (and most of the time the call comes from C++). For example, Table$create() does:

[R] Table$create()

[C++] Table__from_dots()

[C++] arrow::r::RConverter (using all the old code pathways if we handle that object type internally, or)

arrow::r::RExtensionConverter -> [R] as_arrow_array().

Other R calls that use this or a similar path are Array$create(), ChunkedArray$create() and RecordBatch$create(). Perhaps some of that usage should use as_arrow_array() instead, but table creation in particular makes good use of threading and this was the best I came up with that didn't undo all of that optimization!

What happens if you don't error if from_constructor = TRUE?

I think I understand how you get here now, but I don't know that I'll remember next time I look at this. IIUC it's because the extension type handler is calling as_arrow_array() from C++, and if you have implemented as_arrow_array.YourType, you won't hit this default method. But if you do, it means that we don't know what to do with this extension type, so we error. Right?

But aren't all extension types based on regular types, which we can convert?

I updated some names and rewrote the comments to see if this can be clearer. If there's no error when from_vec_to_array = TRUE, we'd get an infinite loop. We always will need to check all the S3 methods and vec_to_Array(), regardless of whether somebody calls as_arrow_array() or something that goes through the Converter API in C++, so we need this kind of logic somewhere (in theory it could live on the C++ side too, but I think it's easier to write here).

I also renamed the RExensionType converter to AsArrowArrayConverter...there's no requirement that as_arrow_array() return an extension array (although most useful methods probably will).

A few places I had used NextMethod() to fall back and error, and, as you pointed out, that runs the risk of an infinite loop. I moved that error to a function to make it clearer/less error-prone when a conversion is not possible.

r/R/chunked-array.R

r/R/record-batch-reader.R

r/R/record-batch.R

r/R/type.R

nealrichardson · 2022-04-18T14:14:40Z

r/R/type.R

@@ -69,16 +70,43 @@ FLOAT_TYPES <- c("float16", "float32", "float64", "halffloat", "float", "double"
 #' type(mtcars)
 #' type(Sys.Date())
 #' @export
-type <- function(x) UseMethod("type")
+type <- function(x, ...) UseMethod("type")


Should this migrate to be called data_type? Seems inconsistent with as_data_type.

..or as_data_type could be as_type()? (I don't remember why I didn't do that in the first place). If renaming type() is an option, my vote would be infer_type() or infer_data_type() since we're not exactly constructing a type here, we're giving a hypothetical future type from something that might be converted to an Array.

r/R/type.R

nealrichardson · 2022-04-18T20:40:14Z

r/R/type.R

 #' @export
-type <- function(x) UseMethod("type")
+type <- function(x) infer_type(x)


This is probably safe to do but you should add to deprecated.R something like

type <- function(x) { .Deprecated("infer_type") infer_type(x) }

in case anyone was using this.

nealrichardson

One other thing you might want to add here: as_arrow_array.pyarrow.lib.Array and the similar reticulations

r/R/array.R

nealrichardson · 2022-04-19T16:43:08Z

r/R/array.R

@@ -217,6 +217,125 @@ Array$create <- function(x, type = NULL) {
 Array$import_from_c <- ImportArray


+#' Convert an object to an Arrow Array
+#'
+#' Whereas `Array$create()` constructs an [Array] from the built-in data types


This leaves me with some questions. Are as_arrow_array() conversions not optimized? If as_arrow_array() is going to call the regular Arrow converters for normal types, and it would work with extension types, why would I ever call Array$create()? When would Array$create() and as_arrow_array() do different things?

Maybe you can illustrate through some @examples.

I updated the docs to clarify this...they're almost the same, but Array$create() tries to convert in C++ first and doesn't have S3 dispatch overhead in all the common cases. I don't know if that tiny difference matters other than that it helps when testing to make sure that the C++-first pathway does the right thing.

nealrichardson · 2022-04-19T16:43:50Z

r/R/array.R

+      storage_type = type$storage_type()
+    )
+  } else {
+    stop_cant_convert_array(x, type)


Why not NextMethod anymore?

Because you were right about the possibility of infinite loops! We sometimes try the S3 methods first (i.e., when calling as_arrow_array()) and sometimes try C++ conversion first (e.g., when calling Table$create())...I found it a bit easier to predict what will happen if we always just stop when conversion isn't possible.

nealrichardson · 2022-04-19T16:49:42Z

r/R/array.R

+    arrays <- Map(as_arrow_array, x, types)
+    names(arrays) <- names
+
+    # ...because there isn't a StructArray$create() yet


There isn't (but there is in C++), but I think the reason we didn't add bindings to create from a vector of Arrays is because Array$create(data.frame) produces a StructArray. Why can't we use that here?

@wjones127's review highlighted the fact that the C++ conversion didn't handle ExtensionType fields within a StructArray, and because as_arrow_array() doesn't necessarily return an ExtensionType, it might do the wrong thing for an S3 vctr that maps to a regular Array type (e.g., narrow::narrow_vctr()). I went down a bit of a rabbit hole trying to make it work in C++; however, the StructArray converter is tightly coupled to the StructBuilder, the AsArrowArrayConverter completely circumvents the Builder.

All this to say that the C++ level can_convert_native() now recurses through data.frames and returns false if any nested column won't work in C++ 🤯. This means we also need an S3 method to handle that case, and hence, have to create a StructArray from Array objects. I have a vague recollection that I didn't see the point of StructArray$create() at the time (I was wrong!). I added a note about that next to the S3 method for data.frame since it's a little confusing.

Ok. Since using the C interface here is clearly a hack, and apparently there is a reason to add a StructArray$create method, could you please make (or find, it could already exist) a jira and link the id here as a TODO?

Done! ARROW-16266 (also added as a TODO(ARROW-16266) comment here)

nealrichardson · 2022-04-19T16:51:37Z

r/R/array.R

+  # If from_constructor is TRUE, this is a call from C++ for which S3 dispatch
+  # failed to find a method for the object. If this is the case, we error.


Presumably C++ has also tried using the regular type inference as well?

Yes! I added that to the comment (and to the one in type.R).

nealrichardson · 2022-04-19T17:07:13Z

r/tests/testthat/test-Array.R

+    float64()
+  }
+
+  # slightly different error if type was specified


Forgive the dead-horse beating, but it's hard for me to tell that this is the case by looking at the test file.

The flip side of that is that using regex it's impossible to tell that the error message is useful! With expect_error_snapshot(), the full errors are here: https://github.com/apache/arrow/pull/12817/files#diff-b944d8c5dbb671c5af92fedc0598a90f27d0aed3d3811175fa557c3ccf6cd339 )

You mean you don\'t speak regex[\?!]{2}

r/tests/testthat/test-RecordBatch.R

nealrichardson · 2022-04-19T17:11:05Z

r/tests/testthat/test-backwards-compatibility.R

+  # df <- data.frame(x = I(list(structure(1, foo = "bar"), structure(2, baz = "qux"))))
+  # class(df$x) <- c("sfc_MULTIPOLYGON", "sfc", "list")
+  # withr::with_options(
+  #   list("arrow.preserve_row_level_metadata" = TRUE), {


But this isn't the default in 7.0.0 is it?

It isn't...maybe a better name for the file would be 6.0.0? I wanted to keep this test to make sure that if somebody had written files this way that they could be read back in again (since it seems like that's our general policy), but if row-level metadata was never enabled by default then maybe we don't need this test?

r/tests/testthat/test-ipc_stream.R

r/tests/testthat/test-type.R

nealrichardson

This is looking good! Just a few more notes

nealrichardson · 2022-04-20T19:49:07Z

r/R/array.R

+        array <- vctrs_extension_array(
+          x,
+          ptype = type$ptype(),
+          storage_type = type$storage_type()
+        )
+        return(array)


The other cases don't return() so maybe this one doesn't need to either?

Suggested change

array <- vctrs_extension_array(

x,

ptype = type$ptype(),

storage_type = type$storage_type()

)

return(array)

vctrs_extension_array(

x,

ptype = type$ptype(),

storage_type = type$storage_type()

)

nealrichardson · 2022-04-20T19:51:25Z

r/R/array.R

+    arrays <- Map(as_arrow_array, x, types)
+    names(arrays) <- names
+
+    # ...because there isn't a StructArray$create() yet


Ok. Since using the C interface here is clearly a hack, and apparently there is a reason to add a StructArray$create method, could you please make (or find, it could already exist) a jira and link the id here as a TODO?

nealrichardson · 2022-04-20T19:58:17Z

r/R/python.R

-  pa <- reticulate::import("pyarrow", convert = FALSE)
-  out <- pa$Table$from_arrays(x$columns, schema = x$schema)
-  # But set the convert attribute on the return object to the requested value
+  # Going through RecordBatchReader maintains schema metadata (e.g.,


This change is clever, though I have two concerns:

Why would we lose schema metadata the other way? We pass the schema through the C interface. If it is losing metadata, that sounds like a bug to fix.

I'm not sure how the Table to RecordBatchReader code works but we should make sure that it's not allocating/concatenating Arrays so that the chunks of the ChunkedArrays line up into RecordBatches. If it did, that would be suboptimal I haven't read the source but it would be worth a look.

I made a JIRA (ARROW-16269) for the schema metadata thing...the gist of it is that the schema metadata roundtrips fine but you end up with a situation where roundtripped_table$col1$type isn't the same as roundtripped_table$schema$col1$type.

Going through RecordBatchReader does re-chunk everything to line up into record batches BUT the slicing is zero-copy (according to the comments:

arrow/cpp/src/arrow/table.h

Lines 240 to 244 in 1157e67

/// \brief Compute a stream of record batches from a (possibly chunked) Table

///

/// The conversion is zero-copy: each record batch is a view over a slice

/// of the table's columns.

class ARROW_EXPORT TableBatchReader : public RecordBatchReader {

). It looks like it zero-copy slices everything to match the column with the smallest batches (see details below). I don't think this will materially matter although it would be nice to avoid the potential re-chunking (I made ARROW-16269 to see if we can't reinstate column-wise conversion).

nealrichardson · 2022-04-20T20:01:36Z

r/R/record-batch-reader.R

+#' @rdname as_record_batch_reader
+#' @export
+as_record_batch_reader.RecordBatch <- function(x, ...) {
+  as_record_batch_reader(as_arrow_table(x))


You could go directly to the reader https://github.com/apache/arrow/blob/master/cpp/src/arrow/record_batch.h#L316

though this is probably fine

It's in scope and was easy so I added it!

r/R/record-batch.R

r/R/type.R

r/src/arrow_types.h

nealrichardson · 2022-04-22T11:55:43Z

@github-actions crossbow submit -g r

github-actions · 2022-04-22T12:37:40Z

Revision: 39c05b1

Submitted crossbow builds: ursacomputing/crossbow @ actions-1906

Task	Status
conda-linux-gcc-py37-cpu-r40
conda-linux-gcc-py37-cpu-r41
conda-osx-clang-py37-r40
conda-osx-clang-py37-r41
conda-win-vs2017-py37-r40
conda-win-vs2017-py37-r41
homebrew-r-autobrew
homebrew-r-brew
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-install-local
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-ubuntu-gcc-release-latest
test-r-rocker-r-base-latest
test-r-rstudio-r-base-4.1-centos7-devtoolset-8
test-r-rstudio-r-base-4.1-focal
test-r-rstudio-r-base-4.1-opensuse15
test-r-rstudio-r-base-4.1-opensuse42
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-18.04-r-sanitizer

nealrichardson · 2022-04-22T16:00:40Z

@github-actions crossbow submit test-r-arrow-backwards-compatibility

github-actions · 2022-04-22T16:01:47Z

Revision: aae6133

Submitted crossbow builds: ursacomputing/crossbow @ actions-1911

Task	Status
test-r-arrow-backwards-compatibility

paleolimbot · 2022-04-22T17:21:42Z

@github-actions crossbow submit test-r-arrow-backwards-compatibility

paleolimbot · 2022-04-22T17:33:38Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-04-22T17:34:37Z

Revision: f3d3f30

Submitted crossbow builds: ursacomputing/crossbow @ actions-1915

Task	Status
test-r-arrow-backwards-compatibility

…ata.frame

…nd as_arrow_array()

…create() instead of direct C++ call

…d vectors

paleolimbot · 2022-04-22T18:35:46Z

@github-actions crossbow submit test-r-arrow-backwards-compatibility

github-actions · 2022-04-22T18:36:53Z

Revision: b21126e

Submitted crossbow builds: ursacomputing/crossbow @ actions-1918

Task	Status
test-r-arrow-backwards-compatibility

ursabot · 2022-04-28T17:11:37Z

Benchmark runs are scheduled for baseline = c04c3ff and contender = c13ef50. c13ef50 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.71% ⬆️0.71%] ursa-i9-9960x
[Finished ⬇️0.29% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] c13ef50a ec2-t3-xlarge-us-east-2
[Failed] c13ef50a test-mac-arm
[Finished] c13ef50a ursa-i9-9960x
[Finished] c13ef50a ursa-thinkcentre-m75q
[Finished] c04c3ff2 ec2-t3-xlarge-us-east-2
[Failed] c04c3ff2 test-mac-arm
[Finished] c04c3ff2 ursa-i9-9960x
[Finished] c04c3ff2 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-04-28T17:11:46Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

The `type()` function was deprecated in #12817 (ARROW-15168), but the documentation did not indicate that it had been deprecated. Closes #13748 from eitsupi/r-deprecated-type Authored-by: SHIMA Tatsuya <[email protected]> Signed-off-by: Nic Crane <[email protected]>

github-actions bot added Component: R Component: Parquet labels Apr 6, 2022

paleolimbot force-pushed the r-s3-generics branch 2 times, most recently from 0f351f8 to 18de4ce Compare April 11, 2022 19:09

paleolimbot marked this pull request as ready for review April 14, 2022 18:43

wjones127 reviewed Apr 14, 2022

View reviewed changes

nealrichardson reviewed Apr 18, 2022

View reviewed changes

r/R/type.R Show resolved Hide resolved

nealrichardson reviewed Apr 18, 2022

View reviewed changes

paleolimbot force-pushed the r-s3-generics branch 2 times, most recently from e19827d to 245b0bf Compare April 19, 2022 16:53

nealrichardson reviewed Apr 19, 2022

View reviewed changes

paleolimbot force-pushed the r-s3-generics branch from 0132e49 to d4536ac Compare April 19, 2022 23:52

nealrichardson reviewed Apr 20, 2022

View reviewed changes

nealrichardson approved these changes Apr 22, 2022

View reviewed changes

paleolimbot added 4 commits April 22, 2022 14:45

add as_arrow_array()

08843dd

add as_chunked_array()

6403ab4

Add as_arrow_table

696fb4f

as_record_batch()

7c48b24

paleolimbot added 20 commits April 22, 2022 14:55

update table/record batch tests now that metadata is identical

82c8b82

fix Array error snapshots

ec68ce0

no more examplesIf arrow_available() for files in this PR

a5b9313

better documentation for as_arrow_array()

929dc18

make sure Array$create() and as_arrow_array() do the same thing for d…

2482e1d

…ata.frame

update comments to clarify that C++ is tried first for infer_type() a…

46e8467

…nd as_arrow_array()

examples for as_record_batch() and as_arrow_table(), use RecordBatch$…

36bab18

…create() instead of direct C++ call

move old_example_with_metadata to test-backwards-compatability

59f726f

handle x where vctrs::vec_is() in default methods

fd167c0

fix typo in test title, move vctrs stuff to the default method

864dfce

fix roundtripping of Table with metadata/extension type

5f6756b

infer vctrs storage type based on the whole vector (not just the ptype)

2790d1e

simplify if/else in as_arrow_array/infer_type default methods

d832d02

link JIRA for StructArray$create() hack

560458b

add RecordBatchReader from batches

edec1d5

add todo for column-wise construction of tables for python bridge

6859e87

Don't try to write Feather V1 with VctrsExtensionType column

752e115

update test-read-files for feather v1 and for roundtripping of classe…

df42412

…d vectors

clean up C pointers

f4e98cf

fix snapshot test files

6a6ea12

paleolimbot force-pushed the r-s3-generics branch from f3d3f30 to 6a6ea12 Compare April 22, 2022 18:09

fix more references to the special string column

b21126e

nealrichardson closed this in c13ef50 Apr 23, 2022

eitsupi mentioned this pull request Jul 29, 2022

MINOR: [R] Add note about the deprecated type() function #13748

Closed

paleolimbot deleted the r-s3-generics branch December 9, 2022 16:35

		# If from_constructor is TRUE, this is a call from C++ for which S3 dispatch
		# failed to find a method for the object. If this is the case, we error.

	/// \brief Compute a stream of record batches from a (possibly chunked) Table
	///
	/// The conversion is zero-copy: each record batch is a view over a slice
	/// of the table's columns.
	class ARROW_EXPORT TableBatchReader : public RecordBatchReader {

ARROW-15168: [R] Add S3 generics to create main Arrow objects #12817

ARROW-15168: [R] Add S3 generics to create main Arrow objects #12817

Conversation

paleolimbot commented Apr 6, 2022

github-actions bot commented Apr 6, 2022

github-actions bot commented Apr 6, 2022

paleolimbot commented Apr 14, 2022

wjones127 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

nealrichardson commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

paleolimbot commented Apr 22, 2022

paleolimbot commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

paleolimbot commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

ursabot commented Apr 28, 2022

ursabot commented Apr 28, 2022