ARROW-5588: [C++] Better support for building union arrays #4781

bkietz · 2019-07-02T23:51:31Z

Simplify DenseUnionBuilder
Add SparesUnionBuilder
MakeBuilder can now produce a {Sparse,Dense}UnionBuilder
ArrayFromJSON can now produce union arrays

wesm · 2019-07-02T23:57:54Z

[ RUN      ] TestDictionary.ListOfDictionary
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0702 23:55:33.778414  6046 array.cc:287]  Check failed: list_type_->value_type()->Equals(data->child_data[0]->type)
*** Check failure stack trace: ***
/buildbot/AMD64_Conda_C__/cpp/build-support/run-test.sh: line 97:  6046 Aborted                 (core dumped) $TEST_EXECUTABLE "$@" 2>&1
      6047 Done                    | $ROOT/build-support/asan_symbolize.py
      6048 Done                    | ${CXXFILT:-c++filt}
      6049 Done                    | $ROOT/build-support/stacktrace_addr2line.pl $TEST_EXECUTABLE
      6050 Done                    | $pipe_cmd 2>&1
      6051 Done                    | tee $LOGFILE
/buildbot/AMD64_Conda_C__/cpp/build/src/arrow

bkietz · 2019-07-03T00:04:21Z

I'll disable that test until I can address the issue in ListBuilder

cpp/src/arrow/array/builder_union.cc

cpp/src/arrow/array/builder_union.h

fsaintjacques · 2019-07-05T19:40:56Z

cpp/src/arrow/array/builder_union.cc

+  type_id_to_child_num_.resize(union_type->max_type_code() + 1, -1);
+  DCHECK_EQ(union_type->mode(), UnionMode::DENSE);
+  int child_i = 0;
+  for (auto type_id : union_type->type_codes()) {


Would it simplify to require for children.size() the same as max_type_code()? If you're not, you're asking for trouble. The user can always fill with NullBuilder.

I can let children_[i] be the builder for type_id == i instead of the builder which will finish into child_data[i]. The latter seems like it might be an implicit contract though; I'll investigate

cpp/src/arrow/array/builder_union.cc

fsaintjacques · 2019-07-05T20:01:48Z

cpp/src/arrow/array/builder_union.h

  }

 private:
  TypedBufferBuilder<int8_t> types_builder_;
  TypedBufferBuilder<int32_t> offsets_builder_;
  std::vector<std::string> field_names_;
+  std::vector<int8_t> type_id_to_child_num_;


This might be unnecessary if we maintain an invariant on children_.

bkietz · 2019-07-10T20:05:51Z

@fsaintjacques Travis is green https://travis-ci.org/bkietz/arrow/builds/556909415

fsaintjacques

LGTM, some small changes and nothing serious. Good job on simplifying the indirection in the builders.

cpp/src/arrow/array/builder_union.cc

cpp/src/arrow/array/builder_union.h

cpp/src/arrow/compare.cc

cpp/src/arrow/array/builder_union.cc

fsaintjacques · 2019-07-11T18:34:05Z

cpp/src/arrow/array-dict-test.cc

+  ASSERT_OK(ValidateArray(*array));
+
+  auto expected_type = list(dictionary(int16(), utf8()));
+  ASSERT_EQ(array->type()->ToString(), expected_type->ToString());


ToString? Applies to other reference.

This was a hack to get more info out of the assertion failure than "types are unequal".

NB: serialize.cc is probably broken

… by type_id

fsaintjacques · 2019-07-12T14:37:41Z

Ok, it passes locally with lint, so I'll proceed to merge.

- Simplify DenseUnionBuilder - Add SparesUnionBuilder - MakeBuilder can now produce a {Sparse,Dense}UnionBuilder - ArrayFromJSON can now produce union arrays Author: Benjamin Kietzman <[email protected]> Closes #4781 from bkietz/5588-Better-support-for-building-UnionArrays and squashes the following commits: 38e2828 <Benjamin Kietzman> iwyu #include <limits> 0efe91d <Benjamin Kietzman> address review comments 17e6e27 <Benjamin Kietzman> construct offset_builder_ with a MemoryPool 4131fe3 <Benjamin Kietzman> separate child builder array indexable by type_id fd64c1b <Benjamin Kietzman> rewrite union builder to share a base class, let children_ be indexed by type_id 37de5f2 <Benjamin Kietzman> explicit uint8_t for msvc 673916e <Benjamin Kietzman> Disable ListOfDictionary test until ListBuilder is updated cf1c5be <Benjamin Kietzman> revert changes to reader.cc 5742db9 <Benjamin Kietzman> debugging: highlight the broken case and a similar one 5b1ec93 <Benjamin Kietzman> improve doccomments, dedupe test code 33fade1 <Benjamin Kietzman> Adding support for DenseUnions to ArrayFromJSON 6245c82 <Benjamin Kietzman> add SparseUnionBuilder and MakeBuilder case 8d4f36d <Benjamin Kietzman> add tests for building lists where the value builder has mutable type 351905d <Benjamin Kietzman> add test for lazily typed union builder 7902d12 <Benjamin Kietzman> first pass at updating DenseUnionBuilder d20055a <Benjamin Kietzman> minor refactors, adding some asserts

kou force-pushed the master branch from e3ae1b0 to 03576af Compare July 4, 2019 05:51

kszucs force-pushed the 5588-Better-support-for-building-UnionArrays branch from 566014c to 7b2b41d Compare July 4, 2019 07:57

kszucs changed the title ~~ARROW-5588: [C++] better support for building union arrays~~ ARROW-5588: [C++] Better support for building union arrays Jul 4, 2019

kszucs force-pushed the 5588-Better-support-for-building-UnionArrays branch from 7b2b41d to a20f739 Compare July 5, 2019 07:11

emkornfield force-pushed the 5588-Better-support-for-building-UnionArrays branch from a20f739 to a1c6225 Compare July 5, 2019 09:04

fsaintjacques self-requested a review July 5, 2019 18:42

fsaintjacques requested changes Jul 5, 2019

View reviewed changes

bkietz force-pushed the 5588-Better-support-for-building-UnionArrays branch from a1c6225 to bab294c Compare July 10, 2019 15:16

fsaintjacques approved these changes Jul 11, 2019

View reviewed changes

bkietz added 15 commits July 11, 2019 17:58

minor refactors, adding some asserts

d20055a

first pass at updating DenseUnionBuilder

7902d12

NB: serialize.cc is probably broken

add test for lazily typed union builder

351905d

add tests for building lists where the value builder has mutable type

8d4f36d

add SparseUnionBuilder and MakeBuilder case

6245c82

Adding support for DenseUnions to ArrayFromJSON

33fade1

improve doccomments, dedupe test code

5b1ec93

debugging: highlight the broken case and a similar one

5742db9

revert changes to reader.cc

cf1c5be

Disable ListOfDictionary test until ListBuilder is updated

673916e

explicit uint8_t for msvc

37de5f2

rewrite union builder to share a base class, let children_ be indexed…

fd64c1b

… by type_id

separate child builder array indexable by type_id

4131fe3

construct offset_builder_ with a MemoryPool

17e6e27

address review comments

0efe91d

bkietz force-pushed the 5588-Better-support-for-building-UnionArrays branch from cdcd799 to 0efe91d Compare July 11, 2019 21:58

iwyu #include <limits>

38e2828

fsaintjacques closed this in 1bcfbe1 Jul 12, 2019

bkietz deleted the 5588-Better-support-for-building-UnionArrays branch February 25, 2021 16:39

asfimport mentioned this pull request Aug 1, 2019

[C++] Better support for building UnionArrays #22029

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-5588: [C++] Better support for building union arrays #4781

ARROW-5588: [C++] Better support for building union arrays #4781

bkietz commented Jul 2, 2019

wesm commented Jul 2, 2019

bkietz commented Jul 3, 2019

fsaintjacques Jul 5, 2019

bkietz Jul 10, 2019

fsaintjacques Jul 5, 2019

bkietz commented Jul 10, 2019

fsaintjacques left a comment

fsaintjacques Jul 11, 2019

bkietz Jul 11, 2019

fsaintjacques commented Jul 12, 2019

ARROW-5588: [C++] Better support for building union arrays #4781

ARROW-5588: [C++] Better support for building union arrays #4781

Conversation

bkietz commented Jul 2, 2019

wesm commented Jul 2, 2019

bkietz commented Jul 3, 2019

fsaintjacques Jul 5, 2019

Choose a reason for hiding this comment

bkietz Jul 10, 2019

Choose a reason for hiding this comment

fsaintjacques Jul 5, 2019

Choose a reason for hiding this comment

bkietz commented Jul 10, 2019

fsaintjacques left a comment

Choose a reason for hiding this comment

fsaintjacques Jul 11, 2019

Choose a reason for hiding this comment

bkietz Jul 11, 2019

Choose a reason for hiding this comment

fsaintjacques commented Jul 12, 2019