ARROW-8866: [C++] Split UNION into SPARSE_UNION and DENSE_UNION #7378

bkietz · 2020-06-08T18:59:52Z

Also splits UnionType -> SparseUnionType and DenseUnionType and similar for UnionArray, UnionScalar.

SparseUnionArray no longer includes the unused offsets buffer

github-actions · 2020-06-08T19:02:57Z

https://issues.apache.org/jira/browse/ARROW-8866

cpp/src/arrow/c/bridge.cc

wesm · 2020-06-08T19:46:59Z

Ah wish I had synced up with you about compute/kernels/vector_selection_internal.h, hope that didn't take up too much of your time. I've deleted all the Take-related union code for now since I'm revamping the entire approach to performing takes. Don't worry about it -- I will rebase my patch since this one should probably go in first.

bkietz · 2020-06-08T20:48:27Z

@wesm no problem it didn't take much time. Sparse and Dense were basically distinct code paths already in vector_selection_internal. No need to rebase yours; rebasing this one across that file shouldn't be a problem

wesm · 2020-06-09T01:01:19Z

@bkietz OK, I likely won't be done with what I'm doing until tomorrow. To be clear I am removing Union take functionality altogether until it can be adapted to the new approach at a later time (and disabling the existing unit tests). This is such a seldom traveled path that I don't feel it's worth investing the effort right now.

wesm · 2020-06-09T18:39:28Z

My patch #7382 should not conflict with this so no need to worry about rebasing

cpp/src/arrow/array/array_nested.h

pitrou · 2020-06-10T10:39:17Z

cpp/src/arrow/array/array_primitive.h

  const uint8_t* raw_values() const { return raw_values_ + data_->offset * byte_width(); }

 protected:
+  static constexpr int32_t byte_width() { return sizeof(TypeClass::DayMilliseconds); }


Not sure why you made this protected? The byte width may be useful.

It's not used anywhere except in this class. If needed it can easily be reintroduced but it seemed better to minimize the public API

Well, it was public so could be used by anybody really. I'm ok with not making the public API too plethoric but this seems a rather useful function (and trivial to maintain).

I also don't think this change belongs in this PR

pitrou · 2020-06-10T10:46:35Z

cpp/src/arrow/type_fwd.h

@@ -351,39 +358,21 @@ std::shared_ptr<DataType> ARROW_EXPORT time64(TimeUnit::type unit);
 std::shared_ptr<DataType> ARROW_EXPORT
 struct_(const std::vector<std::shared_ptr<Field>>& fields);

-/// \brief Create a UnionType instance
-std::shared_ptr<DataType> ARROW_EXPORT
-union_(const std::vector<std::shared_ptr<Field>>& child_fields,


I think we should keep this factory function, for compatibility and for convenience. UnionMode still exists, right?

UnionMode still exists but I've already factored out usage of union_. If we need it later it'll be straightforward to add

This is more of a "don't break third-party code" concern.

I'm quickly restoring backward compatibility functions

pitrou · 2020-06-10T10:47:43Z

cpp/src/arrow/type.h

  static constexpr int8_t kMaxTypeCode = 127;
  static constexpr int kInvalidChildId = -1;

-  static constexpr const char* type_name() { return "union"; }
-
-  UnionType(const std::vector<std::shared_ptr<Field>>& fields,


I'm not sure it's a good idea to remove these constructors, while external code may rely on them.

I'm not sure what you mean; UnionType is an abstract base class now and should never be constructed except as a base object of SparseUnionType or DenseUnionType. If we want to maintain compatibility of code which constructs UnionType directly then we'll need to have a single concrete UnionType which corresponds to both Type::SPARSE_UNION and Type::DENSE_UNION. That's possible but very much not our pattern with DataType subclasses.

Ah, I see. At least keep the static factory function then?

wesm · 2020-06-12T00:50:46Z

@bkietz can you rebase and address @pitrou's comments tomorrow? This is going to collide with ARROW-9075 so would prefer that this go in first and then I can rebase my patch on that

wesm

+1. Awaiting CI

bkietz · 2020-06-12T17:05:02Z

@wesm thanks, I've been trying to get those to pass locally.

wesm · 2020-06-12T17:13:41Z

Sorry about the noise I think I've got it now. We might need to create an "arrow-deprecated-test" at some point where we verify that deprecated APIs still work as advertised

wesm · 2020-06-12T18:02:44Z

Appveyor was passing two commits ago: https://ci.appveyor.com/project/BenjaminKietzman/arrow/builds/33489614. The Ursabot CI failure looks transient. I'll go ahead and merge this so I can rebase what I'm working on

wesm · 2020-06-12T18:03:31Z

thanks @bkietz!

bkietz force-pushed the 8866-Split-TypeUNION-into-Type branch from 652e5fb to bb5ce46 Compare June 8, 2020 19:01

bkietz commented Jun 8, 2020

View reviewed changes

cpp/src/arrow/c/bridge.cc Outdated Show resolved Hide resolved

bkietz force-pushed the 8866-Split-TypeUNION-into-Type branch from 8ec562a to 5eb29ef Compare June 8, 2020 21:40

bkietz force-pushed the 8866-Split-TypeUNION-into-Type branch from 5eb29ef to 290ad5b Compare June 9, 2020 16:16

pitrou reviewed Jun 10, 2020

View reviewed changes

bkietz added 4 commits June 12, 2020 10:55

ARROW-8866: [C++] Split UNION into SPARSE_UNION and DENSE_UNION

fb8fea5

lint fixes

a9922a1

fix ruby binding

5ee17ef

review comments

f1612d0

bkietz force-pushed the 8866-Split-TypeUNION-into-Type branch from 10a6b31 to f1612d0 Compare June 12, 2020 15:15

Misc changes per code reviews. Try to fix Ruby unit tests

0961cd4

wesm approved these changes Jun 12, 2020

View reviewed changes

wesm added 2 commits June 12, 2020 12:05

Add additional ARROW_DEPRECATED

d4c5b1b

Fix deprecated usages

8be24e0

wesm closed this in 89cf7bd Jun 12, 2020

bkietz deleted the 8866-Split-TypeUNION-into-Type branch February 25, 2021 16:32

asfimport mentioned this pull request Jun 12, 2020

[C++] Split Type::UNION into Type::SPARSE_UNION and Type::DENSE_UNION #25003

Closed

ARROW-8866: [C++] Split UNION into SPARSE_UNION and DENSE_UNION #7378

ARROW-8866: [C++] Split UNION into SPARSE_UNION and DENSE_UNION #7378

Uh oh!

Conversation

bkietz commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 8, 2020

Uh oh!

Uh oh!

wesm commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bkietz commented Jun 8, 2020

Uh oh!

wesm commented Jun 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wesm commented Jun 9, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wesm commented Jun 12, 2020

Uh oh!

wesm left a comment

Choose a reason for hiding this comment

Uh oh!

bkietz commented Jun 12, 2020

Uh oh!

wesm commented Jun 12, 2020

Uh oh!

wesm commented Jun 12, 2020

Uh oh!

wesm commented Jun 12, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bkietz commented Jun 8, 2020 •

edited

Loading

wesm commented Jun 8, 2020 •

edited

Loading

wesm commented Jun 9, 2020 •

edited

Loading

pitrou Jun 12, 2020 •

edited

Loading