ARROW-972: UnionArray in pyarrow

[pcmoritz]

This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:

• Support UnionType::SPARSE
• Add doc strings

[wesm]

Can you rebase? What more work needs to be done here?

[wesm]

@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0

[pcmoritz]

@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.

[pcmoritz]

This is now ready to review

[wesm]

Made some minor comments about names and a few other things. Thank you for doing this!

[wesm]

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)

[wesm]

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes

[wesm]

May also want to assert that value_offsets has 0 null count

[pcmoritz]

done

[wesm]

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer

[pcmoritz]

done

[wesm]

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?

[pcmoritz]

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.

[wesm]

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)

[wesm]

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?

[pcmoritz]

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.

[pcmoritz]

If we don't do this and plug mode in directly below, clang complains:

[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) (((((enum  arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum  arrow::UnionMode)digits[0]));
                                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) (((((((enum  arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum  arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
                                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) (((((((((enum  arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum  arrow::UnionMode)digits[2]) << PyLong_SHIFT...
                                                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) (((enum  arrow::UnionMode)-1)*(((((enum  arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum  arrow::UnionMode)d...
                                                                                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) ((((((enum  arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum  arrow::UnionMode)digits[0])));
                                                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) (((enum  arrow::UnionMode)-1)*(((((((enum  arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum  arrow::UnionMode...
                                                                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) ((((((((enum  arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum  arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
                                                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) (((enum  arrow::UnionMode)-1)*(((((((((enum  arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum  arrow::UnionMo...
                                                                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
                            return (enum  arrow::UnionMode) ((((((((((enum  arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum  arrow::UnionMode)digits[2]) << PyLong_SHIFT...
                                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~
9 errors generated.

[pcmoritz]

@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.

[wesm]

enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly

[wesm]

Done. Will merge this once the build is green

[pcmoritz]

Great, thanks :)

[wesm]

Thanks @pcmoritz!