ARROW-633/ARROW-634: Add Java support and integration test for FixedSizeBinary #1012

alphalfalfa · 2017-08-30T00:38:34Z

The changes include:

Arrow-633: [Java] Add support for FixedWidthBinary type
Arrow-634: Add integration tests for FixedSizeBinary
Fixed a bug of integration test suite that primitive case with non-zero batch sizes was ignored

I also commented on changes that needs more review.

…on tests

alphalfalfa · 2017-08-30T00:41:32Z

integration/integration_test.py

        fields.append(get_field(type_ + "_nonnullable", type_, False))

-    return _generate_file("primitive", fields, batch_sizes)
+    name = "primitive"


Added name suffix for primitive case such that primitive test case at line 823/824 become both effective. Previously the first one was overwritten by the second one and hence ignored.

alphalfalfa · 2017-08-30T00:44:27Z

java/vector/src/main/codegen/data/ValueVectorTypes.tdd

      ]
    },
+    {
+      major: "Fixed",


FixedSizeBinary has a variable width, I am not sure what integer value to put here. I went ahead and put -1 to mark it as a special case. This changes also leads to quite some changes of if statements in the codegen template FixedValueVectors.java.

I am not sure about this change. FixValueVectors have static type width that is specified in ValueVectorTypes.tdd. Having type width to be dynamic would complicated the template and result in inconsistency in the resulting classes.

Is there a reason FixedSizeBinary needs to have dynamic type witdh? i.e., can we have byteWitdh = 1 and use the type width in the Schema to retrieve and store values? This would simplify things.

cc @wesm

The bottom line is that FixedSizeBinaryVector class should have a way to access the information about the dynamic type width. The only way I found is by changing the static type width in the class to non-static and initialize in the constructor. Did I miss a better way to pass the information? I don't like to put a lot of if/elseif/else statements in a codegen template either :)

Regarding the value being set to a positive integer like 1, it is highly misleading for readers in my opinion.

I see. Yeah byteWidth = 1 wouldn't work because of how the template works. But I think we should still have typeBitWidth = 8 in the type layout in the schema to match Binary type.

@icexelloss, the VarBinary type has a width of 4 (instead of 8) which make sense because each element has a 4-byte offset. Do you think it is better to use 4 as well? Btw, VarBinary is using a different template.

Offset is 4 bytes wide. Data width is 1 byte wide.

I am talking about the type layout in the schema. Not the constant in the vector class. Sorry for the confusion.

I think we should probably refactor the templates some before introducing this change. In reality, I'd like to propose that we rework the vector hierarchy entirely and work much more like ArrowBuf and it's related parents work. In that case, we just grab a memory address and bypass all the object hierarchy for high performance methods but still use superclass delegation for non-performance critical methods.

For the ValueVector hierarchy/codegen situation, let's implement two classes:

BaseFixedVector (would take a dynamic size variable)
BaseVariableVector

These would both be non-code generated classes. Then we could generate code generated subclasses of these that only implement methods that need code generation for performance. Most of these methods would probably appear as custom methods in mutator/accessors.

Then, the changes for a new Custom fixed width vector would just be specializations of the BaseFixedWidthVector called something like FixedBinaryVector and FixedVarcharVector and in those specialization, the width in the mutators and accessors could come from a instance variable instead of a constant.

I believe this would allow us to simplify the codebase and make maintenance of the base classes much easier.

I opened up https://issues.apache.org/jira/browse/ARROW-1463 to cover this.

sounds good to me. I'll wait for the refactoring to complete and come back to modify this PR

alphalfalfa · 2017-08-30T00:48:08Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

    if (valueCount == 0) {
      return 0;
    }
-    return valueCount * ${type.width};


Since FixedSizeBinary does not have a static type.width, I replaced all ${type.width} to TYPE_WIDTH.

alphalfalfa · 2017-08-30T00:51:02Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

+  }
+
+   public void set(int index, ArrowBuf buffer){
+     assert TYPE_WIDTH <= buffer.capacity();


Is there a better way to validate an input of ArrowBuf type?

alphalfalfa · 2017-08-30T00:57:11Z

java/vector/src/main/java/org/apache/arrow/vector/schema/VectorLayout.java

        return VALUES_64;
      default:
-        throw new IllegalArgumentException("only 8, 16, 32, or 64 bits supported");
+        return new VectorLayout(DATA, typeBitWidth);


The existing C++ implemetation allows any typeBitWidth, I followed and remove this exception. Is this correct?

icexelloss · 2017-08-31T22:36:02Z

java/vector/src/main/java/org/apache/arrow/vector/schema/TypeLayout.java

      }

+      @Override
+      public TypeLayout visit(ArrowType.FixedSizeBinary type) {


This should just be newVariableWidthTypeLayout() to be consistent with Binary type?

I think it should still be FixedWidthTypeLayout because each element is fixed-size and no offset vector is necessary

Sorry, I meant the data type width should be 8-bit instead of type.getByteWidth() * 8.

VarBinary has 8-bit data type width and FixedBinary should be the same.

Actually, now I read more of the doc, I am not too sure.

https://github.com/apache/arrow/blob/master/format/Schema.fbs#L225

@wesm, can you clarify the what the correct bit_width in the vector layout should be?

icexelloss · 2017-08-31T22:38:12Z

java/vector/src/test/java/org/apache/arrow/vector/file/TestArrowFile.java

+    ByteArrayOutputStream stream = new ByteArrayOutputStream();
+    final int numValues = 10;
+    final int typeWidth = 11;
+    byte[][] byteValues = new byte[numValues][typeWidth];


Can you fix the style of this?

Sure, I assume you mean the spacing if the for loop, right?

elahrvivaz · 2017-09-04T15:19:03Z

lgtm

wesm

LGTM, @jacques-n can you have a look?

wesm · 2017-09-04T23:42:11Z

@alphalfalfa can you rebase?

icexelloss · 2017-09-05T00:08:39Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

 * ${minor.class} implements a vector of fixed width values.  Elements in the vector are accessed
 * by position, starting from the logical start of the vector.  Values should be pushed onto the
 * vector sequentially, but may be randomly accessed.
+<#if (type.width > 0) >


I think we should try to minimize the if else statement in the template to simply. I prefer we remove this if else here and state something like:

The width of each element is defined in getWidth()

icexelloss · 2017-09-05T00:11:13Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

+  <#if (type.width < 0)>
+  public final int TYPE_WIDTH;
+  <#else>
  public static final int TYPE_WIDTH = ${type.width};


I am not sure about having TYPE_WIDTH defined as static in some classes and non static in others.

I think it's cleaner if we define TYPE_WIDTH only as static and define another class method getTypeWidth()

Another reason is all caps are usually used to define static constant. Type width for fixed binary is not constant. so I think it's cleaner to not reuse the same variable for fixed binary.

icexelloss · 2017-09-05T00:21:55Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

+    <#if (type.width < 0 || type.width > 8 || minor.class == "IntervalDay")>
    public ${minor.javaType!type.javaType} get(int index) {
-      return data.slice(index * ${type.width}, ${type.width});
+      <#if (minor.class == "FixedSizeBinary")>


I think it's cleaner if we get FixedSizeBinary into it's top level:

<#if (type.width) < 0 && minor.class == "FixedSizeBinary"> public ${minor.javaType!type.javaType} get(int index) { ... } </#if>

Since it doesn't share logic with the original get method

icexelloss · 2017-09-05T00:27:37Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

   * ${minor.class}.Mutator implements a mutable vector of fixed width values.  Elements in the
   * vector are accessed by position from the logical start of the vector.  Values should be pushed
   * onto the vector sequentially, but may be randomly accessed.
+  <#if (type.width > 0)>


Same comment as above. I prefer we don't have the if else here.

icexelloss · 2017-09-05T00:56:08Z

java/vector/src/main/codegen/templates/NullableValueVectors.java


    <#assign fields = minor.fields!type.fields />
-    public void set(int index, int isSet<#list fields as field>, ${field.type} ${field.name}Field</#list> ){
+    public void set(int index, int isSet<#list fields as field><#if field.include!true >, ${field.type} ${field.name}Field</#if></#list> ){


This is a bit confusing, can you explain this logic? Seems you added field.include = false just for FixedBinaryVector, why?

icexelloss · 2017-09-05T01:01:50Z

java/vector/src/main/codegen/templates/NullableValueVectors.java

     * @param value   array of bytes (or int if smaller than 4 bytes) to write
     */
-    public void set(int index, <#if type.major == "VarLen">byte[]<#elseif (type.width < 4)>int<#else>${minor.javaType!type.javaType}</#if> value) {
+    public void set(int index, <#if type.major == "VarLen">byte[]<#elseif (type.width >= 0 && type.width < 4)>int<#else>${minor.javaType!type.javaType}</#if> value) {


Why this is not:

public void set(int index, <#if type.major == "VarLen" && minor.class == "FixedSizeBinary">byte[]<#elseif (type.width < 4)>int<#else>${minor.javaType!type.javaType}</#if> value)

Nvm, the java type for fixed size binary is byte[] so this would be equivalent.

icexelloss · 2017-09-05T01:54:59Z

java/vector/src/main/codegen/templates/FixedValueVectors.java

+   }
+
+      <#elseif minor.class == "FixedSizeBinary">
+  public void set(int index, ${minor.class}Holder holder){


These set methods looks similar to the existing ones, why do we need to have special code gen for these?

…tion tests (Updated) The original PR is at #1012. Due to the major refactoring last year, changes are big. So I created this separate PR for easier to review and will cancel the other one later. Changes include: * Arrow-633: [Java] Add support for FixedWidthBinary type * Arrow-634: Add integration tests for FixedSizeBinary Author: Jingyuan Wang <jingyuan@live.com> Author: jingyuan <jingyuan.nt@gmail.com> Closes #1492 from alphalfalfa/updated-arrow-633-634 and squashes the following commits: c994a19 [jingyuan] create data buffer layout for FixedSizeBinary directly instead of using dataBuffer() method and some other changes based on PR comments 873c5b2 [jingyuan] do not pass scale or byteWidth into writeValueToGenerator in JsonFileWriter.java 4fbb67d [jingyuan] put arbitrary values for FixSizeBinary nulls 0c0015d [Jingyuan Wang] add a new line a8a9553 [Jingyuan Wang] Pass byteWidth info for FixedSizeBinary type in JsonFileReader 1a64c5c [Jingyuan Wang] expand get() method inside getObject() method to remove duplicate check of isSet() 071fb25 [Jingyuan Wang] ARROW-633/634: Add FixedSizeBinary support in Java and integration tests

…tion tests (Updated) The original PR is at apache#1012. Due to the major refactoring last year, changes are big. So I created this separate PR for easier to review and will cancel the other one later. Changes include: * Arrow-633: [Java] Add support for FixedWidthBinary type * Arrow-634: Add integration tests for FixedSizeBinary Author: Jingyuan Wang <jingyuan@live.com> Author: jingyuan <jingyuan.nt@gmail.com> Closes apache#1492 from alphalfalfa/updated-arrow-633-634 and squashes the following commits: c994a19 [jingyuan] create data buffer layout for FixedSizeBinary directly instead of using dataBuffer() method and some other changes based on PR comments 873c5b2 [jingyuan] do not pass scale or byteWidth into writeValueToGenerator in JsonFileWriter.java 4fbb67d [jingyuan] put arbitrary values for FixSizeBinary nulls 0c0015d [Jingyuan Wang] add a new line a8a9553 [Jingyuan Wang] Pass byteWidth info for FixedSizeBinary type in JsonFileReader 1a64c5c [Jingyuan Wang] expand get() method inside getObject() method to remove duplicate check of isSet() 071fb25 [Jingyuan Wang] ARROW-633/634: Add FixedSizeBinary support in Java and integration tests

alphalfalfa added 3 commits August 29, 2017 15:28

initial draft of adding FixedSizeBinary support in Java and integrati…

d1564d7

…on tests

add unit test for FixedSizeBinaryVector

968c5e9

add unit test for FixedSizeBinaryReader/Writer

d1b2844

alphalfalfa commented Aug 30, 2017

View reviewed changes

alphalfalfa changed the title ~~WIP: ARROW-633/ARROW-634: Add Java support and integration test for FixedSizeBinary~~ ARROW-633/ARROW-634: Add Java support and integration test for FixedSizeBinary Aug 30, 2017

icexelloss reviewed Aug 31, 2017

View reviewed changes

wesm approved these changes Sep 4, 2017

View reviewed changes

icexelloss reviewed Sep 5, 2017

View reviewed changes

alphalfalfa mentioned this pull request Jan 18, 2018

ARROW-633/634: [Java] Add FixedSizeBinary support in Java and integration tests (Updated) #1492

Closed

alphalfalfa closed this Feb 7, 2018

alphalfalfa deleted the arrow-633-634 branch February 8, 2018 22:27

This was referenced Feb 7, 2018

[Java] Add support for FixedSizeBinary type #15364

Closed

[JAVA] Restructure ValueVector hierarchy to minimize compile-time generated code #17483

Closed

ARROW-633/ARROW-634: Add Java support and integration test for FixedSizeBinary #1012

ARROW-633/ARROW-634: Add Java support and integration test for FixedSizeBinary #1012

Uh oh!

Conversation

alphalfalfa commented Aug 30, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

icexelloss Aug 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jacques-n Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elahrvivaz commented Sep 4, 2017

Uh oh!

wesm left a comment

Choose a reason for hiding this comment

Uh oh!

wesm commented Sep 4, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

icexelloss Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

icexelloss Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

icexelloss Aug 31, 2017 •

edited

Loading

jacques-n Sep 5, 2017 •

edited

Loading

icexelloss Sep 5, 2017 •

edited

Loading

icexelloss Sep 5, 2017 •

edited

Loading