[Java] Unable to read unsigned integers outside signed range for bit width in integration tests

[asfimport]

I believe this was introduced recently (perhaps in the refactors), but there was a problem where the integration tests weren't being properly run that hid the error from us

see #1294 (comment)

Reporter: Wes McKinney / @wesm
Assignee: Micah Kornfield / @emkornfield

Original Issue Attachments:

• generated_primitive.json

PRs and other links:

• GitHub Pull Request #4432

_{Note: This issue was originally created as ARROW-1837. Please see the migration documentation for further details.}

[asfimport]

Jingyuan Wang / @alphalfalfa:
Wes mentioned in the GitHub issue page:

------ 

I found the problem – one of the primitive integration test files was being clobbered and not run, which was suppressing a failure that should have been raised a long time ago.

In the meantime, there was also a regression from the Java refactor, and we are no longer able to fully read unsigned integer types anymore. I will hack the integration tests for now and open a JIRA about fixing,

here's an example of trying to read a uint16 vector:

16:49:51.051 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8 Error accessing files Numeric value (65350) out of range of Java short at [Source: /tmp/tmpwgopllpl/generated_primitive.json; line: 1111, column: 18] 16:49:51.065 [main] ERROR org.apache.arrow.tools.Integration - Error accessing files com.fasterxml.jackson.core.JsonParseException: Numeric value (65350) out of range of Java short 

[asfimport]

Jingyuan Wang / @alphalfalfa:
a more complete stack trace

Error accessing files
Numeric value (65350) out of range of Java short
 at [Source: /tmp/tmp7z9ijgqc/generated_primitive.json; line: 783, column: 18]
17:46:40.350 [main] ERROR org.apache.arrow.tools.Integration - Error accessing files
com.fasterxml.jackson.core.JsonParseException: Numeric value (65350) out of range of Java short
 at [Source: /tmp/tmp7z9ijgqc/generated_primitive.json; line: 783, column: 18]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
	at com.fasterxml.jackson.core.JsonParser.getShortValue(JsonParser.java:1075)
	at org.apache.arrow.vector.ipc.JsonFileReader$BufferHelper$3.read(JsonFileReader.java:248)
	at org.apache.arrow.vector.ipc.JsonFileReader$BufferReader.readBuffer(JsonFileReader.java:199)
	at org.apache.arrow.vector.ipc.JsonFileReader.readIntoBuffer(JsonFileReader.java:491)
	at org.apache.arrow.vector.ipc.JsonFileReader.readFromJsonIntoVector(JsonFileReader.java:539)
	at org.apache.arrow.vector.ipc.JsonFileReader.read(JsonFileReader.java:180)
	at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:196)
	at org.apache.arrow.tools.Integration.run(Integration.java:114)
	at org.apache.arrow.tools.Integration.main(Integration.java:65)
	Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1148)
Allocator(ROOT) 0/1148/1148/2147483647 (res/actual/peak/limit)

		at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:406)
		at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:220)
		... 2 common frames omitted

[asfimport]

Jingyuan Wang / @alphalfalfa:
In the defined Java Vector types, there are two 16-bit types: SmallInt and UInt2. The first one is associated with signed short (Java does not support unsigned short) and the latter is associated with char/Character. I need to look at the code a bit more to understand how these two types are determined to use. 

[asfimport]

Wes McKinney / @wesm:
Since Java doesn't have unsigned integer values, I presume that we'll want to return them in getObject as a larger signed integer object for now – as long as unsigned values sent from other applications can be understood correctly. So a UInt2Vector would return int values, UInt4Vector return long values, and so forth (would UInt8 have to return BigInteger?) 

[asfimport]

Jingyuan Wang / @alphalfalfa:
For Java 8 & 9, Integer and Long can partially handle unsigned values, but looks highly inconvenient. The example I found is to assign the value by parsing a string. 

I agree with you on your proposal of promoting unsigned values to to a wider type when retrieved. But I also think this might require other Java developers their opinions as currently UInt2Vector return char/Character. I am not sure about if this vector has any necessary use cases. If so, it is at least confusing if not wrong. 

[asfimport]

Jingyuan Wang / @alphalfalfa:
@icexelloss, any opinion on whether UInt2Vector should return char/Character? 

[asfimport]

Li Jin / @icexelloss:
We can check with Dremio folks whether those are being used. If not we are free to change the them. cc @siddharthteotia

[asfimport]

Wes McKinney / @wesm:
Moving this to 0.10.0. Maybe we can deal with unsigned integers in Java in the next release cycle

[asfimport]

Wes McKinney / @wesm:
@jacques-n @siddharthteotia any thoughts on what would be involved with better supporting unsigned integers in Java? I think they need to be supported. I'm moving this issue to 0.11 so it does not block 0.10

[asfimport]

Wes McKinney / @wesm:
[~emkornfield@gmail.com] if you are interested in unsigned integers this would benefit from some attention

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 4432
#4432