primitive_int64.value maybe an int32 type

[mapleFU]

➜  variant git:(2dc8bf1) xxd primitive_int32.value
00000000: 1440 e201 00                             .@...
➜  variant git:(2dc8bf1) xxd primitive_int64.value
00000000: 144e 61bc 00                             .Na..

Seems that primitive_int64.value actually contains a int32 variant value?

[mapleFU]

cc @alamb

[alamb]

Thanks @mapleFU -- I agree -- by my reading of https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#value-encoding-grammar

the first byte 0x14 is 0b00010100

• low 2 bits are 0b00 => Primitive type
• high 6 bits are 0b000101 ==> 5 (aka Int32 per the Variant basic types) table

I made this value with Spark like this:

parquet-testing/variant/regen.py

Line 66 in 2dc8bf1

INSERT INTO T VALUES ('primitive_int64', 12345678::Long::Variant);

So perhaps there is a bug in Spark

What I suggest is twofold:

1. Update the example data with the correct values of an int64 primitive
2. File a ticket in spark pointing out the problem

[mapleFU]

I'm writing test with my implemented C++ decoder, I think spark might uses "smallest type" to store integers, like:

    "primitive_int16": 1234,
    "primitive_int32": 123456,
    "primitive_int64": 12345678

1234 cannot store with int8, 123456 cannot store with int16, however 12345678 is able to stored with int32.

I didn't gothrough the test code of spark, but other data shows the same result, like "int_field": 1 uses int8 type.

[mapleFU]

Same problem happens on parsing json during insertion of object ( or parse_json )

https://github.com/apache/parquet-testing/blob/master/variant/data_dictionary.json#L49 here "double_field" is parsed as decimal4, and "timestamp_field" is stored as short_string.

[alamb]

Same problem happens on parsing json during insertion of object ( or parse_json )

https://github.com/apache/parquet-testing/blob/master/variant/data_dictionary.json#L49 here "double_field" is parsed as decimal4, and "timestamp_field" is stored as short_string.

That would make sense.

What do you suggest we do in terms of creating / fixing these example binary values? Perhaps you can use your C++ encoder to create the correct values, and we manually verify the binary values in another PR?

[mapleFU]

What do you suggest we do in terms of creating / fixing these example binary values? Perhaps you can use your C++ encoder to create the correct values, and we manually verify the binary values in another PR?

I found iceberg and parquet-java has sample implementation with Java, maybe we can try parquet-java( should we wait for it release) ?

[alamb]

I found iceberg and parquet-java has sample implementation with Java, maybe we can try parquet-java( should we wait for it release) ?

I don't think there is any reason to wait for a release as we can double check the values with the spec manually

I am not familiar enough with java ecosystem to figure out parquet-java however (and I would likely spend the time required working on a Rust implementation instead).

Another thing we could do is use the code from @PinkCrow007 in apache/arrow-rs#7452 to re-create the data.

@mapleFU are you willing to look into this? If not, I can probably find time to do it next week.

[alamb]

I have prepared a PR to fix this:

• Fix Variant int64 example binary data #85