[Go] pqarrow - Provide a way to support Parquet Logical types that do not have or need Arrow equivalent types

[datbth]

Describe the enhancement requested

Parquet Logical types like JSON:

• Do not have a fully representable/equivalent type in Arrow, because Arrow types do not carry such "Logical type" information.
• But have equivalent physical Arrow types. For example, Parquet Logical type JSON can be represented by/converted from arrow.BinaryTypes.String.

Does it make sense to make pqarrow.NewFileWriter support those Logical types by letting the caller specify such Logical types?

Component(s)

Go

[datbth]

I just made this attempt but I'm not sure about the approach #43641

[joellubi]

Hi @datbth, sorry about the delay on this. I've been thinking about a high-level approach that would help address this issue as well as #43624, so I incorporated a potential solution into #43679.

Let me know what you think. The relevant change is the introduction of the CustomParquetType interface which allows Arrow ExtensionTypes to control the Parquet LogicalType they are written as. It should work out of the box for JSON as in your example by just constructing a JSONType extension array and sending it to the parquet writer. See the test for example usage.

This can also be used to write Parquet Interval data as well using the following approach:

• Create a new Arrow extension type with a storage type compatible with the Parquet target.
• For example it could be called MonthDayMillis with storage FixedSizeBinary<12>. This doesn't need to be upstreamed, you can maintain the type in your own codebase.
• Add the method:

func (t *myType) ParquetLogicalType() schema.LogicalType { return schema.IntervalLogicalType{} }

• Now there is zero casting or conversion of the underlying data necessary because you're writing Millis instead of Nanos in the first place, so the writer can efficiently write the underlying storage and update the LogicalType.

[datbth]

I think it is great 👍

One question: If I understand correctly, to implement such MonthDayMillis for writing to parquet, I will only need to implement ParquetLogicalType for it (while other methods like Value and MarshalJSON are not required), am I correct?

p/s If it's true that parquet Interval does not support negative intervals then I can't use it anyway 😄. But this enhancement will probably still be useful in other scenarios.

[joellubi]

One question: If I understand correctly, to implement such MonthDayMillis for writing to parquet, I will only need to implement ParquetLogicalType for it (while other methods like Value and MarshalJSON are not required), am I correct?

Correct. You are only required to implement the ExtensionType interface which is fairly simple, to which you can add ParquetLogicalType. The ExtensionArray and ExtensionBuilder interfaces are optional and just make it more convenient to read/write the array.

If you just write the underlying storage array (FixedSizeBinary<12>) you'll be appending []byte records which can be produced by getting the byte representations of the uint32 values and concatenating in the slice. The ExtensionBuilder interface can optionally be used to consolidate this logic, so that you have a simpler interface while writing e.g. bldr.Append(month, day, millis uint32).

p/s If it's true that parquet Interval does not support negative intervals then I can't use it anyway 😄. But this enhancement will probably still be useful in other scenarios.

Ah that's unfortunate! There may still be some other options, such as perhaps writing multiple columns or a struct array.