Support LZO Compression Codec in Parquet Files (NYI: "The codec type LZO is not supported yet")

[kosiew]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I am trying to use the Parquet file format with LZO compression in a Rust project using arrow-rs. However, I encountered the error NYI("The codec type LZO is not supported yet"), indicating that LZO compression is not yet implemented.

arrow-rs/parquet/src/compression.rs

Lines 194 to 195 in 7aecc3f

	CodecType::UNCOMPRESSED => Ok(None),
	_ => Err(nyi_err!("The codec type {} is not supported yet", codec)),

Describe the solution you'd like

I would like arrow-rs to include support for the LZO compression codec in its Parquet implementation. This would allow me to read and write Parquet files compressed with LZO seamlessly, aligning with the capabilities of other Parquet libraries such as PyArrow or Apache Spark.
Describe alternatives you've considered

Additional context

I encountered this issue while researching apache/datafusion-python#979
LZO is a widely used compression algorithm for its speed and efficiency, particularly in real-time and streaming applications. Adding support for LZO would enhance arrow-rs's capabilities and broaden its use cases.

[tustvold]

I had a brief look and couldn't find a Rust crate providing this compression algorithm...

[kosiew]

@tustvold , thanks for looking.
I found https://github.com/arma-tools/lzokay-native-rs, but I don't know how suitable it is for this issue.

[tustvold]

I suspect the way to support this would be to find some way to allow people to plugin in their own codecs, that way they can depend on whatever LZO implementation they choose. I don't think there are any with a sufficiently mature maintenance story that would allow us to take a dependency on them, but by providing pluggable codecs people can make their own judgements.

Edit: FWIW I thought LZ4 had largely replaced LZO, its compression is slightly worse but it is much faster to read, and much more widely adopted.

[kosiew]

LZ4 had largely replaced LZO, its compression is slightly worse but it is much faster to read, and much more widely adopted.

Closing.