Setting "longEncoding":"auto" and using any CompressionStrategy value for metricCompression other than None on IndexSpec can produce incorrect/broken segments due to a flaw in the interaction between BlockLayoutColumnarLongsSerializer, the TableLongEncodingWriter and DeltaLongEncodingWriter implementations, and the Size1Ser, Size2Ser, and Mult4Ser implementations of VSizeLongSerde.
The issue is that flushing a chunk of values to the output stream in BlockLayoutColumnarLongs calls flush on the LongEncodingWriter, which for Table/DeltaLongEncodingWriter calls close on the open VSizeLongSerde serializer. All serializer implementations in VSizeLongSerde have a close implementation that is as follows
@Override
public void close() throws IOException
{
if (closed) {
return;
}
...
closed = true;
}
but Size1Ser and Mult4Ser also rely on close to flush any remaining values that were being accumulated depending on the row count and bits per value of the block. However, the lifetime of the inner VSizeLongSerde serializer object is the same as BlockLayoutColumnarLongsSerializer and LongEncodingWriter, causing all blocks beyond the first one which is written to potentially be missing one or more final values, which would have been flushed by the close method if the serializer wasn't already closed. The VSizeLongSerde serializer in the 'closed' state will still continue to correctly encode most of the values to the buffer for all remaining chunks because write doesn't check closed, it just may leave off the end of them because the flushing logic will never be hit again.
This is not a problem with EntireLayoutColumnarLongs because it only flushes the writer once, and not a problem with LongLongEncodingWriter because it's flush method does nothing and always writes complete values. Similarly, Mult8Ser of VSizeLongSerde also always write complete values, so also will not run into this issue.
PR with a fix incoming as soon as I write a test for this condition.
Setting
"longEncoding":"auto"and using anyCompressionStrategyvalue formetricCompressionother thanNoneonIndexSpeccan produce incorrect/broken segments due to a flaw in the interaction betweenBlockLayoutColumnarLongsSerializer, theTableLongEncodingWriterandDeltaLongEncodingWriterimplementations, and theSize1Ser,Size2Ser, andMult4Serimplementations ofVSizeLongSerde.The issue is that flushing a chunk of values to the output stream in
BlockLayoutColumnarLongscallsflushon theLongEncodingWriter, which forTable/DeltaLongEncodingWritercallscloseon the openVSizeLongSerdeserializer. All serializer implementations inVSizeLongSerdehave acloseimplementation that is as followsbut
Size1SerandMult4Seralso rely on close to flush any remaining values that were being accumulated depending on the row count and bits per value of the block. However, the lifetime of the innerVSizeLongSerdeserializer object is the same asBlockLayoutColumnarLongsSerializerandLongEncodingWriter, causing all blocks beyond the first one which is written to potentially be missing one or more final values, which would have been flushed by theclosemethod if the serializer wasn't already closed. TheVSizeLongSerdeserializer in the 'closed' state will still continue to correctly encode most of the values to the buffer for all remaining chunks becausewritedoesn't check closed, it just may leave off the end of them because the flushing logic will never be hit again.This is not a problem with
EntireLayoutColumnarLongsbecause it only flushes the writer once, and not a problem withLongLongEncodingWriterbecause it'sflushmethod does nothing and always writes complete values. Similarly,Mult8SerofVSizeLongSerdealso always write complete values, so also will not run into this issue.PR with a fix incoming as soon as I write a test for this condition.