[C++][Parquet] Use RLE for boolean type by default when parquet version is 2.x

[wgtmac]

Describe the enhancement requested

As discussed in https://issues.apache.org/jira/browse/PARQUET-2222, it would be reasonable to switch the default to use RLE for boolean values if the parquet version is 2.x.

Component(s)

C++, Parquet

[mapleFU]

Emm change default encoding is a bit weird in arrow system, because when encoding is not set, it will always use PLAIN.

It's lucky that is when Dictionary enabled and fallback to boolean, we don't need to considering it because currently we don't allow dict boolean

[wgtmac]

I agree it is tricky. I thought it would be easy to make the code change but actually it is more dirty than I think.

[pitrou]

Can you explain why it would be dirty?

[wgtmac]

Can you explain why it would be dirty?

As @mapleFU has explained, the default encoding in the WriterProperties is PLAIN for all columns. Adding an extra default encoding for boolean type require us to check 1) data page version, 2) column type, 3) user does not explicitly set an encoding in the WriterProperties. So the only chance to do this is in the WriterProperties::Builder::build(). We need to iterate all columns when data page version is V2 and replace encoding from PLAIN to RLE for all boolean columns which user does not provide an explicit encoding.

[pitrou]

Ah, it's a pity it's inflexible like that.

But we could instead store a std::optional<Encoding::type> in WriterProperties (or Encoding::UNKNOWN, but that's less pretty). Then if the encoding hasn't been set to the user, the writer can choose its concrete value depending on the column type, and whether it's a v1 or v2 data page.

[wgtmac]

Yes, that's an option I have also considered. If you are good with std::optional<Encoding::type> approach, I can go ahead to implement it.

[pitrou]

Yes, I think that would allow for more flexibility in the future.

[mapleFU]

I've seen a piece in arrow-rs:

// ----------------------------------------------------------------------
// Encoding support for column writer.
// This mirrors parquet-mr default encodings for writes. See:
// https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java
// https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java

/// Trait to define default encoding for types, including whether or not the type
/// supports dictionary encoding.
trait EncodingWriteSupport {
    /// Returns true if dictionary is supported for column writer, false otherwise.
    fn has_dictionary_support(props: &WriterProperties) -> bool;
}

/// Returns encoding for a column when no other encoding is provided in writer properties.
fn fallback_encoding(kind: Type, props: &WriterProperties) -> Encoding {
    match (kind, props.writer_version()) {
        (Type::BOOLEAN, WriterVersion::PARQUET_2_0) => Encoding::RLE,
        (Type::INT32, WriterVersion::PARQUET_2_0) => Encoding::DELTA_BINARY_PACKED,
        (Type::INT64, WriterVersion::PARQUET_2_0) => Encoding::DELTA_BINARY_PACKED,
        (Type::BYTE_ARRAY, WriterVersion::PARQUET_2_0) => Encoding::DELTA_BYTE_ARRAY,
        (Type::FIXED_LEN_BYTE_ARRAY, WriterVersion::PARQUET_2_0) => {
            Encoding::DELTA_BYTE_ARRAY
        }
        _ => Encoding::PLAIN,
    }
}

/// Returns true if dictionary is supported for column writer, false otherwise.
fn has_dictionary_support(kind: Type, props: &WriterProperties) -> bool {
    match (kind, props.writer_version()) {
        // Booleans do not support dict encoding and should use a fallback encoding.
        (Type::BOOLEAN, _) => false,
        // Dictionary encoding was not enabled in PARQUET 1.0
        (Type::FIXED_LEN_BYTE_ARRAY, WriterVersion::PARQUET_1_0) => false,
        (Type::FIXED_LEN_BYTE_ARRAY, WriterVersion::PARQUET_2_0) => true,
        _ => true,
    }
}

Should we make all of these consistent?

[wgtmac]

That sounds good. Let me check.