Skip to content

ARROW-10755: [Rust] [Parquet] Add support for writing boolean type #8790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ghost wants to merge 3 commits into from
Closed

ARROW-10755: [Rust] [Parquet] Add support for writing boolean type #8790

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
007285c
Add boolean type support for parquet writer
Nov 28, 2020
e04c0e9
Run formatting
wjones127 Nov 28, 2020
542c6bd
Fix clippy issues
wjones127 Nov 28, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions rust/parquet/src/arrow/arrow_writer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ fn write_leaves(
) -> Result<()> {
match array.data_type() {
ArrowDataType::Null
| ArrowDataType::Boolean
| ArrowDataType::Int8
| ArrowDataType::Int16
| ArrowDataType::Int32
Expand Down Expand Up @@ -263,7 +264,6 @@ fn write_leaves(
"Float16 arrays not supported".to_string(),
)),
ArrowDataType::FixedSizeList(_, _)
| ArrowDataType::Boolean
| ArrowDataType::FixedSizeBinary(_)
| ArrowDataType::Decimal(_, _)
| ArrowDataType::Union(_) => Err(ParquetError::NYI(
Expand Down Expand Up @@ -351,8 +351,13 @@ fn write_leaf(
levels.repetition.as_deref(),
)?
}
ColumnWriter::BoolColumnWriter(ref mut _typed) => {
unreachable!("Currently unreachable because data type not supported")
ColumnWriter::BoolColumnWriter(ref mut typed) => {
let array = arrow_array::BooleanArray::from(column.data());
typed.write_batch(
get_bool_array_slice(&array).as_slice(),
Some(levels.definition.as_slice()),
levels.repetition.as_deref(),
)?
}
ColumnWriter::Int64ColumnWriter(ref mut typed) => {
let array = arrow_array::Int64Array::from(column.data());
Expand Down Expand Up @@ -525,8 +530,8 @@ fn get_levels(
definition: list_def_levels,
repetition: Some(list_rep_levels),
}],
ArrowDataType::Boolean => unimplemented!(),
ArrowDataType::Int8
ArrowDataType::Boolean
| ArrowDataType::Int8
| ArrowDataType::Int16
| ArrowDataType::Int32
| ArrowDataType::Int64
Expand Down Expand Up @@ -665,6 +670,16 @@ where
values
}

fn get_bool_array_slice(array: &arrow_array::BooleanArray) -> Vec<bool> {
let mut values = Vec::with_capacity(array.len() - array.null_count());
Copy link
Contributor

@nevi-me nevi-me Nov 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic that we were using was incorrect (at least once we get to lists). #8792 fixes this, so if it gets merged first, we would need to change this to populate values correctly .

for i in 0..array.len() {
if array.is_valid(i) {
values.push(array.value(i))
}
}
values
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down Expand Up @@ -1023,9 +1038,6 @@ mod tests {
}

#[test]
#[should_panic(
Copy link
Contributor

@alamb alamb Nov 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

expected = "Attempting to write an Arrow type that is not yet implemented"
)]
fn bool_single_column() {
required_and_optional::<BooleanArray, _>(
[true, false].iter().cycle().copied().take(SMALL_SIZE),
Expand Down