GH-43944: [C++][Parquet] Add support for arrow::ArrayStatistics: non zero-copy int based types #43945

kou · 2024-09-04T08:36:43Z

Rationale for this change

Statistics is useful for fast processing.

Target types:

UInt8
Int8
UInt16
Int16
UInt32
UInt64
Date32
Time32
Time64
Duration

What changes are included in this PR?

Map ColumnChunkMetaData information to arrow::ArrayStatistics.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

GitHub Issue: [C++][Parquet] Add support for arrow::ArrayStatistics: non zero-copy int based types #43944

…: non zero-copy int based types Target types: * `UInt8` * `Int8` * `UInt16` * `Int16` * `UInt32` * `UInt64` * `Date32` * `Time32` * `Time64` * `Duration`

github-actions · 2024-09-04T08:37:15Z

⚠️ GitHub issue #43944 has been automatically assigned in GitHub to PR creator.

mapleFU · 2024-09-04T08:41:58Z

cpp/src/parquet/arrow/reader_internal.cc

+      array_statistics->is_min_exact = true;
+      array_statistics->is_max_exact = true;


Can you add correspond comment here? This might be a bit tricky

Sure. We should document about the discussion at #43595 (comment) , right?

BTW, could you share the e-mail URL for #43595 (comment) ?

I guess no, I'll send a mail to maillist to make it sure

I couldn't find it at https://lists.apache.org/list.html?dev@parquet.apache.org .

Ah, I forgot to add a writer check here. I should have set true only when a writer is Apache Parquet C++. I'll fix it.

Sorry, let me setup a discussion, generally if it's from Parquet C++, it will works. I'm a bit busy this morning preparing for my tour, I'll try to work it out this noon

No problem. Thanks!

https://lists.apache.org/thread/zfnmg5p51b7oylft5w5k4670wgkd4zv4

I've added a comment that we can always use true for integer based min/max.

I didn't need if (::arrow::internal::StartsWith(ctx->reader->metadata()->created_by(), "parquet-cpp-arrow")) for this case based on your e-mail.

FYI, I found the string and FLBA might being truncated, other types in public impl will not being truncated if exists

mapleFU · 2024-09-04T08:43:07Z

cpp/src/parquet/arrow/reader_internal.cc

+  auto array_data =
+      ::arrow::ArrayData::Make(field->type(), length, std::move(buffers), null_count);
+  auto array_statistics = std::make_shared<::arrow::ArrayStatistics>();
+  array_statistics->null_count = null_count;


The null_count for some type ( nested ) would be a bit weird, FYI

Thanks for the information.
Let's revisit it when we add support for arrow::ArrayStatistics of nested types.

wgtmac · 2024-09-05T05:59:10Z

cpp/src/parquet/arrow/reader_internal.cc

+  array_statistics->null_count = null_count;
+  auto statistics = metadata->statistics().get();
+  if (statistics) {
+    if (statistics->HasDistinctCount()) {


Do we need a separate function for the stats conversion?

I'll do it when I add more target types as the next pull request.
I'll know what is common pattern when I add more target types.

wgtmac

+1

conbench-apache-arrow · 2024-09-06T04:08:53Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 262d6f6.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

…: non zero-copy int based types (apache#43945) ### Rationale for this change Statistics is useful for fast processing. Target types: * `UInt8` * `Int8` * `UInt16` * `Int16` * `UInt32` * `UInt64` * `Date32` * `Time32` * `Time64` * `Duration` ### What changes are included in this PR? Map `ColumnChunkMetaData` information to `arrow::ArrayStatistics`. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * GitHub Issue: apache#43944 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

romankarlstetter · 2025-01-23T10:17:15Z

cpp/src/parquet/arrow/reader.cc

-    RETURN_NOT_OK(
-        TransferColumnData(record_reader_.get(), field_, descr_, ctx_->pool, &out_));
+    RETURN_NOT_OK(TransferColumnData(record_reader_.get(),
+                                     input_->column_chunk_metadata(), field_, descr_,


The call to input_->column_chunk_metadata() fails here if the list of row_groups in input_ is empty, because input_ is not yet initialized properly at this point in that case

Via row_group_metadata() -> RowGroup(-1)

Good catch!
Could you open a new issue for it?

apacheGH-43944: [C++][Parquet] Add support for arrow::ArrayStatistics…

aa47e50

…: non zero-copy int based types Target types: * `UInt8` * `Int8` * `UInt16` * `Int16` * `UInt32` * `UInt64` * `Date32` * `Time32` * `Time64` * `Duration`

kou requested a review from wgtmac as a code owner September 4, 2024 08:36

github-actions bot added Component: Parquet Component: C++ awaiting committer review Awaiting committer review labels Sep 4, 2024

mapleFU reviewed Sep 4, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 5, 2024

Add a comment why we can use true for is_{min,max}_exact

f768813

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 5, 2024

Remove a needless code

dcd3bdf

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 5, 2024

wgtmac reviewed Sep 5, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 5, 2024

wgtmac approved these changes Sep 5, 2024

View reviewed changes

kou merged commit 262d6f6 into apache:main Sep 5, 2024

kou deleted the cpp-parquet-statistics branch September 5, 2024 20:41

kou removed the awaiting changes Awaiting changes label Sep 5, 2024

kou mentioned this pull request Sep 5, 2024

[C++][Parquet] Add support for arrow::ArrayStatistics: non zero-copy int based types #43944

Closed

romankarlstetter reviewed Jan 23, 2025

View reviewed changes

github-actions bot added the awaiting committer review Awaiting committer review label Jan 23, 2025

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jan 23, 2025

romankarlstetter mentioned this pull request Jan 23, 2025

[Parquet][C++] Reading parquet with an empty list of row group indices fails #45339

Closed

		array_statistics->is_min_exact = true;
		array_statistics->is_max_exact = true;

GH-43944: [C++][Parquet] Add support for arrow::ArrayStatistics: non zero-copy int based types #43945

GH-43944: [C++][Parquet] Add support for arrow::ArrayStatistics: non zero-copy int based types #43945

Uh oh!

Conversation

kou commented Sep 4, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Sep 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrow bot commented Sep 6, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kou commented Sep 4, 2024 •

edited by github-actions bot

Loading

mapleFU Sep 5, 2024 •

edited

Loading