GH-44010: [C++] Add `arrow::RecordBatch::MakeStatisticsArray()` #44252

kou · 2024-09-30T08:08:18Z

Rationale for this change

Statistics schema for Arrow C data interface (GH-43553) is complex because it uses nested types (struct, map and union). So reusable implementation to make statistics array is useful.

What changes are included in this PR?

arrow::RecordBatch::MakeStatisticsArray() is a convenient function that converts arrow::ArrayStatistics in a arrow::RecordBatch to arrow::Array for the Arrow C data interface.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

GitHub Issue: [C++] Add a convenient function that converts arrow::ArrayStatistics to arrow::Array for the Arrow C data interface #44010

github-actions · 2024-09-30T08:08:51Z

⚠️ GitHub issue #44010 has been automatically assigned in GitHub to PR creator.

cpp/src/arrow/compare.cc

cpp/src/arrow/record_batch.cc

kou · 2024-10-02T05:48:21Z

@pitrou @ianmcook What do you think about this?

Statistics schema https://github.com/apache/arrow/pull/43553/files#diff-f3758fb6986ea8d24bb2e13c2feb625b68bbd6b93b3fbafd3e2a03dcdc7ba263R86-R95 is compact but it may be complex to build. Because it uses many nested types.

mapleFU · 2024-10-30T06:40:17Z

cpp/src/arrow/array/statistics.h

I may forgot a bit but we don't distinct "bytes" and "utf8" in stats?

Ah, we didn't discuss it...
Let's discuss it in #44579.

We can assume "utf8" here for now.

We can add a // TODO(GH-44579) here?

Yes. I should have added it...
I've added it.

Oh, I forgot to push the commit... I pushed now.

mapleFU · 2024-10-30T06:41:37Z

cpp/src/arrow/c/abi.h

Don't know constexpr std::string_view is better or this is better

We can't use constexpr because this header may be used by C programs.

mapleFU · 2024-10-30T06:45:05Z

cpp/src/arrow/record_batch.cc

So RowCount is also handled as a stats 🤔?

Statistics array will be passed to consumer before consumer receives a record batch.
So this may be useful for consumer.

But DuckDB doesn't have row count in its BaseStatistics...: https://github.com/duckdb/duckdb/blob/670cd341249e266de384e0341f200f4864b41b27/src/include/duckdb/storage/statistics/base_statistics.hpp#L38-L146
This may not be useful...

Let's keep this for now to demonstrate table/record batch level statistics.

cpp/src/arrow/record_batch.cc

mapleFU

Will take a careful pass tonight

mapleFU · 2024-11-04T06:13:18Z

cpp/src/arrow/array/statistics.h

We can add a // TODO(GH-44579) here?

mapleFU

General LGTM but I'm not an expert on C ABI and data layer

mapleFU · 2024-11-04T14:50:33Z

cpp/src/arrow/record_batch.cc

So actually this is logically a "set" prepared for items?

Right.

If there are the same types, the first type is only used.

mapleFU · 2024-11-04T14:55:56Z

cpp/src/arrow/record_batch.cc

So actually this is for a two-phase building, one pass for types, and one-pass for data?

Right.

I think that it's one of complexities.
So I sent https://lists.apache.org/thread/0c9jftkspvj7yw1lpo73s3vtp6vfjqv8 to the mailing list. But nobody agreed it. So this complexity will be acceptable...

It's a convenient function that converts `arrow::ArrayStatistics` in a `arrow::RecordBatch` to `arrow::Array` for the Arrow C data interface.

kou

I'll merge this in a few days is nobody objects this.

kou · 2024-11-06T22:01:52Z

cpp/src/arrow/array/statistics.h

Oh, I forgot to push the commit... I pushed now.

conbench-apache-arrow · 2024-11-09T04:22:02Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit d748ace.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

…dBatch::MakeStatisticsArray()`'s docstring (#45588) ### Rationale for this change `arrow::RecordBatch::MakeStatisticsArray()`'s docstring uses https://arrow.apache.org/docs/format/CDataInterfaceStatistics.html not https://arrow.apache.org/docs/format/StatisticsSchema.html for statistics schema URL. Because #44252 assumed that we use #43553 but we use #45058 finally. ### What changes are included in this PR? Fix URL. ### Are these changes tested? It does not need since just a correction in document ### Are there any user-facing changes? No, Just a correction in document * GitHub Issue: #45587 Authored-by: arash andishgar <arashandishgar1@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…dBatch::MakeStatisticsArray()`'s docstring (#45588) ### Rationale for this change `arrow::RecordBatch::MakeStatisticsArray()`'s docstring uses https://arrow.apache.org/docs/format/CDataInterfaceStatistics.html not https://arrow.apache.org/docs/format/StatisticsSchema.html for statistics schema URL. Because apache/arrow#44252 assumed that we use apache/arrow#43553 but we use apache/arrow#45058 finally. ### What changes are included in this PR? Fix URL. ### Are these changes tested? It does not need since just a correction in document ### Are there any user-facing changes? No, Just a correction in document * GitHub Issue: #45587 Authored-by: arash andishgar <arashandishgar1@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions bot added Component: C++ awaiting committer review Awaiting committer review labels Sep 30, 2024

kou commented Sep 30, 2024

View reviewed changes

cpp/src/arrow/compare.cc Outdated Show resolved Hide resolved

cpp/src/arrow/record_batch.cc Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 30, 2024

kou force-pushed the cpp-record-batch-make-statistics-array branch from 903e3f4 to 92afc83 Compare September 30, 2024 08:17

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 30, 2024

kou force-pushed the cpp-record-batch-make-statistics-array branch from 92afc83 to b194430 Compare September 30, 2024 09:03

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 30, 2024

kou force-pushed the cpp-record-batch-make-statistics-array branch from 5a00c48 to 12b1a97 Compare October 30, 2024 06:32

mapleFU reviewed Oct 30, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 30, 2024

kou force-pushed the cpp-record-batch-make-statistics-array branch from 8e4d618 to 9c529d1 Compare October 31, 2024 23:48

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 31, 2024

mapleFU reviewed Nov 4, 2024

View reviewed changes

cpp/src/arrow/array/statistics.h Outdated

Copy link

Member

mapleFU Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add a // TODO(GH-44579) here?

mapleFU reviewed Nov 4, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Nov 4, 2024

kou force-pushed the cpp-record-batch-make-statistics-array branch from 9c529d1 to ab80bf9 Compare November 6, 2024 22:00

kou added 3 commits November 7, 2024 07:00

apacheGH-44010: [C++] Add RecordBatch::MakeStatisticsArray()

0787eeb

It's a convenient function that converts `arrow::ArrayStatistics` in a `arrow::RecordBatch` to `arrow::Array` for the Arrow C data interface.

Add static_cast

79b9915

Add explicit cast

7d87603

kou added 4 commits November 7, 2024 07:00

Extract as a function

397a7af

Add comments

b173bd1

Set start_new_column = false after yielding a table level statistics

bda0f34

Add apacheGH-44579 reference

e93d0f4

kou force-pushed the cpp-record-batch-make-statistics-array branch from ab80bf9 to e93d0f4 Compare November 6, 2024 22:00

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Nov 6, 2024

kou commented Nov 6, 2024

View reviewed changes

cpp/src/arrow/array/statistics.h Outdated

Copy link

Member Author

kou Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I forgot to push the commit... I pushed now.

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Nov 6, 2024

kou merged commit d748ace into apache:main Nov 8, 2024

kou deleted the cpp-record-batch-make-statistics-array branch November 8, 2024 20:37

kou removed the awaiting changes Awaiting changes label Nov 8, 2024

kou mentioned this pull request Nov 8, 2024

[C++] Add a convenient function that converts arrow::ArrayStatistics to arrow::Array for the Arrow C data interface #44010

Closed

kou mentioned this pull request Feb 20, 2025

GH-45587: [C++][Docs] Fix the statistics schema link in arrow::RecordBatch::MakeStatisticsArray()'s docstring #45588

Merged

GH-44010: [C++] Add arrow::RecordBatch::MakeStatisticsArray() #44252

GH-44010: [C++] Add arrow::RecordBatch::MakeStatisticsArray() #44252

Uh oh!

Conversation

kou commented Sep 30, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Sep 30, 2024

Uh oh!

Uh oh!

Uh oh!

kou commented Oct 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrow bot commented Nov 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GH-44010: [C++] Add `arrow::RecordBatch::MakeStatisticsArray()` #44252

GH-44010: [C++] Add `arrow::RecordBatch::MakeStatisticsArray()` #44252

kou commented Sep 30, 2024 •

edited by github-actions bot

Loading