Skip to content

[C++][Compute] Improve numerical stability of variances merging #26259

@asfimport

Description

@asfimport

For chunked array, variance kernel needs to merge variances.
Tested with two single value chunk, [40000008000000490], [40000008000000400]. The merged variance is 3872. If treated as single array with two values, the variance is 3904, same as numpy outputs.
So current merging method is not stable in extreme cases when chunks are very short and with approximate mean values.

Reporter: Yibo Cai / @cyb70289
Assignee: Yibo Cai / @cyb70289

PRs and other links:

Note: This issue was originally created as ARROW-10263. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions