Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Jul 13, 2023

Which issue does this PR close?

Resolves #6955, a bug introduced in #6904

Rationale for this change

The answer is incorrect because the starting value is always 0 and thus 0 & anything else will also be zero which is not right

What changes are included in this PR?

  1. Allow the starting value of the accumulator t be specified
  2. set the default value of the accumulator correctly for bit_and
  3. Add new tests (subset of what is in Minor: Improve aggregate test coverage more #6952)

Are these changes tested?

Yes

Are there any user-facing changes?

@github-actions github-actions bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Jul 13, 2023
@alamb alamb changed the title Alamb/fix gby bit and Fix incorrect results in BitAnd GroupsAccumulator Jul 13, 2023
@alamb alamb force-pushed the alamb/fix_gby_bit_and branch from a27881f to 6459945 Compare July 13, 2023 16:04
GROUP BY tag
ORDER BY tag
----
1 8 11 13 14 11 12 6 11 A
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before this PR, this test fails like

External error: query result mismatch:
[SQL] SELECT
  bit_and(c1),
  bit_and(c2),
  bit_and(c3),
  bit_or(c1),
  bit_or(c2),
  bit_or(c3),
  bit_xor(c1),
  bit_xor(c2),
  bit_xor(c3),
  tag
FROM bit_aggregate_functions
GROUP BY tag
ORDER BY tag
[Diff] (-expected|+actual)
-   1 8 11 13 14 11 12 6 11 A
-   33 11 NULL 33 11 NULL 33 11 NULL B
+   0 0 0 13 14 11 12 6 11 A
+   0 0 NULL 33 11 NULL 33 11 NULL B
at tests/sqllogictests/test_files/aggregate.slt:1546

DataType::Int8 => {
instantiate_primitive_accumulator!(self, Int8Type, |x, y| x
.bitand_assign(y))
instantiate_accumulator!(self, -1, Int8Type, |x, y| x.bitand_assign(y))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fix is passing in MAX / -1 here to get a bitpattern of all 1 initially

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good. It's better to leave a comment like above in the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment added in #6964

let values = values[0].as_primitive::<T>();

// update values
self.values.resize(total_num_groups, T::default_value());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously this always started at zero, now it starts at the specified value.

I think I can now reduce the code in min/max accumulator as well

@alamb alamb marked this pull request as ready for review July 13, 2023 16:06
Copy link
Contributor

@izveigor izveigor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alamb
Copy link
Contributor Author

alamb commented Jul 13, 2023

🤔 seems clippy failure is unrelated. I will fix that

@alamb
Copy link
Contributor Author

alamb commented Jul 13, 2023

Clippy fixes in #6959

@alamb alamb merged commit c07c26c into apache:main Jul 13, 2023
@alamb
Copy link
Contributor Author

alamb commented Jul 13, 2023

Thank you for the quick review @viirya and @izveigor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regression: bit_and aggregate function is incorrect

3 participants