[Fix](parquet-reader) Fix and optimize parquet min-max filtering. #38277
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
kaka11chen
force-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
96e0ee0 to
bd5d879
Compare
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| #pragma once | ||
|
|
||
| #include <gen_cpp/parquet_types.h> |
Contributor
There was a problem hiding this comment.
warning: 'gen_cpp/parquet_types.h' file not found [clang-diagnostic-error]
#include <gen_cpp/parquet_types.h>
^| ", semver=" + (version ? *version : "null") + | ||
| ", appBuildHash=" + (appBuildHash ? *appBuildHash : "null") + ")"; | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
Suggested change
| public: |
Additional context
be/src/vec/exec/format/parquet/parquet_common.h:166: previously declared here
public:
^| return Status::OK(); | ||
| } | ||
|
|
||
| int compareTo(const SemanticVersion& other) const { |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = compareIntegers(_major, other._major); cmp != 0) return cmp; | |
| if (int cmp = compareIntegers(_major, other._major); cmp != 0) { return cmp; | |
| } |
| } | ||
|
|
||
| int compareTo(const SemanticVersion& other) const { | ||
| if (int cmp = compareIntegers(_major, other._major); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = compareIntegers(_minor, other._minor); cmp != 0) return cmp; | |
| if (int cmp = compareIntegers(_minor, other._minor); cmp != 0) { return cmp; | |
| } |
|
|
||
| int compareTo(const SemanticVersion& other) const { | ||
| if (int cmp = compareIntegers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = compareIntegers(_minor, other._minor); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = compareIntegers(_patch, other._patch); cmp != 0) return cmp; | |
| if (int cmp = compareIntegers(_patch, other._patch); cmp != 0) { return cmp; | |
| } |
| ParquetStatisticsTest() {} | ||
| }; | ||
|
|
||
| TEST_F(ParquetStatisticsTest, test_try_read_old_utf8_stats) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
TEST_F(ParquetStatisticsTest, test_try_read_old_utf8_stats) {
^Additional context
be/test/vec/exec/parquet/parquet_statistics_test.cpp:30: 121 lines including whitespace and comments (threshold 80)
TEST_F(ParquetStatisticsTest, test_try_read_old_utf8_stats) {
^| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #include <gtest/gtest.h> |
Contributor
There was a problem hiding this comment.
warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
#include <gtest/gtest.h>
^
Comment on lines
+24
to
+25
| namespace doris { | ||
| namespace vectorized { |
Contributor
There was a problem hiding this comment.
warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]
Suggested change
| namespace doris { | |
| namespace vectorized { | |
| namespace doris::vectorized { |
be/test/vec/exec/parquet/parquet_version_test.cpp:219:
- } // namespace vectorized
- } // namespace doris
+ } // namespace doris| namespace vectorized { | ||
| class ParquetVersionTest : public testing::Test { | ||
| public: | ||
| ParquetVersionTest() {} |
Contributor
There was a problem hiding this comment.
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
Suggested change
| ParquetVersionTest() {} | |
| ParquetVersionTest() = default; |
| ParquetVersionTest() {} | ||
| }; | ||
|
|
||
| TEST_F(ParquetVersionTest, test_version_parser) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
TEST_F(ParquetVersionTest, test_version_parser) {
^Additional context
be/test/vec/exec/parquet/parquet_version_test.cpp:30: 91 lines including whitespace and comments (threshold 80)
TEST_F(ParquetVersionTest, test_version_parser) {
^
TPC-H: Total hot run time: 39895 ms |
TPC-DS: Total hot run time: 174183 ms |
ClickBench: Total hot run time: 31.61 s |
kaka11chen
force-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
bd5d879 to
e1d5ba2
Compare
Contributor
Author
|
run buildall |
kaka11chen
force-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
e1d5ba2 to
1ce8ced
Compare
Contributor
Author
|
run buildall |
kaka11chen
force-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
1ce8ced to
b42b73a
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39838 ms |
TPC-DS: Total hot run time: 173837 ms |
ClickBench: Total hot run time: 30.21 s |
Contributor
Author
|
run buildall |
| } | ||
|
|
||
| int SemanticVersion::compare_to(const SemanticVersion& other) const { | ||
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | |
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) { return cmp; | |
| } |
|
|
||
| int SemanticVersion::compare_to(const SemanticVersion& other) const { | ||
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; | |
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) { return cmp; | |
| } |
| int SemanticVersion::compare_to(const SemanticVersion& other) const { | ||
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) return cmp; | |
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) { return cmp; | |
| } |
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) return cmp; | ||
| if (int cmp = _compare_booleans(other._prerelease, _prerelease); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_booleans(other._prerelease, _prerelease); cmp != 0) return cmp; | |
| if (int cmp = _compare_booleans(other._prerelease, _prerelease); cmp != 0) { return cmp; | |
| } |
| std::vector<NumberOrString> _identifiers; | ||
| }; | ||
|
|
||
| private: |
Contributor
There was a problem hiding this comment.
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
Suggested change
| private: |
Additional context
be/src/vec/exec/format/parquet/parquet_common.h:218: previously declared here
private:
^
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41555 ms |
TPC-DS: Total hot run time: 168434 ms |
ClickBench: Total hot run time: 29.91 s |
kaka11chen
force-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
7cc53a3 to
e8f82a9
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 42085 ms |
TPC-DS: Total hot run time: 169444 ms |
ClickBench: Total hot run time: 29.73 s |
kaka11chen
force-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
af7382c to
7ebb634
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41876 ms |
TPC-DS: Total hot run time: 168872 ms |
ClickBench: Total hot run time: 30.16 s |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 14, 2024
…ache#38277) ## Proposed changes Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
kaka11chen
added a commit
to kaka11chen/doris
that referenced
this pull request
Aug 14, 2024
…ache#38277) ## Proposed changes Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
yiguolei
pushed a commit
that referenced
this pull request
Aug 15, 2024
kaka11chen
added a commit
to kaka11chen/doris
that referenced
this pull request
Oct 9, 2024
…ache#38277) Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
morningman
pushed a commit
that referenced
this pull request
Oct 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Refer to trino's implementation
Some bugs in the historical version paquet-mr. Use
CorruptStatistics::should_ignore_statistics()to handle.The old version of parquet uses
minandmaxstats, and later implementsmin_valueandmax_value.Min/maxstats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values.If it is double or float, special cases such as NaN, -0, and 0 must be handled.
If the string type only has min and max stats, but no min_value or max_value, use
ParquetPredicate::_try_read_old_utf8_stats()to expand the range reading optimization method for optimization.