Skip to content

[fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morningman merged 5 commits into from
Oct 21, 2024
Merged

[fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ccea5ba
fix filter_by_dict
suxiaogang223 Oct 18, 2024
5504130
add regression case
suxiaogang223 Oct 18, 2024
510a031
fix
suxiaogang223 Oct 18, 2024
ef65c9f
fix regression
suxiaogang223 Oct 18, 2024
1a564aa
fix
suxiaogang223 Oct 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions be/src/vec/exec/format/orc/vorc_reader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2018,9 +2018,9 @@ bool OrcReader::_can_filter_by_dict(int slot_id) {
// the implementation of NULL values because the dictionary itself does not contain
// NULL value encoding. As a result, many NULL-related functions or expressions
// cannot work properly, such as is null, is not null, coalesce, etc.
// Here we first disable dictionary filtering when predicate contains functions.
// Here we first disable dictionary filtering when predicate expr is not slot.
// Implementation of NULL value dictionary filtering will be carried out later.
if (expr->node_type() == TExprNodeType::FUNCTION_CALL) {
if (expr->node_type() != TExprNodeType::SLOT_REF) {
return false;
}
for (auto& child : expr->children()) {
Expand Down
4 changes: 2 additions & 2 deletions be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,9 +212,9 @@ bool RowGroupReader::_can_filter_by_dict(int slot_id,
// the implementation of NULL values because the dictionary itself does not contain
// NULL value encoding. As a result, many NULL-related functions or expressions
// cannot work properly, such as is null, is not null, coalesce, etc.
// Here we first disable dictionary filtering when predicate contains functions.
// Here we first disable dictionary filtering when predicate is not slot.
// Implementation of NULL value dictionary filtering will be carried out later.
if (expr->node_type() == TExprNodeType::FUNCTION_CALL) {
if (expr->node_type() != TExprNodeType::SLOT_REF) {
return false;
}
for (auto& child : expr->children()) {
Expand Down
12 changes: 12 additions & 0 deletions regression-test/data/external_table_p0/hive/test_string_dict_filter.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ null
-- !q14 --
null

-- !q15 --
5

-- !q01 --
3 123314 F 193846.25 1993-10-14 5-LOW Clerk#000000955 0 sly final accounts boost. carefully regular ideas cajole carefully. depos
5 44485 F 144659.20 1994-07-30 5-LOW Clerk#000000925 0 quickly. bold deposits sleep slyly. packages use slyly
Expand Down Expand Up @@ -113,6 +116,9 @@ null
-- !q14 --
null

-- !q15 --
5

-- !q01 --
3 123314 F 193846.25 1993-10-14 5-LOW Clerk#000000955 0 sly final accounts boost. carefully regular ideas cajole carefully. depos
5 44485 F 144659.20 1994-07-30 5-LOW Clerk#000000925 0 quickly. bold deposits sleep slyly. packages use slyly
Expand Down Expand Up @@ -170,6 +176,9 @@ null
-- !q14 --
null

-- !q15 --
5

-- !q01 --
3 123314 F 193846.25 1993-10-14 5-LOW Clerk#000000955 0 sly final accounts boost. carefully regular ideas cajole carefully. depos
5 44485 F 144659.20 1994-07-30 5-LOW Clerk#000000925 0 quickly. bold deposits sleep slyly. packages use slyly
Expand Down Expand Up @@ -227,3 +236,6 @@ null
-- !q14 --
null

-- !q15 --
5

6 changes: 6 additions & 0 deletions regression-test/suites/external_table_p0/hive/test_string_dict_filter.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ suite("test_string_dict_filter", "p0,external,hive,external_docker,external_dock
qt_q14 """
select * from ( select COALESCE(o_orderpriority, 'null') AS o_orderpriority from test_string_dict_filter_parquet ) as A where o_orderpriority = 'null';
"""
qt_q15 """
select count(o_orderpriority) from ( select (case when o_orderpriority = 'x' then '1' when o_orderpriority = 'y' then '2' else '0' end) as o_orderpriority from test_string_dict_filter_parquet ) as A where o_orderpriority = '0';
"""
}
def q_orc = {
qt_q01 """
Expand Down Expand Up @@ -103,6 +106,9 @@ suite("test_string_dict_filter", "p0,external,hive,external_docker,external_dock
qt_q14 """
select * from ( select COALESCE(o_orderpriority, 'null') AS o_orderpriority from test_string_dict_filter_orc ) as A where o_orderpriority = 'null';
"""
qt_q15 """
select count(o_orderpriority) from ( select (case when o_orderpriority = 'x' then '1' when o_orderpriority = 'y' then '2' else '0' end) as o_orderpriority from test_string_dict_filter_orc ) as A where o_orderpriority = '0';
"""
}
String enabled = context.config.otherConfigs.get("enableHiveTest")
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
Expand Down