[fix] (bloom filter) Fix the bloom filter calculation for date and datetime #43351

csun5285 · 2024-11-06T08:16:09Z

What problem does this PR solve?

Problem Summary:
After adding a bloom filter to the date type, queries return errors following the upgrade.

Check List (For Committer)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No colde files have been changed.
  - Other reason

Manual Test

Create a table with a date type column and a corresponding bloom filter.
Upgrade to the 2.1
Queries are correct.

Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
Release note

None

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-06T08:16:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

csun5285 · 2024-11-06T08:16:44Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/olap/comparison_predicate.h

csun5285 · 2024-11-06T08:48:24Z

run buildall

doris-robot · 2024-11-06T08:48:58Z

TeamCity be ut coverage result:
Function Coverage: 37.88% (9848/26000)
Line Coverage: 29.03% (81887/282034)
Region Coverage: 28.26% (42170/149242)
Branch Coverage: 24.84% (21397/86132)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cd558242382ec350b76ecd1c308145aba05af812_cd558242382ec350b76ecd1c308145aba05af812/report/index.html

github-actions · 2024-11-06T08:52:38Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-06T09:13:49Z

TeamCity be ut coverage result:
Function Coverage: 37.88% (9850/26000)
Line Coverage: 29.03% (81880/282034)
Region Coverage: 28.26% (42181/149242)
Branch Coverage: 24.85% (21404/86132)
Coverage Report: http://coverage.selectdb-in.cc/coverage/06cda32cf864f2653527c5f04a8617541ad09fcd_06cda32cf864f2653527c5f04a8617541ad09fcd/report/index.html

doris-robot · 2024-11-06T09:28:15Z

TPC-H: Total hot run time: 41185 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06cda32cf864f2653527c5f04a8617541ad09fcd, data reload: false

------ Round 1 ----------------------------------
q1	17581	7427	7255	7255
q2	2077	172	167	167
q3	10593	1087	1154	1087
q4	10559	897	857	857
q5	7768	3060	3030	3030
q6	238	144	142	142
q7	996	601	598	598
q8	9369	1982	2025	1982
q9	6619	6422	6390	6390
q10	7123	2392	2431	2392
q11	478	264	256	256
q12	411	215	210	210
q13	17797	2984	2977	2977
q14	244	217	208	208
q15	587	520	499	499
q16	686	579	592	579
q17	976	628	554	554
q18	7335	6783	6632	6632
q19	1327	1018	1112	1018
q20	475	182	174	174
q21	3910	3226	3181	3181
q22	1094	1009	997	997
Total cold run time: 108243 ms
Total hot run time: 41185 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7239	7407	7239
q2	347	248	256	248
q3	2985	2975	3015	2975
q4	2116	1905	1782	1782
q5	5738	5752	5795	5752
q6	233	141	144	141
q7	2313	1844	1817	1817
q8	3451	3514	3504	3504
q9	8906	8897	8818	8818
q10	3596	3530	3547	3530
q11	590	503	500	500
q12	825	635	621	621
q13	9716	3182	3185	3182
q14	312	272	282	272
q15	608	570	554	554
q16	680	630	640	630
q17	1862	1646	1585	1585
q18	8323	7880	7571	7571
q19	1697	1599	1547	1547
q20	2097	1848	1870	1848
q21	5597	5493	5423	5423
q22	1140	1075	1046	1046
Total cold run time: 70422 ms
Total hot run time: 60585 ms

doris-robot · 2024-11-06T09:40:34Z

TPC-DS: Total hot run time: 193979 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06cda32cf864f2653527c5f04a8617541ad09fcd, data reload: false

query1	2253	2078	2092	2078
query2	6241	2129	2054	2054
query3	14698	11010	231	231
query4	33269	23801	23666	23666
query5	3487	434	420	420
query6	261	172	173	172
query7	3981	288	289	288
query8	285	230	229	229
query9	9588	2702	2683	2683
query10	453	251	249	249
query11	17947	15228	15420	15228
query12	155	108	101	101
query13	1575	415	427	415
query14	9931	6540	7553	6540
query15	250	184	180	180
query16	8188	475	468	468
query17	1594	604	585	585
query18	2502	634	619	619
query19	400	188	180	180
query20	130	115	115	115
query21	227	103	106	103
query22	5042	4471	4256	4256
query23	34950	34804	34435	34435
query24	9860	3242	3282	3242
query25	531	400	395	395
query26	727	176	177	176
query27	1911	280	280	280
query28	6536	2411	2428	2411
query29	681	423	413	413
query30	404	318	319	318
query31	1039	805	810	805
query32	96	54	55	54
query33	606	281	271	271
query34	910	519	519	519
query35	863	707	719	707
query36	1066	923	958	923
query37	121	69	77	69
query38	4567	4309	4262	4262
query39	1535	1420	1420	1420
query40	203	98	99	98
query41	48	47	46	46
query42	111	96	96	96
query43	554	495	488	488
query44	1197	800	812	800
query45	183	167	168	167
query46	1141	697	701	697
query47	1950	1852	1874	1852
query48	402	321	316	316
query49	892	403	387	387
query50	821	402	382	382
query51	7337	7149	7051	7051
query52	99	93	89	89
query53	252	179	180	179
query54	680	422	390	390
query55	75	76	73	73
query56	259	240	231	231
query57	1255	1200	1127	1127
query58	219	204	194	194
query59	3220	2918	2977	2918
query60	262	255	249	249
query61	106	111	111	111
query62	840	671	658	658
query63	215	187	184	184
query64	3669	679	597	597
query65	3303	3174	3269	3174
query66	843	291	294	291
query67	16107	15888	15718	15718
query68	4464	601	587	587
query69	424	247	248	247
query70	1179	1068	1122	1068
query71	343	249	242	242
query72	6056	3980	4060	3980
query73	789	349	357	349
query74	10166	8915	8986	8915
query75	3397	2689	2669	2669
query76	2275	1047	1151	1047
query77	379	284	269	269
query78	10493	9425	9472	9425
query79	1201	586	601	586
query80	990	428	431	428
query81	557	243	241	241
query82	1297	113	116	113
query83	246	165	167	165
query84	238	71	78	71
query85	1256	298	280	280
query86	392	303	286	286
query87	4835	4615	4623	4615
query88	3429	2181	2137	2137
query89	422	287	283	283
query90	1942	183	182	182
query91	140	106	102	102
query92	61	47	49	47
query93	1471	530	532	530
query94	892	294	280	280
query95	359	239	238	238
query96	613	279	283	279
query97	2893	2712	2722	2712
query98	212	192	197	192
query99	1691	1297	1291	1291
Total cold run time: 300205 ms
Total hot run time: 193979 ms

doris-robot · 2024-11-06T09:46:04Z

ClickBench: Total hot run time: 32.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06cda32cf864f2653527c5f04a8617541ad09fcd, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.07
query4	1.64	0.11	0.11
query5	0.41	0.38	0.40
query6	1.16	0.67	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.49	0.48
query10	0.55	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.13	0.11
query13	0.60	0.61	0.59
query14	2.72	2.73	2.71
query15	0.91	0.82	0.83
query16	0.38	0.38	0.38
query17	1.05	1.08	1.06
query18	0.20	0.20	0.20
query19	1.97	1.87	1.86
query20	0.01	0.01	0.01
query21	15.36	0.59	0.60
query22	2.94	2.36	1.92
query23	16.82	0.98	0.80
query24	2.58	2.23	0.93
query25	0.25	0.13	0.08
query26	0.49	0.14	0.13
query27	0.04	0.05	0.05
query28	10.17	1.10	1.09
query29	12.60	3.25	3.24
query30	0.25	0.06	0.06
query31	2.89	0.38	0.38
query32	3.27	0.46	0.45
query33	2.98	3.00	3.00
query34	16.99	4.44	4.45
query35	4.57	4.51	4.49
query36	0.68	0.51	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 32.59 s

code format add ut add ut

csun5285 · 2024-11-08T11:27:46Z

run buildall

github-actions

clang-tidy made some suggestions

be/test/olap/date_bloom_filter_test.cpp

doris-robot · 2024-11-08T12:06:40Z

TeamCity be ut coverage result:
Function Coverage: 37.93% (9871/26027)
Line Coverage: 29.11% (82316/282783)
Region Coverage: 28.26% (42371/149934)
Branch Coverage: 24.80% (21448/86488)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d1308f8599de369e0df4a75745dec2e6b37ca587_d1308f8599de369e0df4a75745dec2e6b37ca587/report/index.html

csun5285 · 2024-11-08T12:20:02Z

run buildall

doris-robot · 2024-11-08T12:57:50Z

TeamCity be ut coverage result:
Function Coverage: 37.92% (9869/26027)
Line Coverage: 29.10% (82295/282783)
Region Coverage: 28.25% (42354/149934)
Branch Coverage: 24.80% (21445/86488)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d07e45639ecfa464062bcf1c84556b84c270ccf7_d07e45639ecfa464062bcf1c84556b84c270ccf7/report/index.html

mrhhsg

LGTM

github-actions · 2024-11-11T07:21:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-11T07:21:34Z

PR approved by anyone and no changes requested.

airborne12

LGTM

…tetime (#43351) ### What problem does this PR solve? Problem Summary: After adding a bloom filter to the date type, queries return errors following the upgrade. Co-authored-by: csun5285 <sunchenyang@selectdb.com>

…tetime (apache#43351) ### What problem does this PR solve? Problem Summary: After adding a bloom filter to the date type, queries return errors following the upgrade. Co-authored-by: csun5285 <sunchenyang@selectdb.com>

… date and datetime (#43612) Cherry-picked from #43351 Co-authored-by: csun5285 <sunchenyang@selectdb.com>

…tetime (#43351) (#43622) pick from master #43351 --------- Co-authored-by: csun5285 <sunchenyang@selectdb.com>

…ter index (#46961) Problem Summary: #43351 fix date/datetime v1 support in comparison predicate, this PR try to fix it in inlist predicate.

…ter index (apache#46961) Problem Summary: apache#43351 fix date/datetime v1 support in comparison predicate, this PR try to fix it in inlist predicate.

github-actions bot reviewed Nov 6, 2024

View reviewed changes

be/src/olap/comparison_predicate.h Outdated Show resolved Hide resolved

csun5285 force-pushed the fix_bf branch from cd55824 to 06cda32 Compare November 6, 2024 08:48

fix

d1308f8

code format add ut add ut

csun5285 force-pushed the fix_bf branch from 06cda32 to d1308f8 Compare November 8, 2024 11:27

github-actions bot reviewed Nov 8, 2024

View reviewed changes

be/test/olap/date_bloom_filter_test.cpp Show resolved Hide resolved

be/test/olap/date_bloom_filter_test.cpp Outdated Show resolved Hide resolved

fix

d07e456

mrhhsg approved these changes Nov 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 11, 2024

github-actions bot added the reviewed label Nov 11, 2024

airborne12 approved these changes Nov 11, 2024

View reviewed changes

airborne12 added the dev/3.0.x label Nov 11, 2024

airborne12 merged commit e93b7ba into apache:master Nov 11, 2024

github-actions bot mentioned this pull request Nov 11, 2024

branch-3.0: [fix] (bloom filter) Fix the bloom filter calculation for date and datetime #43612

Merged

airborne12 added the dev/2.1.x label Nov 11, 2024

csun5285 mentioned this pull request Nov 11, 2024

[fix] (bloom filter) Fix the bloom filter calculation for date and datetime (#43351) #43622

Merged

airborne12 pushed a commit that referenced this pull request Nov 12, 2024

branch-3.0: [fix] (bloom filter) Fix the bloom filter calculation for…

60f17d8

… date and datetime (#43612) Cherry-picked from #43351 Co-authored-by: csun5285 <sunchenyang@selectdb.com>

airborne12 added dev/3.0.3-merged and removed dev/3.0.x labels Nov 12, 2024

airborne12 pushed a commit that referenced this pull request Nov 12, 2024

[fix] (bloom filter) Fix the bloom filter calculation for date and da…

15f85e2

…tetime (#43351) (#43622) pick from master #43351 --------- Co-authored-by: csun5285 <sunchenyang@selectdb.com>

airborne12 added dev/2.1.8-merged and removed dev/2.1.x labels Nov 12, 2024

airborne12 mentioned this pull request Jan 14, 2025

[fix](bloomfilter) fix inlist support for date/datetimev1 in bloomfilter index #46961

Merged

16 tasks

[fix] (bloom filter) Fix the bloom filter calculation for date and datetime #43351

[fix] (bloom filter) Fix the bloom filter calculation for date and datetime #43351

Uh oh!

Conversation

csun5285 commented Nov 6, 2024 • edited by airborne12 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Committer)

Manual Test

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

csun5285 commented Nov 6, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

csun5285 commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

github-actions bot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

csun5285 commented Nov 8, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Nov 8, 2024

Uh oh!

csun5285 commented Nov 8, 2024

Uh oh!

doris-robot commented Nov 8, 2024

Uh oh!

mrhhsg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

github-actions bot commented Nov 11, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

csun5285 commented Nov 6, 2024 •

edited by airborne12

Loading