Skip to content

[Fix](inverted index) fix missed array inverted index null bitmap #38907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Aug 6, 2024
Merged

[Fix](inverted index) fix missed array inverted index null bitmap #38907

airborne12 merged 1 commit into from
Aug 6, 2024

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Aug 6, 2024

Proposed changes

We forgot to set null bitmap in inverted index for array values.
In this PR, we added it.

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

airborne12 commented Aug 6, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

qidaye
qidaye approved these changes Aug 6, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-H: Total hot run time: 42119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fe671d3b4ea33ce0ef64b5353b7a14a992c8caa, data reload: false

------ Round 1 ----------------------------------
q1	18866	4204	4169	4169
q2	3097	204	200	200
q3	11188	1258	1345	1258
q4	11199	863	964	863
q5	8245	3048	2982	2982
q6	224	140	141	140
q7	1059	605	616	605
q8	9447	1872	1989	1872
q9	8524	6618	6595	6595
q10	8725	3844	3841	3841
q11	417	245	250	245
q12	414	228	228	228
q13	17758	2941	2943	2941
q14	279	244	246	244
q15	524	474	487	474
q16	483	403	385	385
q17	969	929	920	920
q18	8134	7590	7591	7590
q19	1418	1241	1237	1237
q20	593	344	355	344
q21	5375	4708	4859	4708
q22	350	278	285	278
Total cold run time: 117288 ms
Total hot run time: 42119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4112	4099	4127	4099
q2	330	225	225	225
q3	2996	2995	3014	2995
q4	1934	1878	1866	1866
q5	5238	5249	5246	5246
q6	217	130	130	130
q7	2071	1664	1691	1664
q8	3207	3282	3266	3266
q9	8328	8282	8310	8282
q10	3771	3858	3846	3846
q11	552	458	445	445
q12	722	584	573	573
q13	9764	2931	2991	2931
q14	281	258	258	258
q15	525	472	481	472
q16	450	394	398	394
q17	1766	1712	1713	1712
q18	7973	7349	7301	7301
q19	1699	1672	1699	1672
q20	1962	1764	1751	1751
q21	5518	5247	5335	5247
q22	510	453	455	453
Total cold run time: 63926 ms
Total hot run time: 54828 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-DS: Total hot run time: 169016 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fe671d3b4ea33ce0ef64b5353b7a14a992c8caa, data reload: false

query1	918	382	371	371
query2	6491	1741	1680	1680
query3	6688	212	226	212
query4	20392	17574	17454	17454
query5	4249	529	507	507
query6	274	169	171	169
query7	4602	306	295	295
query8	257	198	204	198
query9	8533	2405	2386	2386
query10	436	282	269	269
query11	10614	10008	10007	10007
query12	145	90	87	87
query13	1642	379	380	379
query14	9429	7265	7529	7265
query15	215	170	163	163
query16	7072	440	464	440
query17	962	564	564	564
query18	1814	286	282	282
query19	196	150	152	150
query20	97	88	86	86
query21	209	99	98	98
query22	4093	3925	3931	3925
query23	33553	32751	33134	32751
query24	10372	3021	2997	2997
query25	726	402	427	402
query26	1741	155	154	154
query27	2861	280	280	280
query28	6858	1997	1987	1987
query29	1287	440	434	434
query30	293	154	152	152
query31	948	737	782	737
query32	103	55	60	55
query33	720	336	327	327
query34	911	472	492	472
query35	870	732	732	732
query36	1114	865	865	865
query37	283	83	83	83
query38	2840	2768	2761	2761
query39	891	819	803	803
query40	292	118	117	117
query41	53	47	49	47
query42	120	101	107	101
query43	476	447	433	433
query44	1233	737	749	737
query45	209	182	179	179
query46	1085	812	815	812
query47	1791	1709	1708	1708
query48	372	297	295	295
query49	1204	440	446	440
query50	920	450	437	437
query51	6726	6636	6680	6636
query52	102	96	92	92
query53	262	187	182	182
query54	707	441	465	441
query55	78	77	76	76
query56	271	270	250	250
query57	1141	1048	1050	1048
query58	276	278	297	278
query59	2574	2365	2344	2344
query60	304	283	287	283
query61	110	97	95	95
query62	940	658	664	658
query63	229	188	184	184
query64	5888	1938	1871	1871
query65	3169	3117	3111	3111
query66	1428	368	329	329
query67	15435	14801	14988	14801
query68	4553	560	587	560
query69	696	354	317	317
query70	1120	1090	1084	1084
query71	458	279	282	279
query72	7985	2665	2531	2531
query73	794	334	330	330
query74	6011	5641	5592	5592
query75	3756	2721	2743	2721
query76	2916	1215	1275	1215
query77	684	320	315	315
query78	9461	8981	8820	8820
query79	3017	541	536	536
query80	1144	524	515	515
query81	557	228	227	227
query82	1073	135	135	135
query83	307	173	183	173
query84	276	83	81	81
query85	1556	318	295	295
query86	474	268	301	268
query87	3278	3082	3110	3082
query88	3797	2395	2422	2395
query89	410	293	290	290
query90	1928	199	194	194
query91	126	103	102	102
query92	65	51	51	51
query93	4338	623	612	612
query94	865	277	294	277
query95	389	268	271	268
query96	615	282	284	282
query97	3255	3066	3026	3026
query98	211	195	200	195
query99	1633	1309	1284	1284
Total cold run time: 270842 ms
Total hot run time: 169016 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fe671d3b4ea33ce0ef64b5353b7a14a992c8caa, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.06
query5	0.49	0.49	0.49
query6	1.15	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.57	0.51	0.51
query10	0.56	0.56	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.61	0.61	0.59
query14	0.76	0.80	0.78
query15	0.91	0.85	0.87
query16	0.35	0.35	0.35
query17	1.00	1.02	0.99
query18	0.23	0.22	0.21
query19	1.80	1.79	1.72
query20	0.01	0.01	0.01
query21	15.40	0.74	0.65
query22	3.88	6.40	2.13
query23	17.91	1.29	1.33
query24	2.22	0.22	0.22
query25	0.18	0.08	0.08
query26	0.33	0.22	0.22
query27	0.46	0.23	0.22
query28	13.17	0.99	0.97
query29	12.54	3.29	3.22
query30	0.25	0.06	0.05
query31	2.88	0.40	0.40
query32	3.25	0.48	0.48
query33	2.97	2.95	2.99
query34	15.44	4.27	4.30
query35	4.29	4.30	4.34
query36	0.68	0.47	0.49
query37	0.19	0.16	0.17
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.17	0.13	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.45 s
Total hot run time: 30.76 s

amorynan
amorynan approved these changes Aug 6, 2024
View reviewed changes
Copy link
Contributor

@amorynan amorynan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 81ff2a0 into apache:master Aug 6, 2024
@airborne12 airborne12 deleted the fix-array branch August 6, 2024 06:06
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Aug 6, 2024
@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap (ap…
6aa2303 
…ache#38907)

We forgot to set null bitmap in inverted index for array values.
In this PR, we added it.
This was referenced Aug 6, 2024
Merged
Merged
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Aug 6, 2024
@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap (ap…
0826682 
…ache#38907)

## Proposed changes

We forgot to set null bitmap in inverted index for array values.
In this PR, we added it.
airborne12 added a commit that referenced this pull request Aug 6, 2024
@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap #38907
13bd2bd 
 (#38933)

cherry pick from #38907
airborne12 added a commit that referenced this pull request Aug 6, 2024
@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap #38907
fe6ea3b 
 (#38934)

cherry pick from #38907
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@airborne12 @dataroaring
[Fix](inverted index) fix missed array inverted index null bitmap (#3…
6ba1394 
…8907)

## Proposed changes

We forgot to set null bitmap in inverted index for array values.
In this PR, we added it.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@airborne12 @dataroaring
[Fix](inverted index) fix missed array inverted index null bitmap (#3…
d9d627e 
…8907)

## Proposed changes

We forgot to set null bitmap in inverted index for array values.
In this PR, we added it.
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap apa…
b916db0 
…che#38907 (apache#38933)

cherry pick from apache#38907
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
@airborne12 airborne12 mentioned this pull request Feb 13, 2025
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qidaye qidaye qidaye approved these changes

+1 more reviewer

@amorynan amorynan amorynan approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@airborne12 @doris-robot @amorynan @qidaye @dataroaring