Skip to content

[Fix](inverted index) fix missed array inverted index null bitmap #38907 #38933

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 2 commits into from
Aug 6, 2024
Merged

[Fix](inverted index) fix missed array inverted index null bitmap #38907 #38933

airborne12 merged 2 commits into from
Aug 6, 2024

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Aug 6, 2024

cherry pick from #38907

@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap (ap…
6aa2303 
…ache#38907)

We forgot to set null bitmap in inverted index for array values.
In this PR, we added it.
@airborne12
Copy link
Member Author

airborne12 commented Aug 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@airborne12
Copy link
Member Author

airborne12 commented Aug 6, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-H: Total hot run time: 50583 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 495e6d801ba58b975d624dee2a8ac291967fe481, data reload: false

------ Round 1 ----------------------------------
q1	17538	4422	4359	4359
q2	2076	162	146	146
q3	10254	1914	1924	1914
q4	10100	1271	1338	1271
q5	8524	3963	3917	3917
q6	236	125	128	125
q7	2063	1616	1624	1616
q8	9315	2750	2729	2729
q9	10804	10727	10739	10727
q10	8677	3515	3507	3507
q11	422	248	250	248
q12	478	298	310	298
q13	18340	3984	4039	3984
q14	354	337	326	326
q15	504	446	456	446
q16	680	580	575	575
q17	1148	1001	969	969
q18	7213	6949	6939	6939
q19	1791	1677	1642	1642
q20	549	340	288	288
q21	4515	4117	4124	4117
q22	546	440	442	440
Total cold run time: 116127 ms
Total hot run time: 50583 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4306	4301	4301
q2	326	227	231	227
q3	4173	4187	4158	4158
q4	2763	2763	2773	2763
q5	7108	7204	7069	7069
q6	245	118	117	117
q7	3254	2944	2870	2870
q8	4371	4474	4501	4474
q9	17090	16817	16842	16817
q10	4202	4421	4292	4292
q11	752	705	670	670
q12	1025	861	868	861
q13	6780	3724	3759	3724
q14	464	414	432	414
q15	507	463	453	453
q16	741	682	675	675
q17	3931	3934	3859	3859
q18	8767	8819	8743	8743
q19	1757	1746	1661	1661
q20	2374	2135	2114	2114
q21	8599	8542	8446	8446
q22	1043	983	1018	983
Total cold run time: 84602 ms
Total hot run time: 79691 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TeamCity be ut coverage result:
Function Coverage: 37.74% (8102/21467)
Line Coverage: 29.37% (66344/225871)
Region Coverage: 28.88% (34241/118548)
Branch Coverage: 24.77% (17597/71040)
Coverage Report: http://coverage.selectdb-in.cc/coverage/495e6d801ba58b975d624dee2a8ac291967fe481_495e6d801ba58b975d624dee2a8ac291967fe481/report/index.html

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-DS: Total hot run time: 204438 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 495e6d801ba58b975d624dee2a8ac291967fe481, data reload: false

query1	927	390	407	390
query2	6520	2806	2777	2777
query3	6924	215	205	205
query4	20451	17871	17912	17871
query5	19743	6541	6550	6541
query6	295	237	232	232
query7	4167	303	319	303
query8	432	420	441	420
query9	3133	2674	2584	2584
query10	440	300	279	279
query11	11223	10715	10687	10687
query12	128	78	79	78
query13	5616	688	677	677
query14	17770	13899	13495	13495
query15	363	239	251	239
query16	6471	285	271	271
query17	1701	1467	881	881
query18	2288	418	418	418
query19	215	149	149	149
query20	83	81	84	81
query21	191	100	96	96
query22	5213	5020	5003	5003
query23	32646	31974	31939	31939
query24	6937	6561	6456	6456
query25	511	441	426	426
query26	531	164	163	163
query27	1885	300	295	295
query28	6128	2374	2315	2315
query29	3100	2873	2653	2653
query30	243	170	164	164
query31	915	732	749	732
query32	66	58	60	58
query33	393	260	268	260
query34	856	476	469	469
query35	1158	910	944	910
query36	1420	1151	1210	1151
query37	87	59	61	59
query38	3037	2895	2924	2895
query39	1385	1309	1341	1309
query40	196	97	95	95
query41	47	47	43	43
query42	96	86	86	86
query43	744	691	721	691
query44	1133	722	728	722
query45	247	237	232	232
query46	1237	961	979	961
query47	1883	1801	1680	1680
query48	1025	708	715	708
query49	613	370	368	368
query50	879	630	631	630
query51	4762	4676	4678	4676
query52	98	79	82	79
query53	449	332	315	315
query54	2649	2444	2470	2444
query55	94	79	87	79
query56	244	229	211	211
query57	1245	1047	1218	1047
query58	228	202	219	202
query59	4156	4123	3925	3925
query60	225	214	210	210
query61	97	96	99	96
query62	904	502	458	458
query63	495	344	350	344
query64	2532	1534	1513	1513
query65	3655	3548	3569	3548
query66	822	387	378	378
query67	16557	16192	15841	15841
query68	8451	647	651	647
query69	559	354	346	346
query70	1781	1393	1531	1393
query71	396	308	316	308
query72	6579	3474	3463	3463
query73	730	306	316	306
query74	6378	5904	5889	5889
query75	5149	3692	3679	3679
query76	4945	1141	1168	1141
query77	720	260	254	254
query78	12468	11757	19613	11757
query79	6562	619	630	619
query80	916	401	402	401
query81	501	236	238	236
query82	363	98	101	98
query83	184	132	130	130
query84	265	77	71	71
query85	994	341	340	340
query86	343	287	285	285
query87	3204	2996	3050	2996
query88	4337	2280	2292	2280
query89	356	338	303	303
query90	1891	205	209	205
query91	171	144	140	140
query92	60	53	57	53
query93	996	547	562	547
query94	750	213	212	212
query95	1127	1098	1083	1083
query96	623	322	327	322
query97	6426	6395	6420	6395
query98	191	169	172	169
query99	2478	884	857	857
Total cold run time: 303798 ms
Total hot run time: 204438 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

ClickBench: Total hot run time: 31.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 495e6d801ba58b975d624dee2a8ac291967fe481, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.25	0.04	0.05
query4	1.79	0.06	0.06
query5	0.54	0.53	0.52
query6	1.23	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.54	0.49	0.48
query10	0.53	0.54	0.54
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.83
query15	0.79	0.76	0.76
query16	0.39	0.36	0.38
query17	1.02	1.00	1.02
query18	0.24	0.24	0.24
query19	1.92	1.82	1.84
query20	0.01	0.01	0.02
query21	15.47	0.58	0.60
query22	1.94	1.92	1.91
query23	17.40	1.08	0.92
query24	4.65	1.23	1.16
query25	0.41	0.14	0.05
query26	0.55	0.16	0.15
query27	0.05	0.04	0.04
query28	8.05	0.74	0.71
query29	12.60	2.33	2.32
query30	0.59	0.55	0.56
query31	2.82	0.39	0.39
query32	3.35	0.49	0.49
query33	3.06	3.07	3.10
query34	15.27	4.81	4.76
query35	4.86	4.84	4.84
query36	1.04	1.02	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.51 s
Total hot run time: 31.31 s

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 495e6d801ba58b975d624dee2a8ac291967fe481 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@airborne12 airborne12 merged commit 13bd2bd into apache:branch-2.0 Aug 6, 2024
@airborne12 airborne12 deleted the pick_38907_to_origin_branch-2.0 branch August 6, 2024 08:09
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@airborne12
[Fix](inverted index) fix missed array inverted index null bitmap apa…
b916db0 
…che#38907 (apache#38933)

cherry pick from apache#38907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@airborne12 @doris-robot