[fix](array index) Correct null bitmap writing for inverted index #47846

airborne12 · 2025-02-12T14:18:41Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #38907

Problem Summary:
This pull request focuses on improving the handling of null values in the inverted index writer and simplifying the codebase by removing redundant null map checks. The most important changes include removing unnecessary null map handling in several methods and ensuring proper null bitmap updates.

Improvements to null value handling and code simplification:

be/src/olap/rowset/segment_v2/column_writer.cpp: Removed redundant null map handling in Status ArrayColumnWriter::append_nullable method.
be/src/olap/rowset/segment_v2/inverted_index_writer.cpp: Simplified add_array_nulls method to always return Status::OK().
be/src/olap/rowset/segment_v2/inverted_index_writer.cpp: Added null map check before accessing elements in the loop to prevent potential null pointer dereference. [1] [2]
be/src/olap/rowset/segment_v2/inverted_index_writer.cpp: Updated _null_bitmap in the add_null_document method to ensure proper null bitmap updates.
be/src/olap/task/index_builder.cpp: Removed redundant null map handling in Status IndexBuilder::_add_nullable method.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-02-12T14:19:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2025-02-12T14:20:00Z

run buildall

doris-robot · 2025-02-12T14:58:50Z

TPC-H: Total hot run time: 31516 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3bb8bac8c0978fd4dd1de31c1d4de0efa0e692b, data reload: false

------ Round 1 ----------------------------------
q1	17576	5166	5063	5063
q2	2050	305	177	177
q3	10569	1257	728	728
q4	10212	998	530	530
q5	7504	2431	2317	2317
q6	181	161	131	131
q7	905	755	613	613
q8	9665	1391	1142	1142
q9	5018	4889	4628	4628
q10	7169	2347	1907	1907
q11	487	280	264	264
q12	354	367	226	226
q13	18346	3652	3075	3075
q14	226	235	203	203
q15	513	446	473	446
q16	622	624	571	571
q17	587	843	351	351
q18	7027	6200	6107	6107
q19	1217	950	562	562
q20	319	333	201	201
q21	3025	2190	1954	1954
q22	377	338	320	320
Total cold run time: 103949 ms
Total hot run time: 31516 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5193	5117	5094	5094
q2	234	328	249	249
q3	2151	2687	2307	2307
q4	1439	1814	1348	1348
q5	4254	4135	4188	4135
q6	208	171	126	126
q7	1872	1896	1798	1798
q8	2610	2602	2571	2571
q9	7204	7124	7184	7124
q10	3031	3226	2787	2787
q11	582	510	497	497
q12	715	778	582	582
q13	3455	3881	3223	3223
q14	276	303	282	282
q15	504	461	475	461
q16	649	689	646	646
q17	1117	1589	1354	1354
q18	7705	7307	7212	7212
q19	820	827	825	825
q20	1950	2040	1930	1930
q21	5445	4895	4745	4745
q22	619	565	548	548
Total cold run time: 52033 ms
Total hot run time: 49844 ms

doris-robot · 2025-02-12T15:10:51Z

TPC-DS: Total hot run time: 189594 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3bb8bac8c0978fd4dd1de31c1d4de0efa0e692b, data reload: false

query1	1304	950	902	902
query2	6192	1814	1813	1813
query3	11045	4579	4390	4390
query4	53556	24807	22932	22932
query5	5233	560	488	488
query6	346	207	181	181
query7	4991	501	291	291
query8	328	245	225	225
query9	6060	2534	2535	2534
query10	435	299	245	245
query11	15271	15124	14720	14720
query12	156	108	109	108
query13	1122	504	385	385
query14	10378	6751	6920	6751
query15	206	211	176	176
query16	7021	649	476	476
query17	1048	718	557	557
query18	1572	406	316	316
query19	210	201	161	161
query20	126	123	124	123
query21	210	129	119	119
query22	4473	4460	4509	4460
query23	33969	33558	33347	33347
query24	5887	2435	2430	2430
query25	498	498	433	433
query26	707	273	160	160
query27	1759	492	336	336
query28	2876	2435	2381	2381
query29	562	545	417	417
query30	210	186	152	152
query31	918	865	878	865
query32	70	61	60	60
query33	441	358	301	301
query34	756	862	498	498
query35	815	828	770	770
query36	946	975	933	933
query37	120	97	75	75
query38	4202	4266	4156	4156
query39	1506	1410	1436	1410
query40	204	119	107	107
query41	52	48	47	47
query42	115	109	108	108
query43	492	490	501	490
query44	1320	803	790	790
query45	186	189	175	175
query46	885	1063	664	664
query47	1873	1884	1756	1756
query48	385	413	305	305
query49	702	533	416	416
query50	742	754	424	424
query51	4278	4274	4276	4274
query52	111	109	96	96
query53	231	257	191	191
query54	482	472	410	410
query55	84	82	80	80
query56	264	293	276	276
query57	1155	1197	1102	1102
query58	257	258	249	249
query59	2714	2748	2652	2652
query60	303	279	279	279
query61	119	141	190	141
query62	754	763	661	661
query63	223	193	198	193
query64	1526	1033	680	680
query65	3299	3139	3114	3114
query66	833	404	289	289
query67	15847	15567	15497	15497
query68	6022	780	500	500
query69	526	294	267	267
query70	1191	1085	1114	1085
query71	439	290	264	264
query72	5944	3479	3749	3479
query73	1234	732	350	350
query74	8925	9217	9057	9057
query75	3154	3141	2678	2678
query76	3950	1170	729	729
query77	545	388	270	270
query78	10123	10119	9390	9390
query79	2629	784	581	581
query80	657	517	434	434
query81	515	267	239	239
query82	320	130	98	98
query83	173	169	153	153
query84	289	95	78	78
query85	797	344	311	311
query86	356	293	258	258
query87	4402	4470	4430	4430
query88	3625	2159	2164	2159
query89	404	307	277	277
query90	1711	191	188	188
query91	146	133	107	107
query92	72	59	52	52
query93	2553	990	567	567
query94	697	405	301	301
query95	342	258	246	246
query96	474	562	266	266
query97	2747	2854	2765	2765
query98	233	208	205	205
query99	1310	1425	1267	1267
Total cold run time: 293466 ms
Total hot run time: 189594 ms

doris-robot · 2025-02-12T15:16:09Z

ClickBench: Total hot run time: 30.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3bb8bac8c0978fd4dd1de31c1d4de0efa0e692b, data reload: false

query1	0.04	0.03	0.05
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.42	0.42	0.41
query6	1.15	0.67	0.66
query7	0.03	0.02	0.01
query8	0.05	0.03	0.03
query9	0.59	0.51	0.54
query10	0.57	0.58	0.57
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.60	0.61
query14	2.83	2.70	2.82
query15	0.92	0.86	0.85
query16	0.37	0.38	0.36
query17	1.02	1.03	1.00
query18	0.22	0.19	0.20
query19	1.92	1.77	2.04
query20	0.02	0.01	0.02
query21	15.35	0.87	0.53
query22	0.77	1.17	0.74
query23	14.91	1.37	0.61
query24	6.67	2.12	0.71
query25	0.50	0.27	0.15
query26	0.58	0.17	0.15
query27	0.05	0.05	0.04
query28	9.50	0.83	0.42
query29	12.53	3.89	3.26
query30	0.25	0.09	0.06
query31	2.83	0.58	0.38
query32	3.21	0.55	0.47
query33	2.95	3.03	2.96
query34	15.81	5.14	4.57
query35	4.57	4.54	4.58
query36	0.66	0.48	0.46
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.62 s
Total hot run time: 30.46 s

doris-robot · 2025-02-12T15:19:11Z

TeamCity be ut coverage result:
Function Coverage: 42.95% (11300/26312)
Line Coverage: 32.90% (94945/288597)
Region Coverage: 32.06% (48672/151830)
Branch Coverage: 27.91% (24550/87946)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c3bb8bac8c0978fd4dd1de31c1d4de0efa0e692b_c3bb8bac8c0978fd4dd1de31c1d4de0efa0e692b/report/index.html

airborne12 · 2025-02-13T01:38:47Z

run buildall

doris-robot · 2025-02-13T02:17:15Z

TPC-H: Total hot run time: 31709 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cba1cc3fbe7c067791f653b7e432fdd510f3dc8, data reload: false

------ Round 1 ----------------------------------
q1	17608	5193	5131	5131
q2	2047	307	158	158
q3	10577	1272	743	743
q4	10238	995	560	560
q5	7666	2398	2334	2334
q6	187	163	134	134
q7	910	756	591	591
q8	9292	1274	1013	1013
q9	4899	4848	4799	4799
q10	6856	2309	1869	1869
q11	521	269	247	247
q12	347	353	225	225
q13	17762	3655	3060	3060
q14	223	223	223	223
q15	516	465	460	460
q16	616	625	578	578
q17	575	870	335	335
q18	6847	6127	6178	6127
q19	1413	957	560	560
q20	316	318	184	184
q21	2773	2608	2067	2067
q22	380	345	311	311
Total cold run time: 102569 ms
Total hot run time: 31709 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5201	5198	5172	5172
q2	254	333	229	229
q3	2217	2766	2401	2401
q4	1506	1864	1399	1399
q5	4351	4264	4286	4264
q6	214	169	130	130
q7	1927	1882	1791	1791
q8	2635	2697	2678	2678
q9	7187	7134	7135	7134
q10	3016	3222	2787	2787
q11	562	502	485	485
q12	696	760	619	619
q13	3381	3978	3330	3330
q14	284	301	270	270
q15	492	477	473	473
q16	633	685	665	665
q17	1146	1590	1363	1363
q18	7549	7259	7249	7249
q19	805	822	983	822
q20	1957	2067	1886	1886
q21	5418	4988	4662	4662
q22	659	609	552	552
Total cold run time: 52090 ms
Total hot run time: 50361 ms

doris-robot · 2025-02-13T02:29:19Z

TPC-DS: Total hot run time: 190122 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cba1cc3fbe7c067791f653b7e432fdd510f3dc8, data reload: false

query1	1309	974	931	931
query2	6247	1910	1906	1906
query3	10953	4417	4409	4409
query4	57095	25469	23527	23527
query5	5104	501	493	493
query6	353	193	180	180
query7	4933	504	291	291
query8	310	247	234	234
query9	6002	2548	2552	2548
query10	429	301	244	244
query11	15096	14970	14800	14800
query12	155	127	110	110
query13	1061	496	371	371
query14	10129	6514	6384	6384
query15	229	211	194	194
query16	7142	627	443	443
query17	1065	719	567	567
query18	1614	420	304	304
query19	205	189	157	157
query20	123	137	121	121
query21	214	130	105	105
query22	4382	4329	4201	4201
query23	33921	33387	33209	33209
query24	6298	2448	2426	2426
query25	487	457	410	410
query26	701	266	163	163
query27	2221	498	332	332
query28	3318	2418	2422	2418
query29	571	553	444	444
query30	207	190	157	157
query31	878	862	874	862
query32	72	64	62	62
query33	434	389	313	313
query34	777	875	521	521
query35	824	871	787	787
query36	957	1003	917	917
query37	117	97	80	80
query38	4285	4337	4240	4240
query39	1489	1473	1502	1473
query40	232	121	109	109
query41	58	55	55	55
query42	132	109	109	109
query43	504	545	513	513
query44	1291	807	803	803
query45	181	185	169	169
query46	887	1085	686	686
query47	1822	1886	1777	1777
query48	390	422	313	313
query49	727	570	419	419
query50	711	740	439	439
query51	4233	4258	4229	4229
query52	109	109	105	105
query53	241	263	207	207
query54	494	496	427	427
query55	85	87	77	77
query56	273	269	259	259
query57	1170	1175	1102	1102
query58	249	249	234	234
query59	2932	2900	2821	2821
query60	297	293	266	266
query61	122	116	117	116
query62	797	778	657	657
query63	235	200	195	195
query64	1801	1016	669	669
query65	3323	3164	3132	3132
query66	704	394	293	293
query67	15882	15538	15356	15356
query68	5401	766	509	509
query69	512	306	275	275
query70	1204	1132	1115	1115
query71	438	286	260	260
query72	6278	3800	3706	3706
query73	1266	749	346	346
query74	9104	9152	8787	8787
query75	3464	3143	2722	2722
query76	3705	1173	733	733
query77	543	361	279	279
query78	10047	10153	9334	9334
query79	1300	873	582	582
query80	582	552	465	465
query81	492	285	242	242
query82	205	128	180	128
query83	168	169	152	152
query84	288	95	70	70
query85	722	347	299	299
query86	322	314	282	282
query87	4498	4511	4571	4511
query88	2823	2222	2183	2183
query89	411	322	281	281
query90	1821	196	201	196
query91	140	133	110	110
query92	63	60	52	52
query93	1618	997	581	581
query94	635	409	303	303
query95	350	271	261	261
query96	518	560	282	282
query97	2783	2863	2722	2722
query98	227	204	202	202
query99	1287	1397	1281	1281
Total cold run time: 294804 ms
Total hot run time: 190122 ms

doris-robot · 2025-02-13T02:34:36Z

ClickBench: Total hot run time: 30.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cba1cc3fbe7c067791f653b7e432fdd510f3dc8, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.09	0.10
query5	0.40	0.41	0.40
query6	1.16	0.66	0.66
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.62	0.52	0.51
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.70	2.72	2.71
query15	0.92	0.85	0.85
query16	0.38	0.37	0.38
query17	1.01	1.06	1.02
query18	0.22	0.19	0.20
query19	1.93	1.84	2.04
query20	0.01	0.01	0.02
query21	15.37	0.92	0.54
query22	0.75	1.06	0.69
query23	15.01	1.36	0.63
query24	7.87	1.09	0.49
query25	0.51	0.23	0.15
query26	0.66	0.16	0.13
query27	0.05	0.04	0.04
query28	9.34	0.83	0.43
query29	12.56	4.04	3.36
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.22	0.55	0.47
query33	2.98	3.00	3.04
query34	15.71	5.11	4.52
query35	4.54	4.54	4.55
query36	0.68	0.49	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.71 s
Total hot run time: 30.45 s

doris-robot · 2025-02-13T02:36:07Z

TeamCity be ut coverage result:
Function Coverage: 42.95% (11301/26312)
Line Coverage: 32.90% (94952/288597)
Region Coverage: 32.06% (48679/151830)
Branch Coverage: 27.91% (24549/87946)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9cba1cc3fbe7c067791f653b7e432fdd510f3dc8_9cba1cc3fbe7c067791f653b7e432fdd510f3dc8/report/index.html

airborne12 · 2025-02-13T15:35:21Z

run buildall

doris-robot · 2025-02-13T15:56:31Z

TPC-H: Total hot run time: 31527 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 090765eac0df34a3149f22722edd761268f215c8, data reload: false

------ Round 1 ----------------------------------
q1	17595	5223	5070	5070
q2	2054	317	190	190
q3	10364	1319	734	734
q4	10221	1015	552	552
q5	7525	2366	2396	2366
q6	182	164	134	134
q7	892	739	641	641
q8	9306	1247	1090	1090
q9	5036	4525	4668	4525
q10	6816	2310	1894	1894
q11	479	284	264	264
q12	347	367	216	216
q13	17767	3625	3093	3093
q14	230	232	206	206
q15	504	459	474	459
q16	626	606	564	564
q17	592	895	352	352
q18	6634	6165	6232	6165
q19	1210	951	570	570
q20	318	335	200	200
q21	2951	2158	1931	1931
q22	374	341	311	311
Total cold run time: 102023 ms
Total hot run time: 31527 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5152	5494	5143	5143
q2	240	319	228	228
q3	2172	2687	2280	2280
q4	1472	1844	1405	1405
q5	4197	4132	4170	4132
q6	215	167	122	122
q7	1873	1819	1676	1676
q8	2629	2624	2570	2570
q9	7384	7108	7122	7108
q10	3059	3160	2748	2748
q11	568	516	486	486
q12	661	742	625	625
q13	3585	3874	3284	3284
q14	283	298	275	275
q15	505	467	462	462
q16	635	652	623	623
q17	1142	1577	1367	1367
q18	7489	7218	7190	7190
q19	807	902	998	902
q20	1950	2031	1842	1842
q21	5432	4928	4776	4776
q22	665	587	547	547
Total cold run time: 52115 ms
Total hot run time: 49791 ms

doris-robot · 2025-02-13T16:07:54Z

TPC-DS: Total hot run time: 183450 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 090765eac0df34a3149f22722edd761268f215c8, data reload: false

query1	976	376	382	376
query2	6518	1882	1858	1858
query3	6795	216	210	210
query4	26535	23726	23112	23112
query5	5095	663	482	482
query6	310	201	183	183
query7	4607	503	297	297
query8	300	243	234	234
query9	8645	2583	2570	2570
query10	511	314	258	258
query11	15681	15184	15013	15013
query12	170	108	107	107
query13	1651	516	385	385
query14	11182	6654	6346	6346
query15	253	200	178	178
query16	7796	648	447	447
query17	1526	709	559	559
query18	2008	396	281	281
query19	260	173	150	150
query20	117	113	114	113
query21	205	119	104	104
query22	4221	4271	4254	4254
query23	33649	32768	32955	32768
query24	6823	2396	2381	2381
query25	487	459	382	382
query26	1210	272	193	193
query27	2264	465	326	326
query28	4223	2410	2397	2397
query29	696	542	413	413
query30	233	185	157	157
query31	939	847	829	829
query32	76	72	63	63
query33	560	349	306	306
query34	775	850	479	479
query35	810	817	733	733
query36	984	976	888	888
query37	128	94	81	81
query38	4104	4064	4242	4064
query39	1448	1401	1421	1401
query40	215	114	101	101
query41	54	52	52	52
query42	124	101	110	101
query43	496	515	489	489
query44	1282	795	774	774
query45	174	170	163	163
query46	863	1053	632	632
query47	1744	1776	1710	1710
query48	376	413	308	308
query49	806	541	438	438
query50	705	742	420	420
query51	4196	4235	4144	4144
query52	109	110	98	98
query53	227	251	189	189
query54	491	486	416	416
query55	93	84	79	79
query56	301	278	274	274
query57	1141	1129	1051	1051
query58	254	241	247	241
query59	2692	2887	2627	2627
query60	291	279	274	274
query61	145	141	160	141
query62	786	747	656	656
query63	227	193	188	188
query64	4325	976	665	665
query65	3224	3112	3141	3112
query66	1063	417	317	317
query67	15803	15620	15378	15378
query68	6259	772	517	517
query69	463	300	273	273
query70	1212	1141	1047	1047
query71	409	297	276	276
query72	6021	3589	3812	3589
query73	770	737	345	345
query74	8890	9108	8716	8716
query75	3167	3152	2685	2685
query76	3231	1166	755	755
query77	468	366	292	292
query78	9994	10223	9403	9403
query79	2085	821	587	587
query80	974	522	435	435
query81	532	275	243	243
query82	430	129	96	96
query83	171	173	157	157
query84	245	90	71	71
query85	767	344	309	309
query86	409	315	292	292
query87	4460	4448	4356	4356
query88	2840	2232	2234	2232
query89	380	319	287	287
query90	1782	198	194	194
query91	134	144	111	111
query92	68	59	54	54
query93	1337	997	591	591
query94	697	405	296	296
query95	347	266	261	261
query96	489	548	266	266
query97	2747	2875	2788	2788
query98	254	197	193	193
query99	1298	1426	1317	1317
Total cold run time: 270308 ms
Total hot run time: 183450 ms

doris-robot · 2025-02-13T16:13:12Z

ClickBench: Total hot run time: 30.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 090765eac0df34a3149f22722edd761268f215c8, data reload: false

query1	0.04	0.04	0.05
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.40	0.42	0.39
query6	1.17	0.67	0.65
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.58	0.51	0.54
query10	0.58	0.58	0.57
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.61
query14	2.79	2.69	2.81
query15	0.91	0.85	0.85
query16	0.38	0.39	0.36
query17	1.02	1.02	1.04
query18	0.21	0.20	0.20
query19	1.89	1.79	2.05
query20	0.02	0.01	0.02
query21	15.36	0.90	0.55
query22	0.75	1.19	0.77
query23	14.84	1.40	0.64
query24	10.80	0.92	0.63
query25	0.46	0.31	0.08
query26	0.96	0.17	0.13
query27	0.05	0.05	0.05
query28	6.13	0.78	0.43
query29	12.53	3.92	3.28
query30	0.25	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.48
query33	2.96	3.08	3.05
query34	15.76	5.17	4.48
query35	4.50	4.51	4.54
query36	0.66	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.02	0.02	0.03
query40	0.16	0.14	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.4 s
Total hot run time: 30.43 s

airborne12 · 2025-02-14T07:34:19Z

run buildall

airborne12 · 2025-02-14T07:42:54Z

run buildall

doris-robot · 2025-02-14T08:56:52Z

TPC-H: Total hot run time: 31599 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f3799bd2e0a0c8dd58de4cf338f909924f6367e, data reload: false

------ Round 1 ----------------------------------
q1	17583	5237	5073	5073
q2	2054	322	187	187
q3	10484	1256	780	780
q4	10213	1008	527	527
q5	7533	2432	2318	2318
q6	185	166	132	132
q7	906	777	608	608
q8	9290	1248	1096	1096
q9	4757	4898	4800	4800
q10	6815	2315	1879	1879
q11	470	286	261	261
q12	342	354	217	217
q13	17799	3667	3033	3033
q14	222	217	206	206
q15	521	473	468	468
q16	620	610	585	585
q17	575	866	343	343
q18	6873	6341	6142	6142
q19	1463	937	536	536
q20	317	323	187	187
q21	2806	2493	1920	1920
q22	358	343	301	301
Total cold run time: 102186 ms
Total hot run time: 31599 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5172	5154	5182	5154
q2	247	334	234	234
q3	2173	2671	2328	2328
q4	1393	1830	1381	1381
q5	4252	4121	4194	4121
q6	204	163	122	122
q7	1869	1814	1714	1714
q8	2665	2584	2608	2584
q9	7153	7175	7107	7107
q10	3012	3194	2797	2797
q11	578	521	490	490
q12	695	786	636	636
q13	3364	3939	3303	3303
q14	280	300	286	286
q15	533	460	452	452
q16	653	695	637	637
q17	1150	1593	1363	1363
q18	7855	7542	7390	7390
q19	789	813	894	813
q20	1986	2010	1870	1870
q21	5556	4942	4702	4702
q22	607	556	551	551
Total cold run time: 52186 ms
Total hot run time: 50035 ms

doris-robot · 2025-02-14T09:08:55Z

TPC-DS: Total hot run time: 189449 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f3799bd2e0a0c8dd58de4cf338f909924f6367e, data reload: false

query1	1315	962	968	962
query2	6459	1844	1836	1836
query3	10972	4470	4432	4432
query4	53987	24850	22879	22879
query5	5324	541	485	485
query6	393	202	185	185
query7	5373	510	302	302
query8	327	242	234	234
query9	7669	2487	2479	2479
query10	419	300	246	246
query11	15598	15014	14856	14856
query12	158	102	112	102
query13	1318	498	392	392
query14	10824	6869	6832	6832
query15	202	212	184	184
query16	7034	647	498	498
query17	1095	746	587	587
query18	1537	396	321	321
query19	202	218	172	172
query20	127	127	120	120
query21	207	123	109	109
query22	4646	4416	4400	4400
query23	34019	33220	33306	33220
query24	5583	2448	2461	2448
query25	496	485	429	429
query26	670	278	205	205
query27	1753	514	330	330
query28	2718	2421	2395	2395
query29	554	547	418	418
query30	216	188	152	152
query31	901	839	797	797
query32	69	58	58	58
query33	452	352	302	302
query34	767	854	514	514
query35	856	823	764	764
query36	949	997	911	911
query37	121	99	76	76
query38	4275	4287	4170	4170
query39	1516	1457	1439	1439
query40	219	112	101	101
query41	51	51	54	51
query42	119	104	105	104
query43	487	504	471	471
query44	1293	797	802	797
query45	181	171	170	170
query46	864	1073	681	681
query47	1804	1903	1795	1795
query48	396	412	330	330
query49	720	493	423	423
query50	698	741	421	421
query51	4323	4274	4201	4201
query52	104	101	94	94
query53	235	276	186	186
query54	477	504	415	415
query55	86	79	82	79
query56	279	270	277	270
query57	1185	1152	1101	1101
query58	249	239	236	236
query59	2837	2791	2685	2685
query60	288	287	269	269
query61	118	124	114	114
query62	764	723	687	687
query63	231	196	194	194
query64	1412	990	671	671
query65	3337	3207	3141	3141
query66	776	397	292	292
query67	15964	15677	15404	15404
query68	5616	762	507	507
query69	531	316	278	278
query70	1200	1133	1094	1094
query71	432	307	260	260
query72	5914	3791	3731	3731
query73	1010	746	342	342
query74	8917	8965	8692	8692
query75	3236	3137	2681	2681
query76	4029	1170	733	733
query77	530	361	272	272
query78	9930	10160	9347	9347
query79	2488	793	592	592
query80	629	549	436	436
query81	504	272	237	237
query82	394	133	94	94
query83	171	169	194	169
query84	283	101	68	68
query85	770	349	294	294
query86	408	309	273	273
query87	4553	4568	4371	4371
query88	3470	2165	2169	2165
query89	409	314	287	287
query90	1753	189	191	189
query91	134	132	107	107
query92	77	62	53	53
query93	2313	993	567	567
query94	683	416	295	295
query95	346	269	257	257
query96	474	550	272	272
query97	2805	2847	2715	2715
query98	232	212	201	201
query99	1327	1396	1246	1246
Total cold run time: 296166 ms
Total hot run time: 189449 ms

doris-robot · 2025-02-14T09:14:13Z

ClickBench: Total hot run time: 29.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f3799bd2e0a0c8dd58de4cf338f909924f6367e, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.41	0.41	0.39
query6	1.19	0.67	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.52
query10	0.58	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.61	0.59
query14	2.67	2.71	2.70
query15	0.89	0.85	0.86
query16	0.38	0.38	0.36
query17	1.06	1.01	1.03
query18	0.21	0.19	0.19
query19	1.88	1.86	2.02
query20	0.01	0.01	0.01
query21	15.35	0.88	0.54
query22	0.75	1.10	0.58
query23	15.11	1.37	0.60
query24	12.64	0.96	0.30
query25	0.34	0.15	0.09
query26	0.91	0.18	0.13
query27	0.04	0.04	0.05
query28	6.28	0.76	0.42
query29	12.60	3.85	3.24
query30	0.25	0.09	0.06
query31	2.84	0.57	0.38
query32	3.23	0.54	0.47
query33	2.97	2.99	3.00
query34	15.82	5.10	4.47
query35	4.52	4.48	4.55
query36	0.67	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 107.6 s
Total hot run time: 29.75 s

github-actions · 2025-02-19T02:03:48Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-19T02:03:51Z

PR approved by anyone and no changes requested.

eldenmoon

LGTM

…7846) This pull request focuses on improving the handling of null values in the inverted index writer and simplifying the codebase by removing redundant null map checks. The most important changes include removing unnecessary null map handling in several methods and ensuring proper null bitmap updates. Improvements to null value handling and code simplification: * [`be/src/olap/rowset/segment_v2/column_writer.cpp`](diffhunk://#diff-db6023c6e1df0c3616055f02e769cc20fcef7ee083cb3755cec1b661bb7b42ffL952-L958): Removed redundant null map handling in `Status ArrayColumnWriter::append_nullable` method. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L328-R328): Simplified `add_array_nulls` method to always return `Status::OK()`. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426): Added null map check before accessing elements in the loop to prevent potential null pointer dereference. [[1]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426) [[2]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L525-R531) * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165R513): Updated `_null_bitmap` in the `add_null_document` method to ensure proper null bitmap updates. * [`be/src/olap/task/index_builder.cpp`](diffhunk://#diff-df38b3b177cd231676ce7a405526b3419c543e29171143ddec02960a84a930c6L645-R645): Removed redundant null map handling in `Status IndexBuilder::_add_nullable` method.

…ache#47846) This pull request focuses on improving the handling of null values in the inverted index writer and simplifying the codebase by removing redundant null map checks. The most important changes include removing unnecessary null map handling in several methods and ensuring proper null bitmap updates. Improvements to null value handling and code simplification: * [`be/src/olap/rowset/segment_v2/column_writer.cpp`](diffhunk://#diff-db6023c6e1df0c3616055f02e769cc20fcef7ee083cb3755cec1b661bb7b42ffL952-L958): Removed redundant null map handling in `Status ArrayColumnWriter::append_nullable` method. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L328-R328): Simplified `add_array_nulls` method to always return `Status::OK()`. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426): Added null map check before accessing elements in the loop to prevent potential null pointer dereference. [[1]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426) [[2]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L525-R531) * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165R513): Updated `_null_bitmap` in the `add_null_document` method to ensure proper null bitmap updates. * [`be/src/olap/task/index_builder.cpp`](diffhunk://#diff-df38b3b177cd231676ce7a405526b3419c543e29171143ddec02960a84a930c6L645-R645): Removed redundant null map handling in `Status IndexBuilder::_add_nullable` method.

…rted index #47846 (#48180) cherry pick from #47846

…ache#47846) This pull request focuses on improving the handling of null values in the inverted index writer and simplifying the codebase by removing redundant null map checks. The most important changes include removing unnecessary null map handling in several methods and ensuring proper null bitmap updates. Improvements to null value handling and code simplification: * [`be/src/olap/rowset/segment_v2/column_writer.cpp`](diffhunk://#diff-db6023c6e1df0c3616055f02e769cc20fcef7ee083cb3755cec1b661bb7b42ffL952-L958): Removed redundant null map handling in `Status ArrayColumnWriter::append_nullable` method. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L328-R328): Simplified `add_array_nulls` method to always return `Status::OK()`. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426): Added null map check before accessing elements in the loop to prevent potential null pointer dereference. [[1]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426) [[2]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L525-R531) * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165R513): Updated `_null_bitmap` in the `add_null_document` method to ensure proper null bitmap updates. * [`be/src/olap/task/index_builder.cpp`](diffhunk://#diff-df38b3b177cd231676ce7a405526b3419c543e29171143ddec02960a84a930c6L645-R645): Removed redundant null map handling in `Status IndexBuilder::_add_nullable` method.

…ed index #47846 (#48052) Cherry-picked from #47846 Co-authored-by: airborne12 <jiangkai@selectdb.com>

… is null (#48231) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #47846 Problem Summary: Fix wrong adding array index for building index.

…ed index #47846 (#48214) cherry pick from #47846 #48231

… is null (apache#48231) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#47846 Problem Summary: Fix wrong adding array index for building index.

…che#47846 (apache#48180) (apache#3826) cherry pick from apache#47846

…ache#47846) This pull request focuses on improving the handling of null values in the inverted index writer and simplifying the codebase by removing redundant null map checks. The most important changes include removing unnecessary null map handling in several methods and ensuring proper null bitmap updates. Improvements to null value handling and code simplification: * [`be/src/olap/rowset/segment_v2/column_writer.cpp`](diffhunk://#diff-db6023c6e1df0c3616055f02e769cc20fcef7ee083cb3755cec1b661bb7b42ffL952-L958): Removed redundant null map handling in `Status ArrayColumnWriter::append_nullable` method. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L328-R328): Simplified `add_array_nulls` method to always return `Status::OK()`. * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426): Added null map check before accessing elements in the loop to prevent potential null pointer dereference. [[1]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L429-R426) [[2]](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165L525-R531) * [`be/src/olap/rowset/segment_v2/inverted_index_writer.cpp`](diffhunk://#diff-97781916b276f771710ab520c79ca29d5e4e331296fad7573fc9933a376dc165R513): Updated `_null_bitmap` in the `add_null_document` method to ensure proper null bitmap updates. * [`be/src/olap/task/index_builder.cpp`](diffhunk://#diff-df38b3b177cd231676ce7a405526b3419c543e29171143ddec02960a84a930c6L645-R645): Removed redundant null map handling in `Status IndexBuilder::_add_nullable` method.

… is null (apache#48231) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#47846 Problem Summary: Fix wrong adding array index for building index.

dataroaring added dev/3.0.x p0_b labels Feb 14, 2025

xiaokang added the dev/2.1.x label Feb 14, 2025

wm1581066 added the p0_r label Feb 15, 2025

airborne12 added 5 commits February 15, 2025 23:51

[fix](array index) Correct null bitmap writing for inverted index

b14ac31

[fix](array index) Correct null bitmap writing for inverted index

3058b54

update

14636b0

fix

2d5ab30

fix

2681aca

airborne12 force-pushed the fix-array-index branch from 2f3799b to 2681aca Compare February 15, 2025 15:51

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 19, 2025

eldenmoon approved these changes Feb 19, 2025

View reviewed changes

eldenmoon merged commit 376d739 into apache:master Feb 19, 2025
26 of 30 checks passed

github-actions bot mentioned this pull request Feb 19, 2025

branch-3.0: [fix](array index) Correct null bitmap writing for inverted index #47846 #48052

Merged

github-actions bot added the dev/2.1.x-conflict label Feb 19, 2025

airborne12 deleted the fix-array-index branch February 21, 2025 07:09

airborne12 mentioned this pull request Feb 21, 2025

branch-3.0.4: [fix](array index) Correct null bitmap writing for inverted index #47846 #48180

Merged

xiaokang pushed a commit that referenced this pull request Feb 21, 2025

branch-3.0.4: [fix](array index) Correct null bitmap writing for inve…

39f9074

…rted index #47846 (#48180) cherry pick from #47846

airborne12 mentioned this pull request Feb 24, 2025

branch-2.1: [fix](array index) Correct null bitmap writing for inverted index #47846 #48214

Merged

dataroaring pushed a commit that referenced this pull request Feb 24, 2025

branch-3.0: [fix](array index) Correct null bitmap writing for invert…

232f310

…ed index #47846 (#48052) Cherry-picked from #47846 Co-authored-by: airborne12 <jiangkai@selectdb.com>

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Feb 24, 2025

airborne12 mentioned this pull request Feb 24, 2025

[fix](array index) Fix index build failure when all array column data is null #48231

Merged

16 tasks

yiguolei pushed a commit that referenced this pull request Feb 25, 2025

branch-2.1: [fix](array index) Correct null bitmap writing for invert…

1aa57a3

…ed index #47846 (#48214) cherry pick from #47846 #48231

yiguolei added dev/2.1.9-merged and removed dev/2.1.x dev/2.1.x-conflict labels Feb 25, 2025

starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Apr 27, 2025

[fix](array index) Correct null bitmap writing for inverted index apa…

57537d2

…che#47846 (apache#48180) (apache#3826) cherry pick from apache#47846

[fix](array index) Correct null bitmap writing for inverted index #47846

[fix](array index) Correct null bitmap writing for inverted index #47846

Uh oh!

Conversation

airborne12 commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Feb 12, 2025

Uh oh!

airborne12 commented Feb 12, 2025

Uh oh!

doris-robot commented Feb 12, 2025

Uh oh!

doris-robot commented Feb 12, 2025

Uh oh!

doris-robot commented Feb 12, 2025

Uh oh!

doris-robot commented Feb 12, 2025

Uh oh!

airborne12 commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

airborne12 commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

doris-robot commented Feb 13, 2025

Uh oh!

airborne12 commented Feb 14, 2025

Uh oh!

airborne12 commented Feb 14, 2025

Uh oh!

doris-robot commented Feb 14, 2025

Uh oh!

doris-robot commented Feb 14, 2025

Uh oh!

doris-robot commented Feb 14, 2025

Uh oh!

github-actions bot commented Feb 19, 2025

Uh oh!

github-actions bot commented Feb 19, 2025

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

airborne12 commented Feb 12, 2025 •

edited

Loading