[enhancement](nereids)support null partition for list partition #31613

starocean999 · 2024-02-29T11:16:25Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

starocean999 · 2024-02-29T11:16:35Z

run buildall

doris-robot · 2024-02-29T11:23:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

doris-robot · 2024-02-29T12:07:42Z

TPC-H: Total hot run time: 38432 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db09a2fc13626618d5a2067b762a0681779a885d, data reload: false

------ Round 1 ----------------------------------
q1	17888	4188	4169	4169
q2	2293	148	140	140
q3	11431	980	981	980
q4	5004	977	970	970
q5	7941	3087	3100	3087
q6	184	132	128	128
q7	1264	789	781	781
q8	9717	2011	2099	2011
q9	7265	6303	6302	6302
q10	8182	2483	2503	2483
q11	425	220	211	211
q12	767	323	339	323
q13	17950	3223	3224	3223
q14	283	263	251	251
q15	534	509	500	500
q16	475	415	434	415
q17	945	906	852	852
q18	6446	6186	5937	5937
q19	1563	1527	1535	1527
q20	554	281	277	277
q21	6461	3575	3596	3575
q22	806	290	302	290
Total cold run time: 108378 ms
Total hot run time: 38432 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4041	4015	4007	4007
q2	303	215	215	215
q3	3009	2946	2955	2946
q4	1870	1836	1837	1836
q5	5120	5137	5173	5137
q6	201	117	118	117
q7	2158	1745	1707	1707
q8	3144	3224	3203	3203
q9	8294	8293	8327	8293
q10	6229	3699	3662	3662
q11	511	418	415	415
q12	690	523	521	521
q13	3751	3099	3076	3076
q14	274	252	270	252
q15	542	498	492	492
q16	536	473	485	473
q17	1714	1697	1691	1691
q18	7957	7495	7376	7376
q19	1666	1659	1657	1657
q20	2096	1902	1926	1902
q21	4874	4749	4631	4631
q22	504	435	449	435
Total cold run time: 59484 ms
Total hot run time: 54044 ms

doris-robot · 2024-02-29T12:18:25Z

TPC-DS: Total hot run time: 169650 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db09a2fc13626618d5a2067b762a0681779a885d, data reload: false

query1	947	353	351	351
query2	6531	1936	1756	1756
query3	6711	224	212	212
query4	23286	20600	20562	20562
query5	4277	368	353	353
query6	257	178	178	178
query7	4619	294	296	294
query8	259	192	196	192
query9	8291	2281	2268	2268
query10	406	219	212	212
query11	14468	14262	14232	14232
query12	129	82	80	80
query13	1655	423	422	422
query14	8999	6333	6371	6333
query15	203	179	176	176
query16	7097	261	257	257
query17	1033	527	502	502
query18	1908	267	266	266
query19	194	146	154	146
query20	88	81	84	81
query21	202	129	130	129
query22	4862	4628	4553	4553
query23	30543	30049	30388	30049
query24	11855	3127	3074	3074
query25	657	350	357	350
query26	1881	160	168	160
query27	3021	340	311	311
query28	7480	1838	1812	1812
query29	1198	553	559	553
query30	293	134	144	134
query31	864	687	700	687
query32	94	60	58	58
query33	726	236	219	219
query34	1027	488	484	484
query35	855	771	749	749
query36	946	885	829	829
query37	170	57	58	57
query38	3110	2976	2987	2976
query39	1347	1256	1256	1256
query40	284	103	103	103
query41	40	36	35	35
query42	102	96	102	96
query43	475	422	438	422
query44	1068	702	722	702
query45	194	186	176	176
query46	1032	815	796	796
query47	1478	1451	1362	1362
query48	425	348	341	341
query49	1157	302	302	302
query50	772	371	374	371
query51	4373	4317	4386	4317
query52	102	86	93	86
query53	331	270	271	270
query54	290	215	236	215
query55	82	82	90	82
query56	214	201	195	195
query57	966	884	863	863
query58	214	198	202	198
query59	2310	2088	2200	2088
query60	230	229	214	214
query61	91	85	83	83
query62	586	348	354	348
query63	287	256	261	256
query64	6543	2973	3111	2973
query65	3239	3234	3236	3234
query66	1439	325	319	319
query67	14239	14267	14169	14169
query68	4991	577	565	565
query69	536	378	390	378
query70	1220	1184	1202	1184
query71	338	254	249	249
query72	6156	2817	2651	2651
query73	715	321	319	319
query74	6361	5955	5958	5955
query75	3019	2425	2413	2413
query76	2807	1021	1143	1021
query77	346	236	234	234
query78	9056	8707	8399	8399
query79	1086	524	523	523
query80	881	356	354	354
query81	467	201	198	198
query82	1301	87	90	87
query83	220	128	122	122
query84	280	78	78	78
query85	1249	343	350	343
query86	335	303	281	281
query87	3235	3152	3052	3052
query88	2828	2343	2348	2343
query89	404	337	323	323
query90	1956	171	170	170
query91	154	123	127	123
query92	58	54	51	51
query93	1060	541	515	515
query94	1059	194	180	180
query95	436	347	361	347
query96	573	273	269	269
query97	4248	4148	4190	4148
query98	222	209	190	190
query99	1038	663	656	656
Total cold run time: 263687 ms
Total hot run time: 169650 ms

doris-robot · 2024-02-29T12:23:53Z

ClickBench: Total hot run time: 32.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db09a2fc13626618d5a2067b762a0681779a885d, data reload: false

query1	0.04	0.04	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.51	0.52
query6	1.37	0.69	0.68
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.55	0.50	0.51
query10	0.57	0.58	0.56
query11	0.13	0.10	0.09
query12	0.12	0.10	0.10
query13	0.63	0.61	0.62
query14	0.79	0.80	0.79
query15	0.84	0.84	0.82
query16	0.36	0.39	0.38
query17	1.02	1.01	0.97
query18	0.26	0.24	0.26
query19	1.79	1.68	1.73
query20	0.02	0.01	0.01
query21	15.42	0.62	0.63
query22	3.42	3.47	2.38
query23	17.65	1.02	1.00
query24	2.12	0.58	0.38
query25	0.24	0.13	0.04
query26	0.17	0.15	0.14
query27	0.03	0.03	0.04
query28	12.13	0.85	0.84
query29	12.56	3.17	3.24
query30	0.67	0.54	0.51
query31	2.79	0.35	0.36
query32	3.33	0.47	0.48
query33	3.20	3.18	3.25
query34	15.38	4.72	4.72
query35	4.77	4.74	4.72
query36	1.13	1.07	1.08
query37	0.06	0.05	0.06
query38	0.05	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 106.42 s
Total hot run time: 32.04 s

doris-robot · 2024-02-29T12:27:02Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit db09a2fc13626618d5a2067b762a0681779a885d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       18.3 seconds inserted 10000000 Rows, about 546K ops/s

starocean999 · 2024-02-29T13:55:34Z

run buildall

doris-robot · 2024-02-29T14:27:00Z

TPC-H: Total hot run time: 37742 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1cc896277e8bb88d120f1f71766e9a304796f35, data reload: false

------ Round 1 ----------------------------------
q1	17689	4116	4059	4059
q2	2045	143	134	134
q3	10700	966	936	936
q4	4666	910	921	910
q5	7639	2918	2990	2918
q6	180	125	123	123
q7	1238	756	744	744
q8	9263	1965	2002	1965
q9	7172	6260	6235	6235
q10	8185	2472	2511	2472
q11	411	222	222	222
q12	775	313	317	313
q13	17962	3230	3212	3212
q14	277	259	247	247
q15	532	497	490	490
q16	477	399	414	399
q17	930	886	871	871
q18	6698	5915	5977	5915
q19	1551	1499	1505	1499
q20	545	280	277	277
q21	6325	3521	3545	3521
q22	810	296	280	280
Total cold run time: 106070 ms
Total hot run time: 37742 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4040	4028	3999	3999
q2	312	214	213	213
q3	2947	2904	2934	2904
q4	1841	1816	1820	1816
q5	5093	5141	5122	5122
q6	199	115	117	115
q7	2113	1742	1675	1675
q8	3113	3173	3208	3173
q9	8223	8266	8194	8194
q10	6091	3657	3659	3657
q11	508	420	455	420
q12	671	525	512	512
q13	12247	3071	3068	3068
q14	291	252	253	252
q15	542	498	495	495
q16	538	469	467	467
q17	1688	1666	1662	1662
q18	7680	7463	7356	7356
q19	1653	1630	1618	1618
q20	2093	1893	1898	1893
q21	4765	4570	4669	4570
q22	520	453	461	453
Total cold run time: 67168 ms
Total hot run time: 53634 ms

doris-robot · 2024-02-29T14:37:55Z

TPC-DS: Total hot run time: 169641 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1cc896277e8bb88d120f1f71766e9a304796f35, data reload: false

query1	920	341	339	339
query2	6553	1743	1756	1743
query3	6700	209	209	209
query4	23325	20623	20570	20570
query5	4198	365	359	359
query6	264	171	176	171
query7	4611	294	284	284
query8	266	190	195	190
query9	8565	2305	2277	2277
query10	422	216	206	206
query11	14631	14052	14272	14052
query12	135	90	86	86
query13	1652	433	426	426
query14	8822	6687	6343	6343
query15	201	177	175	175
query16	7081	261	251	251
query17	1041	518	506	506
query18	1893	262	260	260
query19	200	153	153	153
query20	87	86	86	86
query21	199	135	130	130
query22	4857	4722	4587	4587
query23	30829	30327	30682	30327
query24	12005	3168	3206	3168
query25	1127	357	336	336
query26	2148	151	162	151
query27	3385	310	309	309
query28	8057	1882	1872	1872
query29	2568	573	580	573
query30	1172	135	146	135
query31	2142	722	702	702
query32	449	64	60	60
query33	1784	232	240	232
query34	2067	508	500	500
query35	1742	748	754	748
query36	1024	861	889	861
query37	256	62	61	61
query38	3127	3030	2988	2988
query39	1315	1261	1245	1245
query40	275	102	102	102
query41	38	38	36	36
query42	104	95	98	95
query43	447	430	428	428
query44	1079	694	709	694
query45	194	180	172	172
query46	1056	798	781	781
query47	1534	1476	1398	1398
query48	422	351	352	351
query49	1171	314	301	301
query50	764	371	365	365
query51	4359	4265	4265	4265
query52	97	98	85	85
query53	329	269	271	269
query54	288	226	231	226
query55	89	78	78	78
query56	224	203	198	198
query57	957	878	876	876
query58	214	191	192	191
query59	2188	2069	2168	2069
query60	234	229	212	212
query61	85	85	85	85
query62	608	349	329	329
query63	291	255	256	255
query64	6544	3075	3132	3075
query65	3255	3206	3218	3206
query66	1445	320	322	320
query67	14324	14136	14209	14136
query68	4918	568	566	566
query69	549	371	374	371
query70	1180	1218	1117	1117
query71	410	249	252	249
query72	6187	2841	2645	2645
query73	706	320	319	319
query74	6363	6092	5899	5899
query75	3078	2440	2381	2381
query76	3107	1041	1164	1041
query77	355	242	242	242
query78	8977	8510	8365	8365
query79	2174	518	522	518
query80	989	365	353	353
query81	497	193	201	193
query82	939	87	90	87
query83	229	126	126	126
query84	280	81	74	74
query85	1426	358	348	348
query86	454	302	302	302
query87	3265	3065	3078	3065
query88	3327	2279	2260	2260
query89	381	324	321	321
query90	1965	168	165	165
query91	151	125	127	125
query92	57	47	53	47
query93	1487	532	508	508
query94	1208	186	176	176
query95	441	336	341	336
query96	570	263	261	261
query97	4243	4086	4124	4086
query98	225	212	197	197
query99	1062	644	629	629
Total cold run time: 275635 ms
Total hot run time: 169641 ms

doris-robot · 2024-02-29T14:43:22Z

ClickBench: Total hot run time: 32.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1cc896277e8bb88d120f1f71766e9a304796f35, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.11	0.10
query5	0.52	0.52	0.51
query6	1.38	0.68	0.69
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.56	0.51	0.52
query10	0.57	0.56	0.57
query11	0.13	0.10	0.10
query12	0.12	0.10	0.10
query13	0.62	0.62	0.61
query14	0.79	0.80	0.81
query15	0.83	0.82	0.81
query16	0.37	0.38	0.38
query17	0.95	0.98	0.97
query18	0.25	0.26	0.26
query19	1.78	1.67	1.77
query20	0.01	0.01	0.01
query21	15.40	0.66	0.64
query22	2.68	3.87	2.82
query23	17.47	1.02	1.01
query24	2.21	0.50	0.18
query25	0.28	0.05	0.03
query26	0.17	0.14	0.14
query27	0.04	0.02	0.03
query28	12.36	0.84	0.84
query29	12.55	3.23	3.28
query30	0.57	0.54	0.53
query31	2.80	0.35	0.36
query32	3.34	0.48	0.48
query33	3.18	3.19	3.18
query34	15.36	4.77	4.75
query35	4.77	4.75	4.75
query36	1.13	1.06	1.08
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.17	0.12	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.03	0.02
Total cold run time: 105.67 s
Total hot run time: 32.48 s

doris-robot · 2024-02-29T14:46:31Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a1cc896277e8bb88d120f1f71766e9a304796f35 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       18.1 seconds inserted 10000000 Rows, about 552K ops/s

starocean999 · 2024-03-03T10:43:52Z

run buildall

github-actions · 2024-03-03T10:53:12Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-03T11:15:46Z

TeamCity be ut coverage result:
Function Coverage: 35.43% (8554/24146)
Line Coverage: 27.19% (69410/255231)
Region Coverage: 26.42% (36016/136338)
Branch Coverage: 23.25% (18393/79102)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d4f59c8ff3c4dcde698d9ac6f132de7ab503751d_d4f59c8ff3c4dcde698d9ac6f132de7ab503751d/report/index.html

starocean999 · 2024-03-04T01:25:23Z

run buildall

github-actions · 2024-03-04T01:35:15Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-04T01:47:17Z

TeamCity be ut coverage result:
Function Coverage: 35.42% (8553/24146)
Line Coverage: 27.19% (69409/255231)
Region Coverage: 26.42% (36014/136338)
Branch Coverage: 23.25% (18395/79102)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a602c9cd22ddab631648a39d6ba2c5bd70b585c0_a602c9cd22ddab631648a39d6ba2c5bd70b585c0/report/index.html

github-actions · 2024-03-04T03:26:49Z

PR approved by anyone and no changes requested.

morningman

LGTM

…che#31736) This PR apache#23026 support the partition prune for hive table with `_HIVE_DEFAULT_PARTITION`, but it will always select partition with `_HIVE_DEFAULT_PARTITION`. This PR apache#31613 support null partition for olap table's list partition, so we can treat `_HIVE_DEFAULT_PARTITION` as null partition of hive table. So this PR change the partition prune logic

) This PR #23026 support the partition prune for hive table with `_HIVE_DEFAULT_PARTITION`, but it will always select partition with `_HIVE_DEFAULT_PARTITION`. This PR #31613 support null partition for olap table's list partition, so we can treat `_HIVE_DEFAULT_PARTITION` as null partition of hive table. So this PR change the partition prune logic

## Proposed changes fix #31613

starocean999 added 3 commits March 3, 2024 18:05

[enhancement](nereids)support null partition for list partition

b915f6c

fix failed case

dfc00cc

add test case

d4f59c8

starocean999 force-pushed the nereids_1101 branch from a1cc896 to d4f59c8 Compare March 3, 2024 10:43

add case

a602c9c

zclllyybb approved these changes Mar 4, 2024

View reviewed changes

github-actions bot added the reviewed label Mar 4, 2024

starocean999 marked this pull request as ready for review March 4, 2024 03:31

morningman approved these changes Mar 4, 2024

View reviewed changes

starocean999 merged commit be41811 into apache:master Mar 4, 2024

morningman mentioned this pull request Mar 4, 2024

[fix](hive) support partition prune for _HIVE_DEFAULT_PARTITION_ #31736

Merged

morningman added the dev/2.0.x label Mar 4, 2024

yiguolei pushed a commit that referenced this pull request Mar 6, 2024

[enhancement](nereids)support null partition for list partition (#31613)

3777ffb

starocean999 removed the dev/2.0.x label Mar 13, 2024

morrySnow added the not-merge/2.0 do not merge into 2.0 branch label Mar 13, 2024

dataroaring mentioned this pull request Aug 14, 2024

[fix](upgrade) fix partition key problem introduced by be418110c #39373

Merged

dataroaring added a commit that referenced this pull request Aug 15, 2024

[fix](upgrade) fix partition key problem introduced by be41811 (#39373)

5c68c92

## Proposed changes fix #31613

[enhancement](nereids)support null partition for list partition #31613

[enhancement](nereids)support null partition for list partition #31613

Uh oh!

Conversation

starocean999 commented Feb 29, 2024

Proposed changes

Further comments

Uh oh!

starocean999 commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

starocean999 commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

doris-robot commented Feb 29, 2024

Uh oh!

starocean999 commented Mar 3, 2024

Uh oh!

github-actions bot commented Mar 3, 2024

Uh oh!

doris-robot commented Mar 3, 2024

Uh oh!

starocean999 commented Mar 4, 2024

Uh oh!

github-actions bot commented Mar 4, 2024

Uh oh!

doris-robot commented Mar 4, 2024

Uh oh!

github-actions bot commented Mar 4, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants