Skip to content

Conversation

@BiteTheDDDDt
Copy link
Contributor

Proposed changes

support predicate apprear both on key and value mv column

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39206 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ebaa1f91b949c8ea13212f09b5bfe9b50c1d447, data reload: false

------ Round 1 ----------------------------------
q1	17605	5379	5361	5361
q2	2060	141	135	135
q3	10654	1198	1185	1185
q4	10685	793	763	763
q5	7972	3195	3209	3195
q6	199	123	125	123
q7	893	511	504	504
q8	9464	1951	2011	1951
q9	7766	6547	6360	6360
q10	8190	3087	3035	3035
q11	418	207	207	207
q12	355	189	191	189
q13	17979	3365	3341	3341
q14	241	217	210	210
q15	541	518	512	512
q16	433	369	379	369
q17	940	496	508	496
q18	7550	7037	6844	6844
q19	1565	1458	1408	1408
q20	582	317	290	290
q21	2768	2422	2455	2422
q22	373	318	306	306
Total cold run time: 109233 ms
Total hot run time: 39206 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5367	5264	5100	5100
q2	323	211	210	210
q3	3622	3230	3236	3230
q4	2043	2088	2019	2019
q5	6051	5954	6038	5954
q6	199	113	116	113
q7	2338	1847	1830	1830
q8	3229	3371	3384	3371
q9	9062	8887	8804	8804
q10	3945	3855	3880	3855
q11	555	456	468	456
q12	772	627	621	621
q13	16919	3156	3115	3115
q14	294	256	266	256
q15	556	508	500	500
q16	518	475	464	464
q17	1892	1853	1808	1808
q18	9464	18293	9672	9672
q19	25575	1570	1515	1515
q20	4601	1955	1926	1926
q21	14657	5305	5308	5305
q22	976	534	539	534
Total cold run time: 112958 ms
Total hot run time: 60658 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176404 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ebaa1f91b949c8ea13212f09b5bfe9b50c1d447, data reload: false

query1	938	339	320	320
query2	6562	2203	2016	2016
query3	6727	201	198	198
query4	32904	22183	22430	22183
query5	4457	391	377	377
query6	246	154	155	154
query7	4593	267	272	267
query8	243	169	178	169
query9	8334	2529	2532	2529
query10	422	230	209	209
query11	17297	15611	15676	15611
query12	126	75	67	67
query13	1687	391	383	383
query14	10578	6871	6844	6844
query15	215	178	186	178
query16	5789	253	246	246
query17	951	466	468	466
query18	1794	248	253	248
query19	174	132	127	127
query20	68	71	67	67
query21	190	136	127	127
query22	4949	4794	4692	4692
query23	31561	31007	30717	30717
query24	11562	2770	2731	2731
query25	573	313	315	313
query26	1581	141	142	141
query27	3222	285	286	285
query28	7410	1838	1821	1821
query29	1513	627	628	627
query30	278	137	140	137
query31	916	743	750	743
query32	75	50	50	50
query33	702	208	211	208
query34	1161	448	455	448
query35	892	760	763	760
query36	1327	1186	1153	1153
query37	95	57	60	57
query38	3351	3223	3255	3223
query39	1328	1270	1261	1261
query40	204	83	88	83
query41	38	36	35	35
query42	95	84	85	84
query43	501	518	472	472
query44	1068	681	689	681
query45	189	178	180	178
query46	1060	643	625	625
query47	1572	1520	1529	1520
query48	381	309	309	309
query49	1124	280	281	280
query50	672	306	315	306
query51	5294	5145	5177	5145
query52	88	73	74	73
query53	328	254	266	254
query54	247	182	179	179
query55	82	78	73	73
query56	185	163	164	163
query57	975	942	936	936
query58	197	151	157	151
query59	3013	2621	2818	2621
query60	218	184	182	182
query61	85	93	83	83
query62	599	379	354	354
query63	288	253	267	253
query64	4997	1753	1732	1732
query65	3321	3256	3236	3236
query66	1293	323	320	320
query67	15682	15492	15198	15198
query68	12311	537	518	518
query69	616	297	300	297
query70	1669	1505	1450	1450
query71	491	207	208	207
query72	5173	2835	2843	2835
query73	2752	316	312	312
query74	7016	6389	6336	6336
query75	4898	2354	2313	2313
query76	6283	1001	986	986
query77	656	227	232	227
query78	9065	9003	8676	8676
query79	987	494	507	494
query80	562	324	310	310
query81	463	200	199	199
query82	204	77	78	77
query83	137	119	119	119
query84	278	70	68	68
query85	1093	325	330	325
query86	372	392	341	341
query87	3535	3346	3374	3346
query88	2856	2201	2199	2199
query89	431	360	358	358
query90	1958	185	183	183
query91	153	128	132	128
query92	51	42	42	42
query93	1442	442	452	442
query94	1198	158	157	157
query95	492	465	453	453
query96	613	314	318	314
query97	4293	4143	4136	4136
query98	213	191	193	191
query99	1026	723	686	686
Total cold run time: 293874 ms
Total hot run time: 176404 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ebaa1f91b949c8ea13212f09b5bfe9b50c1d447, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.05	0.04
query4	1.70	0.06	0.06
query5	0.55	0.51	0.53
query6	1.29	0.63	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.50	0.51
query10	0.56	0.55	0.55
query11	0.11	0.09	0.08
query12	0.12	0.09	0.09
query13	0.61	0.61	0.60
query14	0.80	0.78	0.81
query15	0.80	0.78	0.79
query16	0.37	0.37	0.37
query17	1.05	0.99	1.00
query18	0.23	0.27	0.24
query19	1.86	1.80	1.80
query20	0.01	0.01	0.01
query21	15.61	0.57	0.58
query22	2.19	2.40	2.71
query23	17.54	0.75	0.74
query24	2.53	0.79	1.11
query25	0.39	0.24	0.22
query26	0.40	0.13	0.14
query27	0.06	0.05	0.05
query28	11.77	0.74	0.77
query29	12.52	3.16	3.20
query30	0.54	0.54	0.48
query31	2.78	0.34	0.34
query32	3.39	0.47	0.49
query33	3.25	3.23	3.28
query34	15.75	4.29	4.32
query35	4.45	4.27	4.30
query36	1.11	1.06	1.06
query37	0.06	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.02	0.01
query40	0.15	0.12	0.12
query41	0.08	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.65 s
Total hot run time: 31.29 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7ebaa1f91b949c8ea13212f09b5bfe9b50c1d447 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39156 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ec9bd7e4ef7cdc1f52ae439a145e8a1b3ee1614, data reload: false

------ Round 1 ----------------------------------
q1	18950	5403	5265	5265
q2	2047	136	129	129
q3	10789	1181	1142	1142
q4	10735	753	883	753
q5	8711	3177	3163	3163
q6	200	118	121	118
q7	870	519	521	519
q8	9367	1931	1964	1931
q9	9234	6579	6582	6579
q10	8332	3012	3041	3012
q11	428	211	219	211
q12	360	186	184	184
q13	19260	3404	3431	3404
q14	256	215	217	215
q15	545	512	494	494
q16	472	412	387	387
q17	937	533	506	506
q18	7292	6835	6778	6778
q19	1556	1359	1402	1359
q20	565	328	309	309
q21	2746	2393	2427	2393
q22	375	305	316	305
Total cold run time: 114027 ms
Total hot run time: 39156 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5570	5348	5231	5231
q2	323	215	207	207
q3	3302	3217	3192	3192
q4	2080	2031	1978	1978
q5	6008	5881	5763	5763
q6	195	113	115	113
q7	2333	1800	1825	1800
q8	3216	3378	3361	3361
q9	8857	8801	8855	8801
q10	3844	3842	3777	3777
q11	558	441	451	441
q12	784	627	594	594
q13	16956	3170	3125	3125
q14	287	262	265	262
q15	551	503	506	503
q16	512	469	471	469
q17	1848	1834	1829	1829
q18	9499	16278	9582	9582
q19	23556	1691	1525	1525
q20	4606	1961	1966	1961
q21	14464	5390	5422	5390
q22	959	548	576	548
Total cold run time: 110308 ms
Total hot run time: 60452 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176858 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ec9bd7e4ef7cdc1f52ae439a145e8a1b3ee1614, data reload: false

query1	930	338	338	338
query2	6564	1933	1866	1866
query3	6686	198	196	196
query4	34428	22384	22336	22336
query5	4453	375	362	362
query6	246	157	155	155
query7	4609	260	267	260
query8	223	172	177	172
query9	8445	2544	2510	2510
query10	418	214	232	214
query11	16718	15720	15584	15584
query12	121	70	69	69
query13	1671	386	394	386
query14	10467	6958	6979	6958
query15	214	186	180	180
query16	5787	257	253	253
query17	942	492	465	465
query18	1799	263	256	256
query19	169	137	130	130
query20	70	76	75	75
query21	189	127	133	127
query22	4845	4745	4785	4745
query23	31661	30958	30886	30886
query24	11597	2752	2870	2752
query25	574	313	309	309
query26	1634	140	143	140
query27	3211	283	283	283
query28	7290	1843	1817	1817
query29	1604	653	621	621
query30	283	140	137	137
query31	937	752	781	752
query32	76	52	50	50
query33	709	212	213	212
query34	1146	451	463	451
query35	885	773	780	773
query36	1357	1208	1180	1180
query37	95	61	64	61
query38	3336	3241	3276	3241
query39	1318	1293	1262	1262
query40	199	96	81	81
query41	41	37	35	35
query42	92	81	86	81
query43	505	478	453	453
query44	1090	677	685	677
query45	196	192	177	177
query46	1076	637	647	637
query47	1605	1607	1512	1512
query48	383	313	301	301
query49	1128	293	285	285
query50	700	314	312	312
query51	5273	5173	5196	5173
query52	95	75	80	75
query53	311	265	253	253
query54	241	177	187	177
query55	84	77	80	77
query56	187	173	174	173
query57	1011	979	930	930
query58	199	164	158	158
query59	2893	2661	2488	2488
query60	208	187	187	187
query61	95	85	84	84
query62	599	370	359	359
query63	269	262	260	260
query64	5153	1782	1777	1777
query65	3336	3258	3245	3245
query66	1291	317	321	317
query67	15819	15067	15087	15067
query68	12756	532	522	522
query69	590	308	299	299
query70	1745	1529	1497	1497
query71	473	221	217	217
query72	4903	2893	2866	2866
query73	3098	316	319	316
query74	7300	6334	6372	6334
query75	4906	2315	2341	2315
query76	6361	1048	1049	1048
query77	687	231	234	231
query78	9279	8814	8665	8665
query79	1030	490	502	490
query80	564	324	323	323
query81	469	209	201	201
query82	201	84	78	78
query83	139	116	131	116
query84	274	71	68	68
query85	1066	349	330	330
query86	383	381	409	381
query87	3501	3392	3353	3353
query88	2827	2196	2176	2176
query89	447	363	367	363
query90	1951	191	184	184
query91	150	133	133	133
query92	56	46	42	42
query93	1297	452	447	447
query94	1209	164	162	162
query95	480	457	448	448
query96	607	322	315	315
query97	4281	4202	4156	4156
query98	208	194	183	183
query99	982	724	684	684
Total cold run time: 296006 ms
Total hot run time: 176858 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ec9bd7e4ef7cdc1f52ae439a145e8a1b3ee1614, data reload: false

query1	0.03	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.05	0.06
query4	1.70	0.06	0.06
query5	0.54	0.53	0.53
query6	1.27	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.50	0.50
query10	0.55	0.56	0.57
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.60	0.62	0.60
query14	0.79	0.79	0.81
query15	0.79	0.77	0.77
query16	0.37	0.39	0.38
query17	1.01	1.02	0.99
query18	0.25	0.25	0.24
query19	1.86	1.74	1.77
query20	0.01	0.01	0.01
query21	15.39	0.59	0.60
query22	2.44	2.17	2.09
query23	17.44	0.68	0.82
query24	2.47	0.74	1.18
query25	0.46	0.23	0.09
query26	0.60	0.13	0.13
query27	0.06	0.06	0.04
query28	11.49	0.77	0.75
query29	12.50	3.15	3.12
query30	0.58	0.49	0.50
query31	2.78	0.35	0.34
query32	3.38	0.50	0.47
query33	3.25	3.25	3.25
query34	15.82	4.29	4.34
query35	4.38	4.29	4.28
query36	1.12	1.05	1.09
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.01	0.02
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.47 s
Total hot run time: 30.76 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7ec9bd7e4ef7cdc1f52ae439a145e8a1b3ee1614 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       15.0 seconds inserted 10000000 Rows, about 666K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 23, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 23dfb28 into apache:master Jan 23, 2024
yiguolei pushed a commit that referenced this pull request Jan 23, 2024
…d value mv column (#30215)

support predicate apprear both on key and value mv column
BiteTheDDDDt added a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Apr 8, 2024
…d value mv column (apache#30215)

support predicate apprear both on key and value mv column
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.9-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants