Skip to content

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Apr 15, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Apr 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38477 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba2929ad65e112eb02a86e4322083cb658a0b95f, data reload: false

------ Round 1 ----------------------------------
q1	17590	4339	4197	4197
q2	2013	194	191	191
q3	10443	1194	1235	1194
q4	10197	785	847	785
q5	7534	2734	2706	2706
q6	218	134	136	134
q7	1017	620	577	577
q8	9212	2099	2050	2050
q9	7940	6594	6525	6525
q10	8615	3553	3537	3537
q11	487	243	232	232
q12	485	224	215	215
q13	18905	2906	2973	2906
q14	287	236	235	235
q15	515	492	482	482
q16	512	382	404	382
q17	970	678	647	647
q18	7457	6774	6689	6689
q19	7674	1470	1507	1470
q20	674	311	318	311
q21	3393	2895	2716	2716
q22	361	296	305	296
Total cold run time: 116499 ms
Total hot run time: 38477 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4306	4267	4197	4197
q2	372	265	263	263
q3	2954	2825	2772	2772
q4	1867	1541	1572	1541
q5	5345	5321	5343	5321
q6	208	125	126	125
q7	2249	1893	1857	1857
q8	3200	3327	3347	3327
q9	8567	8587	8785	8587
q10	4077	4035	3975	3975
q11	635	487	517	487
q12	825	619	661	619
q13	17545	3189	3137	3137
q14	321	300	292	292
q15	512	472	479	472
q16	523	442	448	442
q17	1819	1552	1557	1552
q18	8009	7969	7525	7525
q19	1687	1592	1615	1592
q20	2078	1890	1805	1805
q21	5607	4983	4946	4946
q22	556	468	483	468
Total cold run time: 73262 ms
Total hot run time: 55302 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185100 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba2929ad65e112eb02a86e4322083cb658a0b95f, data reload: false

query1	1221	1110	1111	1110
query2	6477	2644	2381	2381
query3	6660	203	206	203
query4	37049	21506	21435	21435
query5	4130	390	398	390
query6	240	187	178	178
query7	4043	285	285	285
query8	219	175	173	173
query9	5775	2303	2264	2264
query10	367	245	244	244
query11	14668	14241	14252	14241
query12	135	93	87	87
query13	992	359	356	356
query14	9189	6811	6834	6811
query15	209	176	175	175
query16	7066	257	252	252
query17	1802	555	545	545
query18	1709	288	264	264
query19	199	144	144	144
query20	94	89	85	85
query21	202	123	123	123
query22	4999	4813	4859	4813
query23	33719	33157	33466	33157
query24	11491	3083	2970	2970
query25	532	386	385	385
query26	745	160	159	159
query27	2948	365	359	359
query28	6733	2111	2100	2100
query29	853	648	599	599
query30	280	169	182	169
query31	944	742	742	742
query32	60	53	55	53
query33	577	272	245	245
query34	963	488	499	488
query35	837	708	736	708
query36	1055	932	946	932
query37	105	72	68	68
query38	3469	3292	3314	3292
query39	1618	1579	1564	1564
query40	169	121	128	121
query41	47	48	47	47
query42	103	100	94	94
query43	582	544	551	544
query44	1397	738	735	735
query45	285	290	278	278
query46	1064	752	772	752
query47	2021	1977	1969	1969
query48	362	298	298	298
query49	833	375	362	362
query50	766	388	392	388
query51	6783	6699	6742	6699
query52	94	88	86	86
query53	338	280	270	270
query54	245	232	233	232
query55	76	75	70	70
query56	238	217	218	217
query57	1202	1124	1136	1124
query58	219	218	196	196
query59	3262	3369	3018	3018
query60	243	229	235	229
query61	90	86	90	86
query62	636	443	453	443
query63	304	281	274	274
query64	4115	3855	3831	3831
query65	3090	3039	3074	3039
query66	741	330	322	322
query67	15619	14909	15239	14909
query68	5104	528	531	528
query69	471	298	292	292
query70	1236	1118	1192	1118
query71	424	267	266	266
query72	6326	2593	2466	2466
query73	733	314	310	310
query74	6769	6414	6355	6355
query75	3322	2696	2659	2659
query76	3228	1100	1129	1100
query77	603	236	236	236
query78	10886	10302	10342	10302
query79	3541	526	535	526
query80	2216	424	437	424
query81	513	230	224	224
query82	1523	98	97	97
query83	357	172	164	164
query84	269	88	79	79
query85	1446	273	264	264
query86	454	308	305	305
query87	3524	3251	3270	3251
query88	5437	2261	2283	2261
query89	459	369	371	369
query90	1823	177	180	177
query91	184	96	98	96
query92	54	45	46	45
query93	4047	498	493	493
query94	1133	172	175	172
query95	1096	1090	1105	1090
query96	574	267	259	259
query97	3134	2922	2912	2912
query98	232	213	219	213
query99	1269	865	866	865
Total cold run time: 291685 ms
Total hot run time: 185100 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba2929ad65e112eb02a86e4322083cb658a0b95f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.06
query5	0.50	0.50	0.52
query6	1.49	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.53	0.48	0.50
query10	0.54	0.55	0.56
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.74	0.79	0.78
query15	0.81	0.80	0.80
query16	0.37	0.34	0.39
query17	0.95	1.01	1.01
query18	0.20	0.24	0.26
query19	1.82	1.66	1.75
query20	0.02	0.01	0.01
query21	15.39	0.66	0.65
query22	4.52	7.30	2.17
query23	18.33	1.35	1.24
query24	1.56	0.27	0.20
query25	0.15	0.07	0.08
query26	0.26	0.16	0.16
query27	0.08	0.07	0.08
query28	13.54	1.00	0.97
query29	12.58	3.26	3.24
query30	0.27	0.08	0.06
query31	2.85	0.37	0.36
query32	3.30	0.49	0.46
query33	2.81	2.83	2.79
query34	17.11	4.44	4.47
query35	4.50	4.50	4.46
query36	0.64	0.46	0.45
query37	0.18	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.17	0.16	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.55 s
Total hot run time: 30.51 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ba2929ad65e112eb02a86e4322083cb658a0b95f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@w41ter w41ter force-pushed the add_index_to_signature branch from ba2929a to 2ca2f68 Compare April 15, 2024 10:16
@w41ter
Copy link
Contributor Author

w41ter commented Apr 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38301 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ca2f68a98f65bc132be68462fdff164e5f2cd26, data reload: false

------ Round 1 ----------------------------------
q1	17611	4283	4221	4221
q2	2012	185	196	185
q3	10467	1124	1234	1124
q4	10189	755	731	731
q5	7517	2701	2663	2663
q6	218	131	130	130
q7	984	613	579	579
q8	9204	2042	2021	2021
q9	7988	6589	6545	6545
q10	8567	3512	3505	3505
q11	458	238	227	227
q12	494	218	214	214
q13	17788	2892	2960	2892
q14	265	232	240	232
q15	523	482	488	482
q16	517	394	381	381
q17	945	666	690	666
q18	7323	6747	6701	6701
q19	6082	1515	1492	1492
q20	682	304	297	297
q21	3431	2707	2809	2707
q22	369	306	308	306
Total cold run time: 113634 ms
Total hot run time: 38301 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4278	4167	4167
q2	363	261	253	253
q3	2926	2752	2794	2752
q4	1866	1591	1621	1591
q5	5313	5361	5283	5283
q6	209	120	122	120
q7	2241	1882	1855	1855
q8	3176	3352	3342	3342
q9	8551	8526	8543	8526
q10	4081	3887	4029	3887
q11	628	491	490	490
q12	787	651	645	645
q13	17544	3189	3132	3132
q14	309	293	298	293
q15	513	478	470	470
q16	521	444	462	444
q17	1816	1522	1517	1517
q18	8145	7848	7722	7722
q19	1661	1534	1566	1534
q20	2023	1869	1833	1833
q21	11013	4842	4928	4842
q22	549	491	480	480
Total cold run time: 78555 ms
Total hot run time: 55178 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182688 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ca2f68a98f65bc132be68462fdff164e5f2cd26, data reload: false

query1	1236	1132	1107	1107
query2	6971	2830	2369	2369
query3	6655	203	201	201
query4	36982	21452	21271	21271
query5	4128	379	382	379
query6	229	175	170	170
query7	4041	295	288	288
query8	219	162	175	162
query9	5757	2290	2293	2290
query10	377	233	242	233
query11	14629	14180	14349	14180
query12	131	89	89	89
query13	991	363	357	357
query14	9945	6831	6702	6702
query15	208	181	177	177
query16	7001	268	252	252
query17	1778	580	561	561
query18	1658	290	275	275
query19	201	153	150	150
query20	95	86	89	86
query21	200	128	127	127
query22	5004	4791	4792	4791
query23	33747	32758	33191	32758
query24	6877	3036	2976	2976
query25	472	377	394	377
query26	718	154	158	154
query27	2720	359	382	359
query28	4550	2093	2030	2030
query29	851	612	598	598
query30	239	179	171	171
query31	953	742	731	731
query32	60	49	52	49
query33	501	256	237	237
query34	875	484	496	484
query35	804	694	737	694
query36	1042	911	953	911
query37	110	71	69	69
query38	3511	3354	3368	3354
query39	1069	1056	1014	1014
query40	173	130	130	130
query41	46	44	45	44
query42	104	97	96	96
query43	587	554	526	526
query44	1228	724	756	724
query45	290	280	279	279
query46	1069	724	724	724
query47	2036	2005	1974	1974
query48	362	296	296	296
query49	691	368	380	368
query50	778	388	390	388
query51	6748	6677	6768	6677
query52	101	90	87	87
query53	362	274	285	274
query54	250	226	224	224
query55	74	78	73	73
query56	252	228	229	228
query57	1273	1209	1214	1209
query58	219	203	202	202
query59	3423	3043	2959	2959
query60	247	226	229	226
query61	88	92	87	87
query62	535	437	431	431
query63	300	277	269	269
query64	4171	3695	3880	3695
query65	3092	2977	2991	2977
query66	751	309	357	309
query67	15493	14898	14907	14898
query68	7250	530	534	530
query69	566	306	303	303
query70	1291	1178	1187	1178
query71	461	254	266	254
query72	6408	2631	2429	2429
query73	832	321	316	316
query74	7090	6438	6370	6370
query75	3464	2654	2692	2654
query76	4147	1143	1131	1131
query77	613	244	234	234
query78	10904	10589	10283	10283
query79	6152	499	513	499
query80	1985	418	414	414
query81	535	239	238	238
query82	1599	96	92	92
query83	345	164	167	164
query84	258	85	81	81
query85	853	268	264	264
query86	470	294	316	294
query87	3508	3263	3267	3263
query88	5779	2424	2261	2261
query89	466	363	362	362
query90	2107	174	176	174
query91	119	93	96	93
query92	56	48	44	44
query93	5483	498	498	498
query94	1257	176	173	173
query95	367	1234	290	290
query96	606	258	264	258
query97	3075	2907	2922	2907
query98	237	221	216	216
query99	1155	864	872	864
Total cold run time: 291716 ms
Total hot run time: 182688 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ca2f68a98f65bc132be68462fdff164e5f2cd26, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.23	0.04	0.04
query4	1.69	0.07	0.07
query5	0.49	0.48	0.48
query6	1.44	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.50	0.50
query10	0.56	0.56	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.58
query14	0.76	0.76	0.77
query15	0.82	0.80	0.80
query16	0.37	0.41	0.37
query17	1.03	0.99	1.01
query18	0.22	0.24	0.22
query19	1.86	1.65	1.78
query20	0.02	0.01	0.00
query21	15.41	0.66	0.64
query22	3.82	7.39	2.81
query23	18.28	1.41	1.16
query24	2.05	0.21	0.22
query25	0.13	0.08	0.09
query26	0.28	0.17	0.16
query27	0.07	0.08	0.07
query28	13.31	1.00	0.97
query29	12.62	3.29	3.29
query30	0.25	0.06	0.05
query31	2.87	0.38	0.38
query32	3.26	0.46	0.46
query33	2.79	2.81	2.81
query34	17.06	4.34	4.47
query35	4.45	4.50	4.55
query36	0.65	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.18	0.17	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.16 s
Total hot run time: 31.05 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2ca2f68a98f65bc132be68462fdff164e5f2cd26 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 16, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 0f19aaf into apache:master Apr 17, 2024
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Apr 17, 2024
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Apr 17, 2024
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Apr 17, 2024
@w41ter w41ter deleted the add_index_to_signature branch April 17, 2024 08:04
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Apr 17, 2024
dataroaring pushed a commit that referenced this pull request Apr 17, 2024
* [fix](restore) Reset index id for restore (#33648)

* [fix](restore) add indexes as part of table signature (#33650)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
* [fix](restore) Reset index id for restore (apache#33648)

* [fix](restore) add indexes as part of table signature (apache#33650)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.9-merged dev/3.0.0-merged p0_b reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants