Skip to content

Conversation

@freemandealer
Copy link
Contributor

This adjustment is based on the analysis that each entry in the params map is roughly 1KB, thus the old limit resulted in a 50KB footprint. With no concurrency issues, expanding the batch size to 1000 should enhance processing efficiency with a reasonable 1MB memory footprint.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@freemandealer
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41750 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3889e9efd82b500d9992fffc9b463f24fc5e93f4, data reload: false

------ Round 1 ----------------------------------
q1	18151	7495	7358	7358
q2	2503	156	155	155
q3	10590	1145	1141	1141
q4	10207	779	757	757
q5	7738	3085	3062	3062
q6	235	150	149	149
q7	1010	608	612	608
q8	9438	2000	2063	2000
q9	6792	6385	6419	6385
q10	7026	2278	2256	2256
q11	438	243	256	243
q12	425	212	212	212
q13	17770	2960	2983	2960
q14	246	223	210	210
q15	566	521	534	521
q16	676	641	605	605
q17	982	843	794	794
q18	7378	6753	6837	6753
q19	1415	1058	1022	1022
q20	581	304	294	294
q21	4025	3360	3253	3253
q22	1104	1012	1019	1012
Total cold run time: 109296 ms
Total hot run time: 41750 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7302	7243	7293	7243
q2	327	232	230	230
q3	2873	2771	2761	2761
q4	1949	1731	1688	1688
q5	5380	5404	5397	5397
q6	234	145	144	144
q7	2081	1689	1675	1675
q8	3165	3301	3348	3301
q9	8416	8415	8445	8415
q10	3406	3363	3351	3351
q11	585	484	474	474
q12	806	595	582	582
q13	8787	2958	2989	2958
q14	309	269	264	264
q15	569	500	511	500
q16	693	654	670	654
q17	1772	1551	1520	1520
q18	7694	7354	7566	7354
q19	1688	1666	1439	1439
q20	2049	1832	1818	1818
q21	5390	5106	5114	5106
q22	1136	1038	996	996
Total cold run time: 66611 ms
Total hot run time: 57870 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193674 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3889e9efd82b500d9992fffc9b463f24fc5e93f4, data reload: false

query1	939	362	373	362
query2	6544	2066	2008	2008
query3	6707	211	221	211
query4	34562	23377	23384	23377
query5	4316	472	456	456
query6	268	177	163	163
query7	4609	291	304	291
query8	283	224	221	221
query9	9599	2656	2656	2656
query10	477	275	279	275
query11	18228	15088	15161	15088
query12	144	98	96	96
query13	1629	406	418	406
query14	10260	7118	6878	6878
query15	250	167	174	167
query16	8059	447	453	447
query17	1644	546	549	546
query18	2121	293	290	290
query19	352	140	145	140
query20	121	104	106	104
query21	207	106	102	102
query22	4587	4237	4376	4237
query23	34589	33894	33948	33894
query24	11143	2917	2849	2849
query25	626	390	383	383
query26	1222	155	161	155
query27	2622	283	283	283
query28	8153	2479	2424	2424
query29	838	403	400	400
query30	321	158	149	149
query31	969	805	808	805
query32	100	56	58	56
query33	775	304	300	300
query34	946	474	485	474
query35	865	747	722	722
query36	1083	944	932	932
query37	157	84	84	84
query38	4196	3878	3931	3878
query39	1457	1402	1426	1402
query40	203	96	96	96
query41	52	49	50	49
query42	118	95	99	95
query43	512	485	475	475
query44	1274	837	803	803
query45	202	168	168	168
query46	1136	756	767	756
query47	1896	1794	1842	1794
query48	449	357	370	357
query49	1145	425	409	409
query50	809	401	415	401
query51	7075	6800	6903	6800
query52	101	90	89	89
query53	255	187	183	183
query54	1164	459	471	459
query55	84	78	76	76
query56	282	268	255	255
query57	1200	1105	1095	1095
query58	243	236	237	236
query59	3042	2954	2853	2853
query60	307	285	282	282
query61	125	138	123	123
query62	850	656	672	656
query63	223	189	185	185
query64	4286	725	707	707
query65	3280	3188	3178	3178
query66	1425	315	314	314
query67	16041	15596	15463	15463
query68	3103	561	558	558
query69	557	311	287	287
query70	1218	1098	1062	1062
query71	328	269	264	264
query72	6170	4121	3956	3956
query73	744	327	329	327
query74	9622	9008	8981	8981
query75	3380	2674	2681	2674
query76	2046	862	937	862
query77	417	303	307	303
query78	9924	9605	9341	9341
query79	952	879	851	851
query80	701	575	574	574
query81	497	254	256	254
query82	373	234	239	234
query83	286	168	160	160
query84	234	103	102	102
query85	752	380	366	366
query86	316	333	298	298
query87	4460	4363	4386	4363
query88	4701	4084	4067	4067
query89	366	359	364	359
query90	2029	314	315	314
query91	165	163	165	163
query92	76	71	71	71
query93	920	912	908	908
query94	767	377	354	354
query95	459	453	413	413
query96	484	488	490	488
query97	3130	3145	3153	3145
query98	240	224	221	221
query99	1396	1296	1286	1286
Total cold run time: 294560 ms
Total hot run time: 193674 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3889e9efd82b500d9992fffc9b463f24fc5e93f4, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.65	0.10	0.10
query5	0.52	0.50	0.53
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.48
query10	0.55	0.57	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.59	0.59
query14	2.97	3.00	3.02
query15	0.90	0.80	0.81
query16	0.39	0.39	0.38
query17	1.06	1.07	1.05
query18	0.22	0.20	0.20
query19	1.94	1.89	2.04
query20	0.01	0.01	0.01
query21	15.36	0.60	0.58
query22	3.20	3.80	1.86
query23	17.38	0.92	0.81
query24	2.94	1.15	1.87
query25	0.21	0.16	0.04
query26	0.46	0.15	0.13
query27	0.04	0.04	0.04
query28	9.99	1.10	1.07
query29	12.57	3.32	3.30
query30	0.24	0.07	0.06
query31	2.86	0.38	0.38
query32	3.29	0.46	0.45
query33	3.01	3.03	3.02
query34	16.90	4.40	4.43
query35	4.43	4.42	4.45
query36	0.69	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.15	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.01
query43	0.04	0.03	0.03
Total cold run time: 107.19 s
Total hot run time: 33.13 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 21b1c9a into apache:master Sep 18, 2024
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
)

This adjustment is based on the analysis that each entry in the params
map is roughly 1KB, thus the old limit resulted in a 50KB footprint.
With no concurrency issues, expanding the batch size to 1000 should
enhance processing efficiency with a reasonable 1MB memory footprint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants