Skip to content

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Mar 12, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32517 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f7a61c874943c6d828a274a0087612ec5056561c, data reload: false

------ Round 1 ----------------------------------
q1	17599	5287	5072	5072
q2	2039	286	172	172
q3	10418	1362	695	695
q4	10219	1061	542	542
q5	7691	2408	2361	2361
q6	192	164	142	142
q7	934	752	601	601
q8	9300	1259	1052	1052
q9	4942	4758	4891	4758
q10	6839	2325	1883	1883
q11	458	285	263	263
q12	353	362	223	223
q13	17754	3670	3084	3084
q14	232	233	207	207
q15	527	481	484	481
q16	633	619	590	590
q17	564	861	352	352
q18	7058	6441	6390	6390
q19	1227	946	554	554
q20	329	322	197	197
q21	2792	2113	1907	1907
q22	1069	1064	991	991
Total cold run time: 103169 ms
Total hot run time: 32517 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5249	5203	5179	5179
q2	245	322	227	227
q3	2231	2702	2317	2317
q4	1441	1847	1392	1392
q5	4233	4131	4139	4131
q6	211	167	125	125
q7	1915	1967	1810	1810
q8	2630	2670	2611	2611
q9	7389	7422	7012	7012
q10	3034	3246	2779	2779
q11	569	522	490	490
q12	692	760	610	610
q13	3475	3909	3363	3363
q14	307	313	281	281
q15	532	483	495	483
q16	664	693	642	642
q17	1151	1613	1372	1372
q18	7802	7707	7497	7497
q19	829	809	861	809
q20	2008	2037	1808	1808
q21	5391	4980	4951	4951
q22	1139	1076	1046	1046
Total cold run time: 53137 ms
Total hot run time: 50935 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192222 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f7a61c874943c6d828a274a0087612ec5056561c, data reload: false

query1	1386	1030	991	991
query2	6059	1889	1872	1872
query3	11114	4764	4758	4758
query4	54680	25795	23034	23034
query5	5009	515	492	492
query6	343	206	224	206
query7	4867	493	291	291
query8	302	240	228	228
query9	5897	2618	2626	2618
query10	431	295	257	257
query11	15201	15111	14914	14914
query12	161	109	106	106
query13	1036	519	394	394
query14	10715	6515	6472	6472
query15	202	195	188	188
query16	7124	660	472	472
query17	1066	692	569	569
query18	1633	422	326	326
query19	204	198	158	158
query20	131	127	123	123
query21	213	127	108	108
query22	4462	4670	4342	4342
query23	34105	33459	33698	33459
query24	5644	2428	2418	2418
query25	465	456	386	386
query26	674	240	163	163
query27	1976	503	338	338
query28	2826	2458	2465	2458
query29	562	521	430	430
query30	327	233	189	189
query31	885	889	820	820
query32	72	60	64	60
query33	450	389	309	309
query34	760	865	486	486
query35	818	858	769	769
query36	963	998	912	912
query37	125	99	72	72
query38	4141	4200	4186	4186
query39	1506	1473	1413	1413
query40	207	123	105	105
query41	53	53	50	50
query42	128	103	108	103
query43	510	516	497	497
query44	1307	843	811	811
query45	185	176	163	163
query46	826	1036	649	649
query47	1834	1911	1791	1791
query48	393	429	319	319
query49	687	490	404	404
query50	693	733	439	439
query51	4280	4307	4223	4223
query52	113	112	95	95
query53	229	262	191	191
query54	491	523	428	428
query55	85	79	88	79
query56	274	263	257	257
query57	1159	1188	1089	1089
query58	257	249	239	239
query59	2767	2797	2585	2585
query60	279	288	258	258
query61	124	119	115	115
query62	716	732	678	678
query63	245	194	195	194
query64	1640	1008	683	683
query65	4411	4351	4428	4351
query66	686	388	292	292
query67	15857	15552	15356	15356
query68	5507	881	502	502
query69	516	293	275	275
query70	1188	1127	1139	1127
query71	429	290	257	257
query72	5727	3624	3730	3624
query73	1374	743	348	348
query74	8984	9218	8800	8800
query75	3324	3198	2748	2748
query76	3723	1187	735	735
query77	547	373	281	281
query78	10073	10193	9373	9373
query79	1301	831	587	587
query80	804	560	499	499
query81	512	263	234	234
query82	422	131	97	97
query83	189	188	173	173
query84	288	101	82	82
query85	801	417	378	378
query86	366	303	297	297
query87	4403	4422	4435	4422
query88	2926	2275	2261	2261
query89	388	304	286	286
query90	1712	217	224	217
query91	139	136	106	106
query92	62	61	59	59
query93	1380	1083	571	571
query94	616	410	298	298
query95	352	273	266	266
query96	487	559	277	277
query97	3322	3415	3304	3304
query98	217	211	203	203
query99	1323	1416	1304	1304
Total cold run time: 292373 ms
Total hot run time: 192222 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f7a61c874943c6d828a274a0087612ec5056561c, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.11
query5	0.56	0.54	0.56
query6	1.18	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.51	0.53
query10	0.60	0.61	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	2.69	2.70	2.83
query15	0.92	0.86	0.85
query16	0.39	0.37	0.39
query17	1.02	1.00	1.01
query18	0.21	0.20	0.20
query19	1.90	1.87	1.97
query20	0.01	0.01	0.01
query21	15.38	0.89	0.55
query22	0.75	1.19	0.65
query23	14.96	1.39	0.64
query24	6.98	2.19	0.89
query25	0.51	0.21	0.09
query26	0.51	0.16	0.13
query27	0.06	0.05	0.05
query28	10.29	0.83	0.42
query29	12.57	3.98	3.30
query30	0.26	0.10	0.07
query31	2.83	0.61	0.39
query32	3.23	0.55	0.46
query33	3.02	3.01	3.03
query34	15.77	5.15	4.56
query35	4.63	4.59	4.58
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.96 s
Total hot run time: 31.05 s

@liaoxin01 liaoxin01 force-pushed the streamload_json_compress branch from f7a61c8 to e9efea4 Compare March 13, 2025 02:47
@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32797 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9efea46ce076eb422934b210005737c1b8adb1a, data reload: false

------ Round 1 ----------------------------------
q1	17647	5191	5048	5048
q2	2047	289	161	161
q3	10514	1248	725	725
q4	10204	1025	531	531
q5	7559	2306	2384	2306
q6	183	162	130	130
q7	917	745	603	603
q8	9305	1296	1064	1064
q9	4899	4835	4873	4835
q10	6856	2315	1938	1938
q11	490	279	253	253
q12	362	359	222	222
q13	17941	3834	3262	3262
q14	232	221	210	210
q15	537	489	481	481
q16	620	622	588	588
q17	570	859	346	346
q18	6744	6530	6397	6397
q19	1519	965	582	582
q20	336	326	201	201
q21	3031	2084	1917	1917
q22	1042	1036	997	997
Total cold run time: 103555 ms
Total hot run time: 32797 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5182	5140	5166	5140
q2	237	336	231	231
q3	2197	2703	2296	2296
q4	1444	1838	1389	1389
q5	4280	4119	4165	4119
q6	206	163	125	125
q7	1903	1932	1802	1802
q8	2571	2634	2529	2529
q9	7213	7226	7168	7168
q10	3044	3230	2758	2758
q11	571	506	502	502
q12	674	744	602	602
q13	3617	4031	3373	3373
q14	304	303	276	276
q15	513	488	476	476
q16	662	712	670	670
q17	1153	1647	1319	1319
q18	7798	7555	7456	7456
q19	847	803	927	803
q20	1971	2060	1895	1895
q21	5478	4923	4677	4677
q22	1090	1067	1048	1048
Total cold run time: 52955 ms
Total hot run time: 50654 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191885 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9efea46ce076eb422934b210005737c1b8adb1a, data reload: false

query1	1420	999	974	974
query2	6217	1871	1856	1856
query3	10972	4327	4306	4306
query4	57981	26032	22964	22964
query5	5254	506	476	476
query6	379	211	187	187
query7	5129	506	287	287
query8	334	262	240	240
query9	7065	2614	2656	2614
query10	423	312	254	254
query11	15356	15054	14910	14910
query12	161	108	109	108
query13	1201	532	416	416
query14	11188	6811	7006	6811
query15	217	209	183	183
query16	7201	684	469	469
query17	1103	754	586	586
query18	1661	411	339	339
query19	221	199	180	180
query20	120	128	119	119
query21	219	129	105	105
query22	4567	4527	4285	4285
query23	33960	33297	33385	33297
query24	5706	2473	2494	2473
query25	445	461	408	408
query26	682	275	163	163
query27	1760	515	343	343
query28	2783	2489	2460	2460
query29	568	558	442	442
query30	268	223	194	194
query31	890	912	792	792
query32	86	62	66	62
query33	479	374	299	299
query34	750	864	509	509
query35	812	838	769	769
query36	971	1028	919	919
query37	124	100	75	75
query38	4236	4326	4144	4144
query39	1482	1440	1457	1440
query40	217	119	102	102
query41	52	51	49	49
query42	123	106	103	103
query43	532	531	500	500
query44	1322	823	810	810
query45	177	171	165	165
query46	866	1043	647	647
query47	1878	1869	1787	1787
query48	389	440	303	303
query49	731	501	420	420
query50	690	741	419	419
query51	4306	4377	4218	4218
query52	116	104	97	97
query53	234	249	186	186
query54	487	507	440	440
query55	84	82	78	78
query56	292	279	254	254
query57	1193	1185	1130	1130
query58	250	243	241	241
query59	2643	2726	2597	2597
query60	278	272	292	272
query61	130	124	120	120
query62	723	753	675	675
query63	226	201	209	201
query64	1733	1041	690	690
query65	4696	4498	4476	4476
query66	768	403	306	306
query67	15901	15596	15567	15567
query68	7610	862	494	494
query69	540	303	253	253
query70	1241	1149	1134	1134
query71	533	294	256	256
query72	5768	3578	3759	3578
query73	1273	729	362	362
query74	9437	8955	8703	8703
query75	3684	3148	2713	2713
query76	4132	1191	746	746
query77	625	370	281	281
query78	10014	10116	9261	9261
query79	2307	821	593	593
query80	661	525	446	446
query81	492	258	243	243
query82	690	130	94	94
query83	172	172	147	147
query84	285	97	73	73
query85	877	354	319	319
query86	425	324	277	277
query87	4494	4492	4458	4458
query88	3399	2281	2280	2280
query89	420	314	282	282
query90	1912	205	204	204
query91	145	143	110	110
query92	69	58	53	53
query93	1286	1058	576	576
query94	678	420	300	300
query95	348	277	264	264
query96	482	564	277	277
query97	3320	3353	3312	3312
query98	223	207	201	201
query99	1692	1398	1284	1284
Total cold run time: 304662 ms
Total hot run time: 191885 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9efea46ce076eb422934b210005737c1b8adb1a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.11	0.11
query5	0.57	0.55	0.55
query6	1.20	0.72	0.71
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.60	0.51	0.53
query10	0.58	0.60	0.58
query11	0.16	0.10	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.59
query14	2.73	2.67	2.67
query15	0.92	0.84	0.84
query16	0.36	0.38	0.38
query17	0.98	1.02	0.99
query18	0.21	0.20	0.19
query19	1.93	1.97	1.76
query20	0.02	0.01	0.01
query21	15.36	0.90	0.56
query22	0.74	1.04	0.64
query23	15.09	1.36	0.63
query24	6.65	2.15	1.32
query25	0.44	0.16	0.12
query26	0.61	0.16	0.14
query27	0.05	0.05	0.04
query28	9.46	0.82	0.44
query29	12.53	4.06	3.34
query30	0.27	0.09	0.07
query31	2.82	0.62	0.38
query32	3.23	0.56	0.46
query33	3.02	2.95	3.08
query34	15.80	5.10	4.52
query35	4.53	4.53	4.51
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.79 s
Total hot run time: 31.17 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring added dev/3.0.x usercase Important user case type label labels Mar 13, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 88d5226 into apache:master Mar 13, 2025
26 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Mar 17, 2025
… format #48990 (#49044)

Cherry-picked from #48990

Co-authored-by: Xin Liao <liaoxin@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants