Skip to content

[chore](rowset writer) print rowset rows number when meet too many segments #39091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Aug 8, 2024
Merged

[chore](rowset writer) print rowset rows number when meet too many segments #39091

liaoxin01 merged 1 commit into from
Aug 8, 2024

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Aug 8, 2024

Print rowset rows number when meet too many segments.

@doris-robot
Copy link

doris-robot commented Aug 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Aug 8, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 8, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@sollhui
Copy link
Contributor Author

sollhui commented Aug 8, 2024

run buildall

@sollhui sollhui closed this Aug 8, 2024
@sollhui sollhui reopened this Aug 8, 2024
@sollhui
Copy link
Contributor Author

sollhui commented Aug 8, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring approved these changes Aug 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 8, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Aug 8, 2024

TPC-H: Total hot run time: 39317 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49781f63da9833a5f81ef43e9bcf2502fe181ad1, data reload: false

------ Round 1 ----------------------------------
q1	17624	4508	4305	4305
q2	2021	178	173	173
q3	10500	1220	1039	1039
q4	10158	796	695	695
q5	7581	2663	2545	2545
q6	223	141	138	138
q7	978	597	587	587
q8	9405	1902	1910	1902
q9	9752	6573	6569	6569
q10	7090	2191	2258	2191
q11	444	241	248	241
q12	497	221	220	220
q13	17760	2970	3013	2970
q14	279	242	235	235
q15	536	474	522	474
q16	513	382	387	382
q17	978	670	710	670
q18	7964	7498	7461	7461
q19	1382	895	1068	895
q20	684	315	329	315
q21	5361	4386	4312	4312
q22	1109	1002	998	998
Total cold run time: 112839 ms
Total hot run time: 39317 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4332	4243	4258	4243
q2	381	271	270	270
q3	2870	2659	2592	2592
q4	1867	1583	1615	1583
q5	5280	5274	5284	5274
q6	219	134	130	130
q7	2058	1641	1698	1641
q8	3167	3380	3287	3287
q9	8421	8363	8315	8315
q10	3361	3174	3107	3107
q11	590	483	495	483
q12	771	630	601	601
q13	17536	2981	2988	2981
q14	309	268	269	268
q15	518	485	481	481
q16	508	415	414	414
q17	1781	1464	1442	1442
q18	7626	7394	7676	7394
q19	1630	1607	1569	1569
q20	1965	1791	1790	1790
q21	5129	5154	4997	4997
q22	1083	1008	969	969
Total cold run time: 71402 ms
Total hot run time: 53831 ms

@doris-robot
Copy link

doris-robot commented Aug 8, 2024

TPC-DS: Total hot run time: 202480 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49781f63da9833a5f81ef43e9bcf2502fe181ad1, data reload: false

query1	953	408	394	394
query2	6454	1894	1803	1803
query3	6638	205	214	205
query4	34016	23188	23096	23096
query5	4293	490	520	490
query6	288	196	191	191
query7	4577	287	288	287
query8	259	213	198	198
query9	8701	2379	2367	2367
query10	907	900	918	900
query11	17921	14933	14913	14913
query12	144	100	96	96
query13	1640	411	376	376
query14	9572	7888	7988	7888
query15	386	351	328	328
query16	7569	497	427	427
query17	1732	550	545	545
query18	1958	359	335	335
query19	254	239	220	220
query20	111	103	102	102
query21	204	106	102	102
query22	4194	4117	3941	3941
query23	33816	33188	33484	33188
query24	12018	2902	2878	2878
query25	678	393	386	386
query26	1771	150	149	149
query27	2836	273	286	273
query28	7242	1979	1971	1971
query29	1005	405	400	400
query30	308	145	145	145
query31	982	740	714	714
query32	95	54	53	53
query33	737	282	291	282
query34	982	465	479	465
query35	993	802	804	802
query36	1121	898	924	898
query37	252	80	76	76
query38	4255	4181	4133	4133
query39	1436	1389	1368	1368
query40	270	110	114	110
query41	48	44	46	44
query42	113	95	97	95
query43	502	464	457	457
query44	1274	734	757	734
query45	404	376	387	376
query46	1127	755	820	755
query47	1829	1777	1745	1745
query48	382	302	308	302
query49	1197	417	435	417
query50	803	409	416	409
query51	6710	6628	6647	6628
query52	96	93	85	85
query53	256	181	180	180
query54	886	437	456	437
query55	76	76	75	75
query56	272	241	236	236
query57	1143	1060	1083	1060
query58	230	230	224	224
query59	3061	2686	2641	2641
query60	303	261	280	261
query61	97	108	94	94
query62	820	635	639	635
query63	221	184	175	175
query64	10542	2407	1944	1944
query65	3219	3164	3159	3159
query66	1358	332	339	332
query67	15386	14710	14800	14710
query68	4607	545	551	545
query69	429	416	366	366
query70	1175	1133	1105	1105
query71	390	266	265	265
query72	18510	17606	17077	17077
query73	748	339	331	331
query74	9131	8693	8838	8693
query75	3344	2618	2641	2618
query76	2526	1011	979	979
query77	470	331	305	305
query78	9460	9782	8895	8895
query79	2476	518	513	513
query80	2484	494	480	480
query81	611	227	225	225
query82	796	141	135	135
query83	297	145	151	145
query84	278	87	80	80
query85	2279	305	296	296
query86	491	297	294	294
query87	4662	4437	4523	4437
query88	4162	2505	2505	2505
query89	393	285	299	285
query90	1857	195	191	191
query91	135	114	112	112
query92	66	48	105	48
query93	2192	542	532	532
query94	965	286	285	285
query95	347	255	261	255
query96	605	282	274	274
query97	3227	3071	3087	3071
query98	222	193	192	192
query99	1605	1253	1241	1241
Total cold run time: 316863 ms
Total hot run time: 202480 ms

@doris-robot
Copy link

doris-robot commented Aug 8, 2024

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49781f63da9833a5f81ef43e9bcf2502fe181ad1, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.05	0.04
query3	0.23	0.05	0.04
query4	1.68	0.08	0.08
query5	0.52	0.48	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.57	0.49	0.49
query10	0.54	0.53	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.59	0.60	0.59
query14	0.76	0.78	0.77
query15	0.89	0.82	0.81
query16	0.37	0.36	0.37
query17	1.02	1.03	1.03
query18	0.23	0.23	0.22
query19	1.90	1.70	1.69
query20	0.01	0.01	0.01
query21	15.39	0.75	0.66
query22	3.72	6.86	2.37
query23	18.32	1.34	1.30
query24	2.11	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.20	0.20
query27	0.45	0.22	0.22
query28	13.30	1.02	1.00
query29	12.61	3.24	3.27
query30	0.26	0.06	0.05
query31	2.87	0.38	0.39
query32	3.30	0.47	0.47
query33	2.92	2.91	2.87
query34	17.10	4.38	4.34
query35	4.40	4.44	4.42
query36	0.65	0.46	0.46
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.52 s
Total hot run time: 31.02 s

liaoxin01
liaoxin01 approved these changes Aug 8, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 6de038b into apache:master Aug 8, 2024
@sollhui sollhui mentioned this pull request Aug 10, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 10, 2024
@sollhui
[chore](rowset writer) print rowset rows number when meet too many se…
0db1583 
…gments (#39091) (#39182)

pick (#39091)
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@sollhui @dataroaring
[chore](rowset writer) print rowset rows number when meet too many se…
ed058c3 
…gments (#39091)
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@sollhui @wyxxxcat
[chore](rowset writer) print rowset rows number when meet too many se…
33ea70a 
…gments (apache#39091)
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@sollhui @dataroaring
[chore](rowset writer) print rowset rows number when meet too many se…
7abcbed 
…gments (#39091)
@sollhui sollhui mentioned this pull request Aug 18, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 19, 2024
@sollhui
[chore](rowset writer) print rowset rows number when meet too many se…
b8f23cb 
…gments (#39091) (#39530)

pick (#39091)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@sollhui @Baymine
[chore](rowset writer) print rowset rows number when meet too many se…
5a30d65 
…gments (apache#39091) (apache#39530)

pick (apache#39091)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged doing reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sollhui @doris-robot @liaoxin01 @dataroaring @yiguolei