Skip to content

branch-3.0-pick: [Fix](mow) Fix some logs for mow (#48523) #48664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 10, 2025
Merged

branch-3.0-pick: [Fix](mow) Fix some logs for mow (#48523) #48664

dataroaring merged 1 commit into from
Mar 10, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 5, 2025

pick #48523

@bobhan1 bobhan1 requested a review from dataroaring as a code owner March 5, 2025 03:11
@Thearas
Copy link
Contributor

Thearas commented Mar 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 5, 2025

run buildall

@bobhan1 @dataroaring
[Fix](mow) Fix some logs for mow (apache#48523)
fcc5fc9 
1. remove rows check for newly generated block in publish in partial
update. When the rowset has multi segments, newly generated segments of
conflict rows on these segments will be flushed concurrently using the
same rowset writer. So `rowset_writer->num_rows()` may be larger than
`new_generated_rows`.
2. correct `rowset_ids to add:` size when calculate delete bitmap in
publish phase. Include invisible rowsets for txn_load
3. add a config `enable_mow_verbose_log` to control whether to print
verbose log for delete bitmap calculation.
@dataroaring dataroaring force-pushed the branch-3.0-pick-48523 branch from efc8719 to fcc5fc9 Compare March 7, 2025 02:18
@dataroaring
Copy link
Contributor

dataroaring commented Mar 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 7, 2025

TPC-H: Total hot run time: 40243 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fcc5fc9ba088a6a606fb99659437a352c6f97a21, data reload: false

------ Round 1 ----------------------------------
q1	17623	6833	6640	6640
q2	2074	177	171	171
q3	10569	1090	1190	1090
q4	10580	769	755	755
q5	7739	2920	2861	2861
q6	221	131	131	131
q7	1001	617	606	606
q8	9371	2015	2047	2015
q9	6659	6406	6468	6406
q10	6992	2240	2308	2240
q11	469	258	268	258
q12	406	209	208	208
q13	17760	2997	2999	2997
q14	229	218	222	218
q15	510	462	478	462
q16	664	610	602	602
q17	982	506	575	506
q18	7355	6733	6637	6637
q19	1402	1112	1051	1051
q20	494	209	201	201
q21	4052	3231	3241	3231
q22	1070	957	989	957
Total cold run time: 108222 ms
Total hot run time: 40243 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6619	6659	6655	6655
q2	336	236	228	228
q3	2955	2764	2952	2764
q4	2096	1814	1830	1814
q5	5793	5807	5739	5739
q6	219	124	127	124
q7	2233	1855	1873	1855
q8	3419	3566	3516	3516
q9	8846	8926	8877	8877
q10	3532	3517	3533	3517
q11	599	478	509	478
q12	820	611	600	600
q13	9718	3243	3178	3178
q14	314	286	270	270
q15	508	474	466	466
q16	701	640	663	640
q17	1860	1642	1627	1627
q18	8428	7760	7796	7760
q19	1670	1587	1551	1551
q20	2067	1842	1895	1842
q21	5486	5355	5338	5338
q22	1117	1044	1056	1044
Total cold run time: 69336 ms
Total hot run time: 59883 ms

@doris-robot
Copy link

doris-robot commented Mar 7, 2025

TPC-DS: Total hot run time: 198452 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fcc5fc9ba088a6a606fb99659437a352c6f97a21, data reload: false

query1	1269	909	883	883
query2	6235	2041	1989	1989
query3	10833	4335	4327	4327
query4	61016	29002	23708	23708
query5	5186	477	458	458
query6	410	179	206	179
query7	5481	328	319	319
query8	310	227	224	224
query9	8362	2655	2652	2652
query10	454	260	260	260
query11	17668	15058	15699	15058
query12	159	107	117	107
query13	1475	447	445	445
query14	10513	7761	7337	7337
query15	208	182	185	182
query16	7022	493	493	493
query17	1297	589	596	589
query18	1821	326	339	326
query19	214	172	159	159
query20	115	112	114	112
query21	209	105	107	105
query22	4794	4442	4512	4442
query23	34441	34168	34231	34168
query24	6484	2930	2944	2930
query25	566	431	427	427
query26	661	178	182	178
query27	1841	389	366	366
query28	4051	2511	2441	2441
query29	697	435	460	435
query30	246	161	158	158
query31	966	848	839	839
query32	66	56	55	55
query33	469	284	296	284
query34	958	520	524	520
query35	831	770	728	728
query36	1074	977	953	953
query37	120	82	69	69
query38	4140	4115	4075	4075
query39	1501	1493	1465	1465
query40	205	103	101	101
query41	46	47	50	47
query42	113	102	103	102
query43	526	500	493	493
query44	1188	835	859	835
query45	192	174	167	167
query46	1156	758	740	740
query47	1987	1928	1932	1928
query48	478	387	393	387
query49	736	406	392	392
query50	864	427	421	421
query51	7519	7124	7328	7124
query52	102	93	89	89
query53	264	182	180	180
query54	587	475	486	475
query55	83	73	81	73
query56	277	296	256	256
query57	1305	1162	1156	1156
query58	221	207	218	207
query59	3211	2948	2901	2901
query60	276	262	250	250
query61	132	128	115	115
query62	768	663	680	663
query63	212	189	188	188
query64	1414	678	653	653
query65	3288	3194	3248	3194
query66	699	302	299	299
query67	16034	15527	15682	15527
query68	4294	604	577	577
query69	433	267	279	267
query70	1179	1092	1142	1092
query71	353	270	255	255
query72	6353	4046	4052	4046
query73	748	360	358	358
query74	10428	9242	9223	9223
query75	3352	2663	2692	2663
query76	1955	1158	1088	1088
query77	503	278	271	271
query78	10593	9701	9692	9692
query79	1293	604	593	593
query80	844	437	423	423
query81	517	238	238	238
query82	1264	93	88	88
query83	250	142	144	142
query84	282	80	81	80
query85	906	317	332	317
query86	338	299	291	291
query87	4440	4233	4392	4233
query88	3609	2432	2392	2392
query89	423	290	286	286
query90	2019	191	184	184
query91	184	151	150	150
query92	62	51	50	50
query93	1614	555	552	552
query94	765	295	299	295
query95	362	257	254	254
query96	611	277	279	277
query97	3360	3167	3201	3167
query98	219	197	204	197
query99	1628	1318	1303	1303
Total cold run time: 313227 ms
Total hot run time: 198452 ms

@doris-robot
Copy link

doris-robot commented Mar 7, 2025

ClickBench: Total hot run time: 31.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fcc5fc9ba088a6a606fb99659437a352c6f97a21, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.63	0.11	0.11
query5	0.52	0.51	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.51
query10	0.56	0.55	0.57
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.77	2.75	2.85
query15	0.89	0.82	0.82
query16	0.40	0.39	0.36
query17	1.04	1.00	1.08
query18	0.22	0.21	0.21
query19	1.97	1.79	2.01
query20	0.01	0.01	0.00
query21	15.36	0.59	0.60
query22	2.37	2.66	1.61
query23	17.23	0.93	0.80
query24	3.02	0.84	1.62
query25	0.16	0.06	0.18
query26	0.52	0.13	0.13
query27	0.04	0.04	0.04
query28	10.07	0.54	0.50
query29	12.58	3.22	3.22
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.26	0.46	0.48
query33	3.00	3.01	3.05
query34	17.01	4.48	4.57
query35	4.56	4.53	4.45
query36	0.64	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.44 s
Total hot run time: 31.72 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 7, 2025

run beut

1 similar comment
@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 10, 2025

run beut

dataroaring
dataroaring approved these changes Mar 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit f662722 into apache:branch-3.0 Mar 10, 2025
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bobhan1 @Thearas @dataroaring @doris-robot