Skip to content

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 12, 2025

pick #48748

…-calculate for all historical data if cache missed in publish phase (apache#48748)
@bobhan1 bobhan1 requested a review from dataroaring as a code owner March 12, 2025 06:15
@Thearas
Copy link
Contributor

Thearas commented Mar 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 12, 2025

run buildall

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 12, 2025

run buildall

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 12, 2025

run cloud_p0

@bobhan1 bobhan1 changed the title branch-3.0-pick: [Fix](cloud) Should calculate delete bitmaps between segments when re… branch-3.0-pick: [Fix](cloud) Should calculate delete bitmaps between segments when re-calculate for all historical data if cache missed in publish phase #48748 Mar 12, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 40198 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be16ec6e19ef5fbeda867d91d65b846013df495e, data reload: false

------ Round 1 ----------------------------------
q1	17581	6757	6603	6603
q2	2059	171	166	166
q3	10765	1107	1179	1107
q4	10539	699	834	699
q5	7754	2857	2794	2794
q6	227	132	130	130
q7	976	615	594	594
q8	9342	1985	2039	1985
q9	6627	6446	6432	6432
q10	7039	2288	2303	2288
q11	476	261	255	255
q12	398	207	212	207
q13	17791	2986	2955	2955
q14	229	211	217	211
q15	508	463	466	463
q16	678	593	594	593
q17	981	560	593	560
q18	7428	6823	6821	6821
q19	1447	1088	1088	1088
q20	469	208	199	199
q21	4005	3058	3152	3058
q22	1088	990	1000	990
Total cold run time: 108407 ms
Total hot run time: 40198 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6606	6600	6529	6529
q2	332	235	241	235
q3	2914	2686	2926	2686
q4	2044	1829	1818	1818
q5	5776	5745	5730	5730
q6	212	130	125	125
q7	2262	1763	1791	1763
q8	3379	3581	3486	3486
q9	8957	8826	8886	8826
q10	3596	3507	3560	3507
q11	599	507	514	507
q12	823	614	586	586
q13	8435	3173	3175	3173
q14	304	273	276	273
q15	508	470	463	463
q16	707	643	649	643
q17	1846	1623	1626	1623
q18	8393	7793	7847	7793
q19	1681	1625	1558	1558
q20	2054	1861	1870	1861
q21	5521	5340	5344	5340
q22	1109	1103	983	983
Total cold run time: 68058 ms
Total hot run time: 59508 ms

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 38.91% (10163/26122)
Line Coverage 30.31% (86579/285603)
Region Coverage 29.38% (44504/151502)
Branch Coverage 25.88% (22634/87442)

@doris-robot
Copy link

TPC-DS: Total hot run time: 197404 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be16ec6e19ef5fbeda867d91d65b846013df495e, data reload: false

query1	1272	890	894	890
query2	6498	2020	2044	2020
query3	10768	4408	4369	4369
query4	60835	32047	23291	23291
query5	4680	463	454	454
query6	411	180	177	177
query7	5484	309	311	309
query8	311	228	216	216
query9	8326	2583	2582	2582
query10	445	279	255	255
query11	16461	15121	15974	15121
query12	158	104	103	103
query13	1441	457	437	437
query14	9973	7348	7059	7059
query15	204	195	171	171
query16	6639	421	477	421
query17	1263	596	593	593
query18	1763	308	302	302
query19	196	158	162	158
query20	116	112	110	110
query21	215	104	107	104
query22	5070	4765	4912	4765
query23	34571	34077	34178	34077
query24	6305	2962	2914	2914
query25	508	413	415	413
query26	683	169	165	165
query27	1818	345	355	345
query28	4268	2473	2417	2417
query29	689	435	422	422
query30	232	162	162	162
query31	1003	850	848	848
query32	65	53	55	53
query33	377	281	286	281
query34	912	502	521	502
query35	877	728	717	717
query36	1094	966	957	957
query37	116	74	69	69
query38	4080	4001	4028	4001
query39	1548	1496	1490	1490
query40	213	105	106	105
query41	54	52	50	50
query42	117	110	103	103
query43	523	507	498	498
query44	1178	802	805	802
query45	186	168	171	168
query46	1154	738	726	726
query47	2065	1987	2026	1987
query48	481	396	394	394
query49	760	398	404	398
query50	882	431	430	430
query51	7522	7208	7223	7208
query52	106	89	89	89
query53	261	188	184	184
query54	584	485	492	485
query55	82	80	79	79
query56	280	262	277	262
query57	1334	1198	1163	1163
query58	227	218	221	218
query59	3186	2857	2930	2857
query60	266	245	241	241
query61	140	105	101	101
query62	745	676	663	663
query63	215	184	185	184
query64	1407	679	633	633
query65	3274	3178	3157	3157
query66	674	298	309	298
query67	16161	15702	15742	15702
query68	4161	579	562	562
query69	435	263	263	263
query70	1154	1109	1141	1109
query71	351	272	257	257
query72	6361	3977	3986	3977
query73	754	349	366	349
query74	10348	8941	9217	8941
query75	3350	2624	2659	2624
query76	1872	1094	1078	1078
query77	492	268	268	268
query78	10848	9663	9605	9605
query79	1766	603	602	602
query80	1116	418	425	418
query81	537	241	238	238
query82	257	87	90	87
query83	183	141	142	141
query84	281	78	75	75
query85	939	303	289	289
query86	390	290	297	290
query87	4490	4226	4236	4226
query88	4001	2417	2385	2385
query89	406	288	290	288
query90	1870	183	184	183
query91	180	145	151	145
query92	57	48	47	47
query93	2588	562	550	550
query94	759	302	289	289
query95	354	259	251	251
query96	619	285	283	283
query97	3316	3173	3196	3173
query98	218	205	200	200
query99	1593	1286	1297	1286
Total cold run time: 311798 ms
Total hot run time: 197404 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be16ec6e19ef5fbeda867d91d65b846013df495e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.11
query5	0.52	0.52	0.52
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.49
query10	0.54	0.55	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.87	2.74	2.84
query15	0.89	0.83	0.81
query16	0.37	0.41	0.40
query17	1.08	1.08	1.06
query18	0.24	0.21	0.21
query19	1.80	1.85	1.99
query20	0.02	0.01	0.01
query21	15.37	0.58	0.58
query22	2.74	2.81	2.61
query23	16.75	1.04	0.83
query24	3.44	0.33	1.64
query25	0.25	0.12	0.08
query26	0.47	0.14	0.13
query27	0.04	0.04	0.04
query28	10.24	0.46	0.45
query29	12.63	3.22	3.26
query30	0.24	0.06	0.06
query31	2.87	0.38	0.37
query32	3.27	0.46	0.46
query33	2.99	3.01	2.97
query34	17.01	4.47	4.46
query35	4.52	4.49	4.47
query36	0.67	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.91 s
Total hot run time: 32.29 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 12, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 02603b3 into apache:branch-3.0 Mar 12, 2025
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants