[enhancement](cloud) fix TTL cache size negative overflow #41987

freemandealer · 2024-10-16T13:15:57Z

reset_range enlarges the size if the download size is larger than the original size. TTL LRU queue missed that while still subtracting the larger size when entries were removed from the queue, causing the size metric negative overflow.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

doris-robot · 2024-10-16T13:16:03Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

freemandealer · 2024-10-16T13:16:06Z

run buildall

github-actions · 2024-10-16T13:20:21Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-16T13:26:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-16T13:26:37Z

PR approved by anyone and no changes requested.

doris-robot · 2024-10-16T13:42:16Z

TeamCity be ut coverage result:
Function Coverage: 37.46% (9709/25917)
Line Coverage: 28.72% (80605/280617)
Region Coverage: 28.17% (41710/148086)
Branch Coverage: 24.75% (21210/85702)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5a6f25c33c67d0b87de51d7276d7034e4bc9b098_5a6f25c33c67d0b87de51d7276d7034e4bc9b098/report/index.html

doris-robot · 2024-10-16T13:46:22Z

TPC-H: Total hot run time: 41289 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a6f25c33c67d0b87de51d7276d7034e4bc9b098, data reload: false

------ Round 1 ----------------------------------
q1	17572	7422	7232	7232
q2	2024	279	282	279
q3	12039	1065	1178	1065
q4	10581	821	888	821
q5	7764	3102	3062	3062
q6	244	151	146	146
q7	1039	634	608	608
q8	9352	1925	1978	1925
q9	7255	6430	6413	6413
q10	7052	2417	2403	2403
q11	442	248	247	247
q12	405	226	216	216
q13	17793	2968	3007	2968
q14	248	205	217	205
q15	554	525	527	525
q16	641	573	593	573
q17	975	630	535	535
q18	7397	6672	6656	6656
q19	1355	1006	1058	1006
q20	478	185	188	185
q21	3980	3233	3350	3233
q22	1133	1003	986	986
Total cold run time: 110323 ms
Total hot run time: 41289 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7292	7211	7208	7208
q2	323	232	239	232
q3	3075	2984	2940	2940
q4	2139	1860	1746	1746
q5	5786	5790	5793	5790
q6	229	143	144	143
q7	2273	1869	1790	1790
q8	3420	3564	3469	3469
q9	9013	8878	8941	8878
q10	3633	3600	3587	3587
q11	593	492	484	484
q12	873	677	633	633
q13	10983	3186	3164	3164
q14	328	294	263	263
q15	568	530	540	530
q16	683	643	642	642
q17	1852	1609	1597	1597
q18	8293	7848	7505	7505
q19	1736	1479	1501	1479
q20	2147	1907	1861	1861
q21	5582	5476	5448	5448
q22	1121	1049	1090	1049
Total cold run time: 71942 ms
Total hot run time: 60438 ms

doris-robot · 2024-10-16T13:58:28Z

TPC-DS: Total hot run time: 192047 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a6f25c33c67d0b87de51d7276d7034e4bc9b098, data reload: false

query1	926	395	412	395
query2	6238	2113	2028	2028
query3	8674	192	204	192
query4	34251	23421	23450	23421
query5	3492	465	489	465
query6	279	162	174	162
query7	4191	296	287	287
query8	286	230	226	226
query9	9616	2709	2712	2709
query10	470	264	265	264
query11	18137	15235	15316	15235
query12	155	103	104	103
query13	1569	422	426	422
query14	9396	7012	7613	7012
query15	242	170	172	170
query16	7795	453	419	419
query17	1663	618	587	587
query18	1882	316	328	316
query19	382	170	158	158
query20	128	111	112	111
query21	211	106	107	106
query22	4844	4755	4765	4755
query23	35118	34106	34871	34106
query24	10963	2748	2752	2748
query25	629	429	419	419
query26	1254	165	164	164
query27	2469	289	290	289
query28	7712	2428	2458	2428
query29	859	423	419	419
query30	251	156	150	150
query31	1091	794	838	794
query32	100	54	54	54
query33	771	299	296	296
query34	930	507	514	507
query35	904	740	767	740
query36	1109	964	914	914
query37	166	90	89	89
query38	4046	3904	3958	3904
query39	1490	1423	1411	1411
query40	266	99	100	99
query41	47	49	45	45
query42	127	100	98	98
query43	529	495	501	495
query44	1203	802	791	791
query45	194	164	167	164
query46	1145	683	712	683
query47	1951	1836	1841	1836
query48	424	316	317	316
query49	885	432	432	432
query50	822	383	385	383
query51	7153	6938	7063	6938
query52	104	86	93	86
query53	257	179	181	179
query54	1126	434	443	434
query55	83	76	79	76
query56	289	275	301	275
query57	1296	1141	1159	1141
query58	226	235	234	234
query59	3184	2983	3088	2983
query60	295	275	280	275
query61	109	106	101	101
query62	825	666	664	664
query63	218	185	182	182
query64	4097	650	604	604
query65	3330	3234	3197	3197
query66	771	316	315	315
query67	16176	15696	15769	15696
query68	4021	575	563	563
query69	534	333	288	288
query70	1190	1127	1138	1127
query71	340	284	278	278
query72	7434	3970	3940	3940
query73	788	357	363	357
query74	10125	8983	9180	8983
query75	3400	2661	2659	2659
query76	3141	972	840	840
query77	616	293	288	288
query78	10573	9690	9623	9623
query79	2797	610	627	610
query80	1119	443	464	443
query81	595	252	246	246
query82	645	142	136	136
query83	305	138	142	138
query84	277	67	75	67
query85	1310	308	288	288
query86	450	308	304	304
query87	4500	4333	4379	4333
query88	3814	2238	2195	2195
query89	401	289	299	289
query90	1892	189	182	182
query91	138	101	105	101
query92	59	48	49	48
query93	2566	533	537	533
query94	816	281	293	281
query95	346	250	245	245
query96	619	278	274	274
query97	3291	3202	3188	3188
query98	216	197	201	197
query99	1673	1306	1315	1306
Total cold run time: 301812 ms
Total hot run time: 192047 ms

doris-robot · 2024-10-16T14:03:58Z

ClickBench: Total hot run time: 32.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a6f25c33c67d0b87de51d7276d7034e4bc9b098, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.50	0.49	0.51
query6	1.13	0.71	0.75
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.49
query10	0.54	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.74	2.75	2.72
query15	0.90	0.84	0.85
query16	0.40	0.41	0.39
query17	1.07	1.05	1.05
query18	0.20	0.19	0.20
query19	1.97	1.78	2.00
query20	0.01	0.01	0.01
query21	15.36	0.59	0.57
query22	2.69	2.15	1.80
query23	16.98	1.05	0.81
query24	3.04	1.42	0.86
query25	0.16	0.13	0.17
query26	0.45	0.14	0.14
query27	0.03	0.04	0.04
query28	10.43	1.10	1.08
query29	12.51	3.28	3.29
query30	0.25	0.06	0.07
query31	2.88	0.38	0.38
query32	3.26	0.46	0.46
query33	2.99	3.03	2.98
query34	17.05	4.45	4.46
query35	4.55	4.54	4.48
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.72 s
Total hot run time: 32.57 s

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

freemandealer · 2024-10-17T07:08:27Z

run buildall

github-actions · 2024-10-17T07:17:23Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-17T07:36:06Z

TeamCity be ut coverage result:
Function Coverage: 37.47% (9711/25919)
Line Coverage: 28.73% (80627/280606)
Region Coverage: 28.17% (41715/148080)
Branch Coverage: 24.74% (21204/85694)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e74efff2bec0a6c77657942fe9930d2db4d174a7_e74efff2bec0a6c77657942fe9930d2db4d174a7/report/index.html

doris-robot · 2024-10-17T07:50:41Z

TPC-H: Total hot run time: 41273 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e74efff2bec0a6c77657942fe9930d2db4d174a7, data reload: false

------ Round 1 ----------------------------------
q1	17558	7353	7283	7283
q2	2032	283	282	282
q3	12242	1035	1155	1035
q4	10577	806	808	806
q5	7757	3120	3038	3038
q6	243	149	146	146
q7	1030	614	626	614
q8	9366	1958	1926	1926
q9	6592	6467	6433	6433
q10	7023	2398	2483	2398
q11	451	251	246	246
q12	428	225	217	217
q13	17775	2995	3044	2995
q14	238	208	212	208
q15	574	518	519	518
q16	645	567	573	567
q17	980	573	558	558
q18	7351	6728	6783	6728
q19	1347	977	1038	977
q20	474	181	184	181
q21	4097	3116	3191	3116
q22	1114	1001	1041	1001
Total cold run time: 109894 ms
Total hot run time: 41273 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7256	7231	7186	7186
q2	328	228	230	228
q3	3030	2942	2933	2933
q4	2114	1847	1794	1794
q5	5700	5793	5800	5793
q6	235	149	147	147
q7	2249	1855	1831	1831
q8	3412	3493	3486	3486
q9	9038	8938	8958	8938
q10	3581	3622	3540	3540
q11	583	489	500	489
q12	834	607	633	607
q13	9453	3185	3165	3165
q14	319	271	297	271
q15	577	524	522	522
q16	687	640	646	640
q17	1862	1612	1635	1612
q18	8228	7858	7532	7532
q19	1725	1433	1354	1354
q20	2134	1843	1869	1843
q21	5621	5454	5370	5370
q22	1170	1097	1061	1061
Total cold run time: 70136 ms
Total hot run time: 60342 ms

doris-robot · 2024-10-17T08:02:46Z

TPC-DS: Total hot run time: 192383 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e74efff2bec0a6c77657942fe9930d2db4d174a7, data reload: false

query1	865	408	401	401
query2	6250	2058	1933	1933
query3	8679	197	206	197
query4	34323	23542	23518	23518
query5	4201	475	464	464
query6	261	161	160	160
query7	4191	285	285	285
query8	293	225	225	225
query9	9310	2707	2711	2707
query10	453	265	285	265
query11	17996	15319	15172	15172
query12	141	106	98	98
query13	1557	428	423	423
query14	9755	7553	7553	7553
query15	284	180	177	177
query16	7898	496	488	488
query17	1636	586	578	578
query18	2173	308	299	299
query19	377	157	156	156
query20	117	126	117	117
query21	220	114	109	109
query22	5039	5174	4940	4940
query23	36312	34255	33851	33851
query24	10866	2747	2774	2747
query25	521	423	394	394
query26	735	155	158	155
query27	2335	280	286	280
query28	7557	2441	2409	2409
query29	667	421	421	421
query30	250	156	157	156
query31	1070	806	810	806
query32	92	54	58	54
query33	762	296	290	290
query34	940	509	511	509
query35	901	740	735	735
query36	1066	920	952	920
query37	153	87	89	87
query38	4050	3901	3864	3864
query39	1480	1424	1475	1424
query40	202	98	101	98
query41	49	46	48	46
query42	121	99	97	97
query43	525	467	485	467
query44	1326	826	808	808
query45	201	167	166	166
query46	1143	705	706	705
query47	1971	1860	1849	1849
query48	410	334	322	322
query49	919	426	420	420
query50	838	375	379	375
query51	7213	6966	7001	6966
query52	105	89	90	89
query53	259	182	175	175
query54	1077	436	436	436
query55	82	76	79	76
query56	290	262	257	257
query57	1336	1143	1162	1143
query58	223	251	243	243
query59	3252	2988	2977	2977
query60	294	271	262	262
query61	105	99	102	99
query62	865	672	666	666
query63	219	195	179	179
query64	3868	631	611	611
query65	3287	3200	3181	3181
query66	924	300	295	295
query67	16082	15715	15777	15715
query68	4883	555	549	549
query69	466	287	283	283
query70	1181	1138	1027	1027
query71	340	307	275	275
query72	6961	3965	3990	3965
query73	779	346	348	346
query74	10357	8983	8979	8979
query75	3454	2661	2657	2657
query76	2674	1017	972	972
query77	688	316	297	297
query78	10670	9710	9613	9613
query79	1946	595	590	590
query80	1115	451	445	445
query81	582	236	239	236
query82	652	149	134	134
query83	347	133	132	132
query84	277	68	67	67
query85	1435	304	280	280
query86	418	293	296	293
query87	4404	4483	4470	4470
query88	3500	2205	2155	2155
query89	407	286	288	286
query90	1969	184	184	184
query91	135	104	101	101
query92	66	49	45	45
query93	2275	535	532	532
query94	968	282	286	282
query95	345	242	241	241
query96	624	281	274	274
query97	3255	3144	3089	3089
query98	214	200	190	190
query99	1656	1323	1304	1304
Total cold run time: 302005 ms
Total hot run time: 192383 ms

doris-robot · 2024-10-17T08:08:17Z

ClickBench: Total hot run time: 32.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e74efff2bec0a6c77657942fe9930d2db4d174a7, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.50	0.51
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.55
query10	0.55	0.56	0.54
query11	0.13	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.60	0.61
query14	2.70	2.70	2.72
query15	0.90	0.82	0.84
query16	0.39	0.38	0.40
query17	1.05	0.99	1.01
query18	0.21	0.19	0.19
query19	2.00	1.87	2.04
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.89	2.94	1.12
query23	16.90	1.00	0.78
query24	3.10	1.67	1.59
query25	0.21	0.15	0.20
query26	0.52	0.13	0.12
query27	0.04	0.05	0.03
query28	9.82	1.10	1.08
query29	12.60	3.28	3.25
query30	0.24	0.06	0.05
query31	2.88	0.39	0.39
query32	3.28	0.46	0.46
query33	3.02	3.02	3.00
query34	17.02	4.50	4.53
query35	4.54	4.55	4.50
query36	0.68	0.49	0.51
query37	0.09	0.05	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.48 s
Total hot run time: 32.68 s

dataroaring

LGTM

github-actions · 2024-10-20T02:59:29Z

PR approved by at least one committer and no changes requested.

reset_range enlarges the size if the download size is larger than the original size. TTL LRU queue missed that while still subtracting the larger size when entries were removed from the queue, causing the size metric negative overflow. Pick #41987 Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

[enhancement](cloud) fix TTL cache size negative overflow

5a6f25c

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

gavinchou previously approved these changes Oct 16, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 16, 2024

github-actions bot added the reviewed label Oct 16, 2024

dataroaring added the dev/3.0.x label Oct 16, 2024

response to the reviewer

e74efff

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

freemandealer dismissed gavinchou’s stale review via e74efff October 17, 2024 07:06

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 17, 2024

dataroaring approved these changes Oct 20, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2024

gavinchou approved these changes Oct 22, 2024

View reviewed changes

gavinchou merged commit ff1dd3d into apache:master Oct 22, 2024

freemandealer mentioned this pull request Oct 29, 2024

[enhancement](cloud) fix TTL cache size negative overflow #42869

Merged

dataroaring added dev/3.0.3-merged and removed dev/3.0.x labels Oct 29, 2024

[enhancement](cloud) fix TTL cache size negative overflow #41987

[enhancement](cloud) fix TTL cache size negative overflow #41987

Uh oh!

Conversation

freemandealer commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Oct 16, 2024

Uh oh!

freemandealer commented Oct 16, 2024

Uh oh!

github-actions bot commented Oct 16, 2024

Uh oh!

github-actions bot commented Oct 16, 2024

Uh oh!

github-actions bot commented Oct 16, 2024

Uh oh!

doris-robot commented Oct 16, 2024

Uh oh!

doris-robot commented Oct 16, 2024

Uh oh!

doris-robot commented Oct 16, 2024

Uh oh!

doris-robot commented Oct 16, 2024

Uh oh!

freemandealer commented Oct 17, 2024

Uh oh!

github-actions bot commented Oct 17, 2024

Uh oh!

doris-robot commented Oct 17, 2024

Uh oh!

doris-robot commented Oct 17, 2024

Uh oh!

doris-robot commented Oct 17, 2024

Uh oh!

doris-robot commented Oct 17, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

freemandealer commented Oct 16, 2024 •

edited

Loading