Skip to content

[fix](log) Support fe log rollover size strategy #34446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
May 9, 2024
Merged

[fix](log) Support fe log rollover size strategy #34446

dataroaring merged 2 commits into from
May 9, 2024

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented May 6, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

Support fe log rollover size strategy, default use age strategy. cloud use size stragegy

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@deardeng
Copy link
Contributor Author

deardeng commented May 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 6, 2024

TPC-DS: Total hot run time: 187281 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4bd7405133285487c32cc89b235b00046a5aa600, data reload: false

query1	915	362	342	342
query2	6278	2469	2331	2331
query3	6770	208	217	208
query4	24985	21794	22014	21794
query5	3790	432	433	432
query6	278	192	170	170
query7	4622	289	289	289
query8	238	182	186	182
query9	8589	2423	2464	2423
query10	411	278	254	254
query11	15134	14799	14763	14763
query12	118	84	88	84
query13	1606	369	370	369
query14	9447	7484	7395	7395
query15	231	161	171	161
query16	8104	249	246	246
query17	1733	561	533	533
query18	2081	275	279	275
query19	225	146	150	146
query20	88	81	81	81
query21	194	119	119	119
query22	5046	4829	4831	4829
query23	33915	33345	33382	33345
query24	10687	2916	2826	2826
query25	583	364	357	357
query26	1168	150	148	148
query27	2948	314	326	314
query28	7564	2041	2040	2040
query29	859	624	604	604
query30	254	151	146	146
query31	944	771	716	716
query32	86	50	50	50
query33	736	239	243	239
query34	1072	468	471	468
query35	839	677	664	664
query36	1051	898	886	886
query37	136	66	69	66
query38	3141	3025	3003	3003
query39	1598	1522	1530	1522
query40	208	125	121	121
query41	39	36	34	34
query42	102	97	97	97
query43	607	567	542	542
query44	1267	729	744	729
query45	269	259	247	247
query46	1095	712	699	699
query47	1918	1872	1860	1860
query48	358	294	290	290
query49	893	386	385	385
query50	763	382	382	382
query51	6824	6679	6571	6571
query52	101	92	91	91
query53	352	283	278	278
query54	299	233	231	231
query55	80	73	74	73
query56	239	225	215	215
query57	1214	1122	1148	1122
query58	222	201	203	201
query59	3486	3396	3232	3232
query60	258	244	241	241
query61	108	118	131	118
query62	655	452	455	452
query63	306	289	275	275
query64	8730	7351	7383	7351
query65	3135	3082	3031	3031
query66	1381	349	329	329
query67	15759	15057	14835	14835
query68	9189	537	536	536
query69	555	311	313	311
query70	1220	1149	1127	1127
query71	513	268	274	268
query72	8234	2565	2357	2357
query73	1121	315	317	315
query74	6470	6050	6104	6050
query75	4383	2663	2731	2663
query76	5451	1010	970	970
query77	704	268	265	265
query78	11007	10160	10275	10160
query79	9897	521	522	521
query80	1274	437	449	437
query81	492	222	217	217
query82	771	93	90	90
query83	196	161	163	161
query84	255	87	83	83
query85	1325	264	263	263
query86	405	277	331	277
query87	3310	3107	3077	3077
query88	4511	2315	2312	2312
query89	528	383	385	383
query90	2186	190	189	189
query91	130	99	97	97
query92	59	46	46	46
query93	7060	501	507	501
query94	1134	181	186	181
query95	393	307	301	301
query96	612	264	262	262
query97	3151	2933	2993	2933
query98	239	218	222	218
query99	1309	858	879	858
Total cold run time: 307216 ms
Total hot run time: 187281 ms

morningman
morningman reviewed May 6, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And please use English to describe your PR

@deardeng deardeng changed the title [fix](log) Compatible logging strategy for cloud and non-cloud [fix](log) Support fe log rollover size strategy May 7, 2024
@deardeng
Copy link
Contributor Author

deardeng commented May 7, 2024

run buildall

@deardeng deardeng closed this May 7, 2024
@deardeng deardeng reopened this May 7, 2024
@deardeng
Copy link
Contributor Author

deardeng commented May 7, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 7, 2024

TPC-DS: Total hot run time: 188172 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 835b09247cf33b22bb96b17878de835aa7cdf410, data reload: false

query1	906	365	345	345
query2	6304	2395	2327	2327
query3	6641	204	201	201
query4	22853	21711	22213	21711
query5	3688	427	441	427
query6	265	197	176	176
query7	4528	310	291	291
query8	238	191	198	191
query9	8638	2489	2503	2489
query10	442	261	249	249
query11	15124	14673	14953	14673
query12	122	89	90	89
query13	1653	384	377	377
query14	8547	8507	8355	8355
query15	246	164	174	164
query16	8205	280	254	254
query17	1768	554	551	551
query18	2081	294	282	282
query19	313	140	144	140
query20	89	82	79	79
query21	188	127	129	127
query22	5045	4817	4859	4817
query23	34005	33307	33275	33275
query24	10674	2863	2873	2863
query25	606	360	359	359
query26	1111	153	151	151
query27	2413	331	316	316
query28	7290	2100	2071	2071
query29	858	609	606	606
query30	244	151	148	148
query31	935	703	725	703
query32	97	48	53	48
query33	734	247	248	247
query34	1025	487	477	477
query35	820	683	670	670
query36	1031	913	942	913
query37	128	62	65	62
query38	3121	3003	2991	2991
query39	1603	1659	1533	1533
query40	193	126	128	126
query41	40	38	37	37
query42	103	92	96	92
query43	574	545	532	532
query44	1247	720	732	720
query45	253	250	253	250
query46	1068	768	737	737
query47	1973	1874	1843	1843
query48	375	303	297	297
query49	853	393	380	380
query50	752	387	395	387
query51	6812	6741	6741	6741
query52	101	92	91	91
query53	353	282	276	276
query54	306	234	227	227
query55	78	70	71	70
query56	255	217	214	214
query57	1217	1158	1130	1130
query58	217	206	213	206
query59	3424	3331	3176	3176
query60	252	228	239	228
query61	87	103	85	85
query62	671	447	450	447
query63	305	284	277	277
query64	8605	7232	7176	7176
query65	3045	3059	3021	3021
query66	909	343	330	330
query67	15739	15034	14939	14939
query68	5224	534	538	534
query69	487	346	296	296
query70	1116	1091	1139	1091
query71	408	263	263	263
query72	7159	2568	2348	2348
query73	704	325	328	325
query74	6533	6028	6135	6028
query75	3438	2639	2651	2639
query76	3205	1087	969	969
query77	365	264	261	261
query78	11079	10274	10086	10086
query79	5048	527	553	527
query80	2075	448	443	443
query81	566	228	227	227
query82	840	90	96	90
query83	308	169	165	165
query84	265	90	90	90
query85	2045	266	268	266
query86	469	317	306	306
query87	3292	3069	3027	3027
query88	5142	2427	2433	2427
query89	479	364	374	364
query90	2010	190	188	188
query91	121	97	98	97
query92	68	47	49	47
query93	5349	513	520	513
query94	1211	181	181	181
query95	393	301	307	301
query96	606	286	266	266
query97	3119	2946	2958	2946
query98	239	211	214	211
query99	1243	867	899	867
Total cold run time: 288992 ms
Total hot run time: 188172 ms

morningman
morningman approved these changes May 7, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 7, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes May 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

dataroaring commented May 8, 2024

run buildall

dataroaring
dataroaring approved these changes May 9, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 75a0de1 into apache:master May 9, 2024
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
@deardeng @ByteYue
[fix](log) Support fe log rollover size strategy (apache#34446)
8d273cb
morningman pushed a commit to morningman/doris that referenced this pull request Jun 4, 2024
@deardeng @morningman
[fix](log) Support fe log rollover size strategy (apache#34446)
f94222a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged kind/behavior-changed reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@deardeng @doris-robot @dataroaring @morningman @gavinchou