Skip to content

[fix](Parquet) add a memory tracker to parquet meta #49037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Apr 3, 2025
Merged

[fix](Parquet) add a memory tracker to parquet meta #49037

morningman merged 1 commit into from
Apr 3, 2025

Conversation

@BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Mar 13, 2025 •
edited
Loading

Problem Summary:

Add a memory tracker to the Parquet metadata, allowing us to monitor the usage of Parquet metadata.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@BePPPower BePPPower changed the title [fix](Parquet) add a memory tracker for parquet meta [fix](Parquet) add a memory tracker to parquet meta Mar 13, 2025
@BePPPower
Copy link
Contributor Author

BePPPower commented Mar 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-H: Total hot run time: 32859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b642246b5586a0cc8e4118752899965f6de86567, data reload: false

------ Round 1 ----------------------------------
q1	24301	5093	5085	5085
q2	2063	334	200	200
q3	10319	1272	693	693
q4	10226	1042	562	562
q5	7577	2443	2338	2338
q6	187	162	132	132
q7	941	753	597	597
q8	9295	1318	1147	1147
q9	4932	4855	4795	4795
q10	6836	2287	1898	1898
q11	473	278	257	257
q12	348	362	219	219
q13	17775	3668	3085	3085
q14	226	228	212	212
q15	537	480	490	480
q16	629	610	581	581
q17	588	851	354	354
q18	7312	6500	6505	6500
q19	1556	971	603	603
q20	329	324	203	203
q21	2866	2215	1925	1925
q22	1098	993	996	993
Total cold run time: 110414 ms
Total hot run time: 32859 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5216	5173	5135	5135
q2	248	333	228	228
q3	2200	2644	2312	2312
q4	1492	1855	1431	1431
q5	4222	4173	4423	4173
q6	224	169	129	129
q7	2014	1932	1777	1777
q8	2620	2686	2558	2558
q9	7240	7261	7209	7209
q10	3002	3115	2805	2805
q11	590	511	506	506
q12	677	832	653	653
q13	3529	3882	3234	3234
q14	286	315	268	268
q15	549	493	477	477
q16	661	700	652	652
q17	1194	1650	1358	1358
q18	7690	7623	7506	7506
q19	898	840	855	840
q20	1991	2059	1876	1876
q21	5540	4875	4956	4875
q22	1190	1102	1025	1025
Total cold run time: 53273 ms
Total hot run time: 51027 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-DS: Total hot run time: 192468 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b642246b5586a0cc8e4118752899965f6de86567, data reload: false

query1	1450	1118	1076	1076
query2	6107	1998	1941	1941
query3	11144	4604	4781	4604
query4	25860	23820	22911	22911
query5	4998	646	469	469
query6	287	193	184	184
query7	3991	510	290	290
query8	281	273	221	221
query9	8484	2636	2642	2636
query10	481	317	242	242
query11	15782	15065	15096	15065
query12	169	113	102	102
query13	1560	527	382	382
query14	9594	6914	6358	6358
query15	202	186	167	167
query16	7606	660	508	508
query17	1117	779	611	611
query18	2035	408	328	328
query19	201	188	167	167
query20	131	126	121	121
query21	204	121	105	105
query22	4368	4804	4358	4358
query23	34689	33736	33503	33503
query24	7677	2419	2446	2419
query25	505	511	428	428
query26	1172	292	158	158
query27	2091	498	367	367
query28	4573	2501	2466	2466
query29	704	570	434	434
query30	276	226	199	199
query31	937	857	792	792
query32	75	61	62	61
query33	547	375	350	350
query34	814	881	514	514
query35	767	839	759	759
query36	978	987	899	899
query37	124	97	75	75
query38	4421	4246	4258	4246
query39	1486	1417	1440	1417
query40	211	117	100	100
query41	52	55	49	49
query42	117	107	106	106
query43	508	517	482	482
query44	1308	800	799	799
query45	181	174	167	167
query46	848	1026	657	657
query47	1883	1904	1841	1841
query48	405	443	316	316
query49	794	566	446	446
query50	701	756	427	427
query51	4350	4365	4283	4283
query52	104	107	96	96
query53	243	260	189	189
query54	505	506	420	420
query55	86	81	85	81
query56	272	274	254	254
query57	1174	1201	1105	1105
query58	244	248	233	233
query59	2824	2859	2882	2859
query60	285	284	269	269
query61	123	121	118	118
query62	787	751	698	698
query63	229	187	226	187
query64	4117	1042	683	683
query65	4663	4426	4573	4426
query66	1072	395	308	308
query67	15928	15397	15329	15329
query68	8226	872	519	519
query69	475	302	258	258
query70	1195	1137	1103	1103
query71	407	286	268	268
query72	5742	3560	3780	3560
query73	727	745	362	362
query74	9109	9096	8935	8935
query75	3194	3183	2665	2665
query76	3151	1187	736	736
query77	457	372	286	286
query78	9942	10127	9421	9421
query79	2204	836	591	591
query80	609	522	460	460
query81	474	254	224	224
query82	435	125	94	94
query83	178	171	153	153
query84	246	97	74	74
query85	759	355	306	306
query86	360	354	291	291
query87	4385	4496	4292	4292
query88	3760	2236	2248	2236
query89	401	321	277	277
query90	1949	207	206	206
query91	140	140	107	107
query92	78	60	55	55
query93	1876	1054	571	571
query94	673	408	307	307
query95	355	268	253	253
query96	478	567	273	273
query97	3344	3383	3281	3281
query98	234	218	196	196
query99	1354	1382	1247	1247
Total cold run time: 278819 ms
Total hot run time: 192468 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b642246b5586a0cc8e4118752899965f6de86567, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.19
query4	1.58	0.20	0.20
query5	0.61	0.57	0.58
query6	1.18	0.72	0.72
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.53	0.52
query10	0.57	0.60	0.56
query11	0.15	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.65	2.81	2.67
query15	0.92	0.85	0.88
query16	0.37	0.39	0.36
query17	0.99	1.02	1.07
query18	0.21	0.20	0.19
query19	1.90	1.96	1.83
query20	0.01	0.01	0.01
query21	15.38	0.91	0.57
query22	0.74	1.31	0.68
query23	14.86	1.33	0.65
query24	7.26	1.47	0.42
query25	0.51	0.28	0.07
query26	0.61	0.16	0.12
query27	0.05	0.05	0.05
query28	9.80	0.83	0.42
query29	12.55	3.92	3.30
query30	0.24	0.08	0.06
query31	2.84	0.58	0.39
query32	3.22	0.54	0.46
query33	2.98	3.00	3.05
query34	15.90	5.25	4.56
query35	4.56	4.59	4.63
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.02	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.59 s
Total hot run time: 30.8 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 17, 2025

BE UT Coverage Report

Increment line coverage 90.91% (10/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.46% (12980/26784)
Line Coverage 37.90% (111227/293450)
Region Coverage 36.87% (56782/154013)
Branch Coverage 32.03% (28559/89156)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 24, 2025

PR approved by anyone and no changes requested.

xinyiZzz
xinyiZzz approved these changes Mar 24, 2025
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 6f2a46f into apache:master Apr 3, 2025
30 of 31 checks passed
BePPPower added a commit to BePPPower/doris that referenced this pull request Apr 17, 2025
@BePPPower
[fix](Parquet) add a memory tracker to parquet meta (apache#49037)
997413e 
Problem Summary:

Add a memory tracker to the Parquet metadata, allowing us to monitor the
usage of Parquet metadata.
@BePPPower BePPPower mentioned this pull request Apr 17, 2025
Merged
16 tasks
BePPPower added a commit to BePPPower/doris that referenced this pull request Apr 27, 2025
@BePPPower
[fix](Parquet) add a memory tracker to parquet meta (apache#49037)
86028cf 
Problem Summary:

Add a memory tracker to the Parquet metadata, allowing us to monitor the
usage of Parquet metadata.
@BePPPower BePPPower mentioned this pull request Apr 27, 2025
Merged
dataroaring pushed a commit that referenced this pull request May 6, 2025
@BePPPower
[Branch3.0](Parquet) add a memory tracker to parquet meta (#50468)
2581501 
picked from: #49037
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@BePPPower @koarz
[fix](Parquet) add a memory tracker to parquet meta (apache#49037)
2fceb95 
Problem Summary:

Add a memory tracker to the Parquet metadata, allowing us to monitor the
usage of Parquet metadata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@xinyiZzz xinyiZzz xinyiZzz approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@BePPPower @Thearas @doris-robot @hello-stephen @morningman @xinyiZzz @yiguolei @dataroaring