Skip to content

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 13, 2025

What problem does this PR solve?

The source should be closed after scanner is done to save memory.
Or there will be some objects remaining like:

  13:        192980        9263040  org.apache.kudu.client.RowwiseRowResult

Although these objects can be gc, eventually.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31649 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d1c9b28a504cf261ac842512c639a27f3db20d69, data reload: false

------ Round 1 ----------------------------------
q1	17580	5223	5083	5083
q2	2046	302	171	171
q3	10415	1231	755	755
q4	10207	1019	535	535
q5	7501	2370	2336	2336
q6	182	165	136	136
q7	901	745	615	615
q8	9310	1292	1056	1056
q9	4866	4697	4800	4697
q10	6841	2335	1906	1906
q11	463	286	261	261
q12	342	355	219	219
q13	19263	3741	3077	3077
q14	243	222	206	206
q15	510	456	470	456
q16	643	626	584	584
q17	563	869	334	334
q18	6743	6263	6238	6238
q19	1203	940	539	539
q20	305	321	196	196
q21	2826	2143	1949	1949
q22	356	320	300	300
Total cold run time: 103309 ms
Total hot run time: 31649 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5196	5156	5112	5112
q2	231	325	225	225
q3	2189	2687	2316	2316
q4	1436	1876	1410	1410
q5	4260	4201	4185	4185
q6	209	168	126	126
q7	1916	1854	1795	1795
q8	2672	2603	2583	2583
q9	7701	7616	7507	7507
q10	3074	3360	2868	2868
q11	596	523	513	513
q12	729	822	625	625
q13	3689	3986	3264	3264
q14	281	300	280	280
q15	523	468	463	463
q16	641	664	649	649
q17	1150	1551	1405	1405
q18	7535	7401	7417	7401
q19	784	749	813	749
q20	1986	2019	1865	1865
q21	5311	4884	4821	4821
q22	674	578	529	529
Total cold run time: 52783 ms
Total hot run time: 50691 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190375 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1c9b28a504cf261ac842512c639a27f3db20d69, data reload: false

query1	1315	965	904	904
query2	6231	1834	1884	1834
query3	10953	4531	4446	4446
query4	57028	25914	23528	23528
query5	5218	500	466	466
query6	358	191	172	172
query7	5190	503	295	295
query8	317	229	210	210
query9	6904	2580	2587	2580
query10	439	305	258	258
query11	15262	15100	14761	14761
query12	157	107	110	107
query13	1197	540	379	379
query14	10391	6652	6957	6652
query15	196	188	186	186
query16	6978	652	491	491
query17	1046	706	569	569
query18	1512	416	330	330
query19	205	183	167	167
query20	150	123	123	123
query21	204	122	107	107
query22	4549	4516	4352	4352
query23	33964	33380	33490	33380
query24	5556	2416	2394	2394
query25	450	479	405	405
query26	640	268	162	162
query27	1616	511	344	344
query28	2812	2475	2424	2424
query29	609	593	451	451
query30	219	189	156	156
query31	915	909	864	864
query32	76	61	61	61
query33	418	357	287	287
query34	744	873	506	506
query35	818	838	747	747
query36	973	1016	922	922
query37	119	96	71	71
query38	4363	4270	4241	4241
query39	1499	1443	1437	1437
query40	204	123	101	101
query41	54	52	48	48
query42	125	102	103	102
query43	511	529	476	476
query44	1322	831	819	819
query45	175	173	175	173
query46	896	1089	664	664
query47	1846	1882	1889	1882
query48	406	425	317	317
query49	702	524	414	414
query50	699	743	448	448
query51	4268	4365	4270	4270
query52	106	100	97	97
query53	237	267	189	189
query54	482	507	421	421
query55	84	80	80	80
query56	252	297	267	267
query57	1143	1184	1148	1148
query58	259	245	243	243
query59	2761	2988	2806	2806
query60	290	284	267	267
query61	142	140	153	140
query62	729	742	714	714
query63	245	206	197	197
query64	1510	1114	667	667
query65	3180	3092	3132	3092
query66	708	385	292	292
query67	15882	15668	15295	15295
query68	5415	775	508	508
query69	521	290	266	266
query70	1205	1082	1110	1082
query71	433	288	265	265
query72	6310	3597	3782	3597
query73	1067	739	349	349
query74	8941	9315	8756	8756
query75	3472	3220	2711	2711
query76	3755	1163	720	720
query77	549	362	275	275
query78	10097	9944	9302	9302
query79	2956	812	592	592
query80	758	524	441	441
query81	488	274	257	257
query82	941	130	106	106
query83	175	165	151	151
query84	293	100	73	73
query85	782	344	301	301
query86	433	307	302	302
query87	4489	4601	4449	4449
query88	3382	2203	2191	2191
query89	412	319	278	278
query90	1872	193	196	193
query91	130	132	110	110
query92	72	62	57	57
query93	2718	1000	579	579
query94	658	374	310	310
query95	342	270	266	266
query96	476	561	275	275
query97	2807	2860	2742	2742
query98	238	205	203	203
query99	1419	1398	1270	1270
Total cold run time: 298415 ms
Total hot run time: 190375 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1c9b28a504cf261ac842512c639a27f3db20d69, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.40	0.42
query6	1.19	0.68	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.61	0.52	0.51
query10	0.56	0.58	0.57
query11	0.15	0.10	0.10
query12	0.13	0.12	0.11
query13	0.61	0.61	0.60
query14	2.69	2.68	2.83
query15	0.92	0.84	0.85
query16	0.37	0.38	0.38
query17	1.01	1.00	1.01
query18	0.21	0.20	0.20
query19	1.94	1.86	1.97
query20	0.01	0.02	0.02
query21	15.39	0.89	0.54
query22	0.77	1.17	0.72
query23	14.85	1.36	0.64
query24	6.39	2.17	0.48
query25	0.50	0.32	0.08
query26	0.47	0.16	0.13
query27	0.06	0.05	0.04
query28	10.25	0.87	0.42
query29	12.56	3.88	3.22
query30	0.26	0.08	0.06
query31	2.83	0.59	0.38
query32	3.22	0.55	0.46
query33	2.96	2.98	3.04
query34	15.75	5.11	4.45
query35	4.51	4.47	4.52
query36	0.65	0.50	0.49
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.73 s
Total hot run time: 30.09 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 13, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 86e1378 into apache:master Feb 13, 2025
31 checks passed
@morningman morningman changed the title [fix](trino-connector) close scanner after finished [opt](trino-connector) close scanner after finished Feb 13, 2025
github-actions bot pushed a commit that referenced this pull request Feb 13, 2025
### What problem does this PR solve?

The source should be closed after scanner is done to save memory.
Or there will be some objects remaining like:

```
  13:        192980        9263040  org.apache.kudu.client.RowwiseRowResult
```

Although these objects can be gc, eventually.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
### What problem does this PR solve?

The source should be closed after scanner is done to save memory.
Or there will be some objects remaining like:

```
  13:        192980        9263040  org.apache.kudu.client.RowwiseRowResult
```

Although these objects can be gc, eventually.
dataroaring pushed a commit that referenced this pull request Feb 24, 2025
#47873)

Cherry-picked from #47857

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
### What problem does this PR solve?

The source should be closed after scanner is done to save memory.
Or there will be some objects remaining like:

```
  13:        192980        9263040  org.apache.kudu.client.RowwiseRowResult
```

Although these objects can be gc, eventually.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants