Skip to content

Conversation

@hubgeter
Copy link
Contributor

What problem does this PR solve?

Related PR: #52114

Problem Summary:
PR #52114 Ignoring lazy materialization of external table, a line from a file may be needed multiple times in a batch.
This PR fixes this issue.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 35086 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 868b8944e67bf732e5da96ec45e3fe6291ebc458, data reload: false

------ Round 1 ----------------------------------
q1	17623	5249	5169	5169
q2	1984	320	240	240
q3	10261	1285	734	734
q4	10238	1025	537	537
q5	7507	2369	2363	2363
q6	182	170	136	136
q7	937	779	636	636
q8	9383	1360	1135	1135
q9	7028	5116	5159	5116
q10	6904	2401	1984	1984
q11	514	302	277	277
q12	353	369	236	236
q13	17786	3661	3118	3118
q14	244	237	220	220
q15	556	492	482	482
q16	1029	1009	956	956
q17	605	860	360	360
q18	8003	7224	7166	7166
q19	1238	960	567	567
q20	349	341	229	229
q21	3656	3225	2438	2438
q22	1055	1023	987	987
Total cold run time: 107435 ms
Total hot run time: 35086 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5105	5040	5051	5040
q2	251	342	251	251
q3	2203	2665	2249	2249
q4	1337	1731	1317	1317
q5	4259	4605	4632	4605
q6	215	174	135	135
q7	2073	2058	1805	1805
q8	2766	2575	2614	2575
q9	7370	7384	7213	7213
q10	3179	3315	2962	2962
q11	586	518	503	503
q12	702	785	675	675
q13	3565	3914	3438	3438
q14	324	299	290	290
q15	529	503	500	500
q16	1106	1137	1060	1060
q17	1189	1579	1417	1417
q18	7926	7648	7871	7648
q19	806	828	870	828
q20	1999	2082	1904	1904
q21	5134	4477	4271	4271
q22	1059	1046	984	984
Total cold run time: 53683 ms
Total hot run time: 51670 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189387 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 868b8944e67bf732e5da96ec45e3fe6291ebc458, data reload: false

query1	1071	451	417	417
query2	6542	1662	1703	1662
query3	6763	229	232	229
query4	26622	23152	24012	23152
query5	5210	672	517	517
query6	360	258	255	255
query7	4680	527	308	308
query8	317	267	271	267
query9	8648	2972	2956	2956
query10	554	383	319	319
query11	15720	14941	14871	14871
query12	189	139	122	122
query13	1725	608	455	455
query14	10820	9223	9235	9223
query15	211	205	184	184
query16	7722	697	504	504
query17	1419	767	663	663
query18	2061	482	334	334
query19	201	193	184	184
query20	143	137	125	125
query21	214	135	115	115
query22	4253	4085	4129	4085
query23	34131	33080	33051	33051
query24	8200	2370	2380	2370
query25	588	518	456	456
query26	1248	281	172	172
query27	2688	505	367	367
query28	4360	2296	2274	2274
query29	790	611	504	504
query30	291	230	198	198
query31	889	810	729	729
query32	96	82	76	76
query33	569	396	348	348
query34	796	857	545	545
query35	809	822	739	739
query36	999	1024	944	944
query37	135	114	97	97
query38	3492	3473	3481	3473
query39	1492	1436	1507	1436
query40	224	145	132	132
query41	63	63	67	63
query42	139	119	128	119
query43	535	538	456	456
query44	1368	869	871	869
query45	183	176	172	172
query46	863	1014	653	653
query47	1770	1826	1752	1752
query48	397	451	321	321
query49	766	528	414	414
query50	653	681	405	405
query51	4109	3949	3915	3915
query52	129	114	108	108
query53	268	277	197	197
query54	609	613	558	558
query55	101	89	94	89
query56	349	337	316	316
query57	1186	1208	1130	1130
query58	290	280	279	279
query59	2579	2734	2537	2537
query60	368	363	366	363
query61	170	160	167	160
query62	799	736	681	681
query63	232	198	202	198
query64	4425	1203	852	852
query65	4060	4052	3989	3989
query66	1122	453	359	359
query67	15503	15260	15021	15021
query68	9457	922	597	597
query69	508	328	297	297
query70	1326	1318	1274	1274
query71	562	355	328	328
query72	5848	4991	5072	4991
query73	740	616	362	362
query74	9272	8828	9065	8828
query75	4397	3285	2786	2786
query76	4044	1144	728	728
query77	976	407	335	335
query78	9550	9801	8889	8889
query79	1967	804	600	600
query80	762	598	531	531
query81	471	256	227	227
query82	264	167	140	140
query83	288	259	256	256
query84	307	106	90	90
query85	838	465	430	430
query86	355	310	404	310
query87	3819	3754	3616	3616
query88	2812	2204	2196	2196
query89	406	329	297	297
query90	1997	224	223	223
query91	166	163	130	130
query92	90	76	75	75
query93	1122	1007	657	657
query94	698	421	326	326
query95	405	338	332	332
query96	477	590	280	280
query97	2932	3003	2860	2860
query98	247	223	223	223
query99	1433	1416	1339	1339
Total cold run time: 278814 ms
Total hot run time: 189387 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 868b8944e67bf732e5da96ec45e3fe6291ebc458, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.60	0.12	0.11
query5	0.27	0.26	0.24
query6	1.16	0.65	0.65
query7	0.03	0.03	0.02
query8	0.06	0.05	0.05
query9	0.61	0.52	0.52
query10	0.58	0.58	0.59
query11	0.17	0.12	0.11
query12	0.16	0.12	0.13
query13	0.67	0.64	0.62
query14	1.03	1.03	1.06
query15	0.86	0.86	0.89
query16	0.40	0.39	0.39
query17	1.06	1.04	1.05
query18	0.21	0.20	0.20
query19	1.95	1.81	1.82
query20	0.01	0.02	0.01
query21	15.43	1.02	0.58
query22	0.80	1.14	0.72
query23	14.91	1.40	0.63
query24	6.80	1.64	0.43
query25	0.41	0.13	0.06
query26	0.58	0.17	0.13
query27	0.05	0.05	0.06
query28	9.04	0.92	0.43
query29	12.59	3.89	3.27
query30	0.29	0.14	0.12
query31	2.83	0.62	0.38
query32	3.25	0.56	0.47
query33	3.13	3.10	3.14
query34	15.86	5.51	4.85
query35	4.87	4.96	5.01
query36	0.69	0.53	0.50
query37	0.11	0.07	0.08
query38	0.07	0.05	0.05
query39	0.03	0.03	0.03
query40	0.19	0.16	0.14
query41	0.10	0.03	0.02
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 103.36 s
Total hot run time: 29.6 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.97% (17302/33295)
Line Coverage 37.30% (157636/422596)
Region Coverage 31.93% (120181/376413)
Branch Coverage 33.32% (52817/158503)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.73% (23125/32694)
Line Coverage 57.11% (241194/422342)
Region Coverage 52.46% (200333/381842)
Branch Coverage 54.12% (86272/159406)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit a85d86c into apache:master Sep 12, 2025
31 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants