Skip to content

branch-3.1: [fix](outfile) fix analysis error when specifying parquet schema (#57232) #57500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Oct 31, 2025
Merged

branch-3.1: [fix](outfile) fix analysis error when specifying parquet schema (#57232) #57500

morningman merged 2 commits into from
Oct 31, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Oct 30, 2025

bp #57232

@morningman
[fix](outfile) fix analysis error when specifying parquet schema (apa…
0cfa249 
…che#57232)

Related PR: apache#33016

Introduced from apache#33016, when specify the "schema" property in outfile
clause with parquet format,
it will return error:
```
Parquet schema number does not equal to select item number
```

This is because we wrongly analyze `OutfileClause` twice.
@morningman morningman requested a review from morrySnow as a code owner October 30, 2025 07:10
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Oct 30, 2025

run buildall

@morningman
Copy link
Contributor Author

morningman commented Oct 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

TPC-H: Total hot run time: 33130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a936a0241c730909574f587eee499cd36ff86f5, data reload: false

------ Round 1 ----------------------------------
q1	17633	5666	5537	5537
q2	2043	456	287	287
q3	12314	1258	767	767
q4	10289	877	463	463
q5	8952	2435	2153	2153
q6	191	165	136	136
q7	904	756	630	630
q8	9331	1475	1180	1180
q9	5242	4998	4928	4928
q10	6803	2274	1860	1860
q11	473	289	265	265
q12	337	359	217	217
q13	17764	3620	3043	3043
q14	227	236	218	218
q15	527	474	464	464
q16	422	437	379	379
q17	599	876	372	372
q18	7094	6412	6467	6412
q19	1211	960	578	578
q20	332	340	214	214
q21	3089	2228	2035	2035
q22	1045	1056	992	992
Total cold run time: 106822 ms
Total hot run time: 33130 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5550	5534	5534	5534
q2	240	337	236	236
q3	2245	2687	2309	2309
q4	1359	1821	1371	1371
q5	4491	5128	5055	5055
q6	174	167	134	134
q7	2146	1980	1839	1839
q8	2641	2900	2701	2701
q9	7353	7299	7222	7222
q10	3056	3296	2774	2774
q11	598	529	512	512
q12	661	758	597	597
q13	3475	3760	3203	3203
q14	279	297	268	268
q15	508	483	490	483
q16	436	493	456	456
q17	1284	1749	1292	1292
q18	7720	7522	7419	7419
q19	843	1062	1156	1062
q20	2052	2049	1921	1921
q21	5487	5051	4589	4589
q22	1088	1076	1098	1076
Total cold run time: 53686 ms
Total hot run time: 52053 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

TPC-DS: Total hot run time: 191698 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a936a0241c730909574f587eee499cd36ff86f5, data reload: false

query1	938	409	387	387
query2	6184	1950	1900	1900
query3	8698	205	199	199
query4	33663	24448	23474	23474
query5	4780	611	470	470
query6	304	196	174	174
query7	4203	499	333	333
query8	313	238	235	235
query9	9550	2626	2622	2622
query10	490	328	274	274
query11	18214	15574	15194	15194
query12	163	110	109	109
query13	1558	592	428	428
query14	9343	7650	6834	6834
query15	234	194	186	186
query16	7980	646	522	522
query17	1485	798	613	613
query18	2129	453	357	357
query19	253	195	173	173
query20	137	133	121	121
query21	221	141	118	118
query22	4727	4676	4408	4408
query23	35630	34339	34037	34037
query24	7391	2755	2742	2742
query25	563	516	451	451
query26	1161	304	188	188
query27	2226	490	369	369
query28	5414	2344	2291	2291
query29	764	624	504	504
query30	249	192	188	188
query31	1047	920	863	863
query32	87	63	60	60
query33	517	386	323	323
query34	778	887	549	549
query35	815	845	768	768
query36	1038	1055	955	955
query37	107	97	72	72
query38	4101	4030	4007	4007
query39	1554	1483	1499	1483
query40	210	124	111	111
query41	50	51	50	50
query42	135	114	109	109
query43	528	522	507	507
query44	1419	868	851	851
query45	193	186	193	186
query46	918	1085	701	701
query47	1989	2000	1903	1903
query48	418	454	370	370
query49	787	508	448	448
query50	704	724	459	459
query51	7256	7414	7133	7133
query52	109	105	98	98
query53	243	274	192	192
query54	591	602	513	513
query55	90	82	81	81
query56	287	285	260	260
query57	1272	1281	1234	1234
query58	240	229	256	229
query59	3079	3366	3004	3004
query60	317	293	287	287
query61	151	117	121	117
query62	815	751	696	696
query63	241	206	205	205
query64	4355	1008	679	679
query65	3416	3315	3318	3315
query66	1122	423	327	327
query67	16315	15935	15553	15553
query68	7282	849	555	555
query69	484	319	271	271
query70	1187	1075	1150	1075
query71	384	309	281	281
query72	5811	3825	2590	2590
query73	668	750	350	350
query74	10349	9417	8985	8985
query75	3249	3153	2680	2680
query76	3166	1153	791	791
query77	753	390	278	278
query78	10430	10423	9625	9625
query79	3285	853	592	592
query80	644	543	448	448
query81	514	256	220	220
query82	571	121	89	89
query83	174	172	148	148
query84	245	112	81	81
query85	788	414	310	310
query86	356	321	302	302
query87	4289	4362	4231	4231
query88	4806	2455	2439	2439
query89	404	335	293	293
query90	1865	196	192	192
query91	143	141	112	112
query92	64	60	52	52
query93	1585	946	555	555
query94	686	419	313	313
query95	351	294	283	283
query96	506	610	297	297
query97	3151	3231	3149	3149
query98	238	214	203	203
query99	1569	1423	1309	1309
Total cold run time: 296256 ms
Total hot run time: 191698 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2025

ClickBench: Total hot run time: 29.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a936a0241c730909574f587eee499cd36ff86f5, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.52	0.51	0.51
query6	1.15	0.75	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.52
query10	0.56	0.55	0.55
query11	0.15	0.11	0.11
query12	0.13	0.11	0.12
query13	0.61	0.59	0.60
query14	0.77	0.79	0.80
query15	0.86	0.83	0.82
query16	0.42	0.39	0.39
query17	1.08	1.09	1.07
query18	0.23	0.22	0.22
query19	1.90	1.79	1.82
query20	0.02	0.01	0.01
query21	15.38	0.90	0.59
query22	0.74	0.72	0.65
query23	15.21	1.42	0.61
query24	3.03	0.99	1.86
query25	0.20	0.06	0.06
query26	0.29	0.14	0.14
query27	0.06	0.05	0.04
query28	14.09	0.99	0.45
query29	12.58	3.84	3.25
query30	0.26	0.09	0.06
query31	2.84	0.61	0.39
query32	3.23	0.55	0.46
query33	2.97	3.07	3.10
query34	16.63	5.16	4.61
query35	4.62	4.60	4.57
query36	0.64	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.03
query40	0.16	0.13	0.13
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.28 s
Total hot run time: 29.2 s

@morningman morningman merged commit 61ddccb into apache:branch-3.1 Oct 31, 2025
22 checks passed
@morrySnow morrySnow mentioned this pull request Nov 13, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @hello-stephen @doris-robot @morrySnow