Skip to content

[fix](multi-catalog)fix max compute null parts table read #40046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Aug 30, 2024
Merged

[fix](multi-catalog)fix max compute null parts table read #40046

morningman merged 1 commit into from
Aug 30, 2024

Conversation

@wsjz
Copy link
Contributor

@wsjz wsjz commented Aug 28, 2024 •
edited
Loading

Proposed changes

fix max compute null parts table read
if no any data in a part, we should make mc jni scanner return 0 rows

@doris-robot
Copy link

doris-robot commented Aug 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wsjz
Copy link
Contributor Author

wsjz commented Aug 29, 2024

run buildall

@wsjz
Copy link
Contributor Author

wsjz commented Aug 29, 2024

run buildall

morningman
morningman approved these changes Aug 29, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 29, 2024

PR approved by anyone and no changes requested.

@morningman morningman added dev/2.1.x dev/3.0.x and removed approved Indicates a PR has been approved by one committer. reviewed labels Aug 29, 2024
@doris-robot
Copy link

doris-robot commented Aug 29, 2024

TPC-H: Total hot run time: 37895 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e21876112a1337d51defa48fef6f7720383a75a2, data reload: false

------ Round 1 ----------------------------------
q1	17597	4371	4311	4311
q2	2026	178	178	178
q3	12036	973	1187	973
q4	10508	786	749	749
q5	7760	2850	2791	2791
q6	224	135	135	135
q7	950	610	591	591
q8	9354	2071	2056	2056
q9	7255	6478	6522	6478
q10	6991	2231	2183	2183
q11	473	252	241	241
q12	430	234	229	229
q13	18097	3076	3018	3018
q14	284	233	240	233
q15	528	495	482	482
q16	571	502	486	486
q17	971	676	772	676
q18	7281	6806	6767	6767
q19	1413	1054	1025	1025
q20	666	317	331	317
q21	3857	2985	3002	2985
q22	1128	991	1070	991
Total cold run time: 110400 ms
Total hot run time: 37895 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4380	4276	4227	4227
q2	373	279	264	264
q3	2891	2677	2713	2677
q4	1936	1631	1627	1627
q5	5543	5692	5643	5643
q6	225	142	137	137
q7	2218	1839	1865	1839
q8	3290	3419	3401	3401
q9	8857	8842	8841	8841
q10	3559	3354	3395	3354
q11	608	515	508	508
q12	827	651	676	651
q13	15578	3168	3214	3168
q14	319	283	289	283
q15	532	509	478	478
q16	615	578	580	578
q17	1859	1560	1542	1542
q18	8025	7724	7845	7724
q19	1730	1578	1559	1559
q20	2143	1950	1911	1911
q21	5869	5508	5470	5470
q22	1218	1035	1032	1032
Total cold run time: 72595 ms
Total hot run time: 56914 ms

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

TPC-DS: Total hot run time: 193207 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e21876112a1337d51defa48fef6f7720383a75a2, data reload: false

query1	1256	892	860	860
query2	6281	2017	1948	1948
query3	10724	4137	3992	3992
query4	60324	27023	23083	23083
query5	5414	496	480	480
query6	406	164	171	164
query7	5896	318	297	297
query8	295	208	202	202
query9	8914	2493	2487	2487
query10	475	267	265	265
query11	17680	15064	15292	15064
query12	161	100	104	100
query13	1546	402	388	388
query14	11395	7671	7383	7383
query15	224	171	176	171
query16	7553	454	479	454
query17	1137	567	574	567
query18	1433	300	295	295
query19	287	146	146	146
query20	125	109	113	109
query21	206	105	110	105
query22	4556	4349	4470	4349
query23	34390	33617	33512	33512
query24	5934	2908	2859	2859
query25	551	409	408	408
query26	691	159	158	158
query27	1785	286	288	286
query28	3692	2134	2110	2110
query29	672	426	436	426
query30	237	152	161	152
query31	941	777	766	766
query32	84	57	58	57
query33	479	300	292	292
query34	872	488	516	488
query35	827	736	718	718
query36	1060	954	918	918
query37	152	93	84	84
query38	3948	3879	3875	3875
query39	1454	1388	1420	1388
query40	201	117	120	117
query41	48	47	45	45
query42	115	98	99	98
query43	512	468	455	455
query44	1119	761	752	752
query45	200	182	168	168
query46	1093	752	755	752
query47	1877	1820	1803	1803
query48	393	307	311	307
query49	785	433	452	433
query50	858	434	445	434
query51	7201	7127	7049	7049
query52	98	89	92	89
query53	252	185	183	183
query54	576	463	457	457
query55	87	79	80	79
query56	289	260	265	260
query57	1200	1092	1087	1087
query58	220	221	249	221
query59	3038	2884	2833	2833
query60	311	280	293	280
query61	125	119	120	119
query62	752	657	642	642
query63	222	185	183	183
query64	2979	760	735	735
query65	3197	3146	3132	3132
query66	689	345	356	345
query67	15611	15232	15159	15159
query68	4371	595	583	583
query69	416	287	277	277
query70	1171	1043	1118	1043
query71	355	285	274	274
query72	6742	4060	3952	3952
query73	747	332	335	332
query74	9160	8831	8786	8786
query75	3353	2695	2739	2695
query76	1477	1001	970	970
query77	572	316	320	316
query78	9718	9057	9973	9057
query79	2510	544	545	544
query80	901	499	496	496
query81	565	237	239	237
query82	385	141	142	141
query83	213	146	148	146
query84	264	70	73	70
query85	940	293	311	293
query86	413	299	296	296
query87	4354	4285	4213	4213
query88	3424	2374	2340	2340
query89	389	281	293	281
query90	1981	196	191	191
query91	123	95	99	95
query92	62	53	49	49
query93	2222	550	545	545
query94	845	306	294	294
query95	361	266	264	264
query96	606	268	267	267
query97	3165	3102	3089	3089
query98	225	209	201	201
query99	1788	1247	1285	1247
Total cold run time: 311923 ms
Total hot run time: 193207 ms

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

ClickBench: Total hot run time: 31.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e21876112a1337d51defa48fef6f7720383a75a2, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.49	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.53	0.49	0.48
query10	0.55	0.55	0.54
query11	0.15	0.12	0.11
query12	0.16	0.12	0.13
query13	0.61	0.59	0.59
query14	2.11	2.04	2.05
query15	0.85	0.81	0.82
query16	0.36	0.37	0.38
query17	1.04	1.03	1.03
query18	0.21	0.20	0.21
query19	1.82	1.77	1.76
query20	0.02	0.01	0.01
query21	15.40	0.68	0.65
query22	4.23	6.56	1.77
query23	18.23	1.46	1.30
query24	2.21	0.24	0.21
query25	0.15	0.09	0.09
query26	0.28	0.18	0.17
query27	0.08	0.09	0.08
query28	13.15	1.02	0.99
query29	12.62	3.32	3.31
query30	0.24	0.06	0.05
query31	2.88	0.39	0.38
query32	3.28	0.48	0.47
query33	2.98	2.96	3.03
query34	17.17	4.44	4.43
query35	4.48	4.46	4.48
query36	0.67	0.48	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.16
query39	0.04	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.05	0.05
Total cold run time: 110.97 s
Total hot run time: 31.95 s

morningman
morningman approved these changes Aug 30, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 30, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 30, 2024

PR approved by anyone and no changes requested.

@morningman morningman merged commit 212b82f into apache:master Aug 30, 2024
morningman pushed a commit to morningman/doris that referenced this pull request Aug 30, 2024
@wsjz @morningman
[fix](multi-catalog)fix max compute null parts table read (apache#40046)
882e43d 
## Proposed changes

fix max compute null parts table read
if no any data in a part, we should make mc jni scanner return 0 rows
@morningman morningman mentioned this pull request Aug 30, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 30, 2024
@morningman @wsjz
[fix](multi-catalog)fix max compute null parts table read (#40046) (#…
e4c48cf 
…40179)

bp #40046

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@wsjz @dataroaring
[fix](multi-catalog)fix max compute null parts table read (#40046)
e22dfd0 
## Proposed changes

fix max compute null parts table read
if no any data in a part, we should make mc jni scanner return 0 rows
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
@wsjz wsjz deleted the mc_null_part branch November 18, 2024 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@wsjz @doris-robot @morningman @wuwenchi @yiguolei @dataroaring