Skip to content

[fix](statistics) NPE when drop partition during publish #35475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 28, 2024
Merged

[fix](statistics) NPE when drop partition during publish #35475

dataroaring merged 1 commit into from
May 28, 2024

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented May 27, 2024 •
edited by morningman
Loading

skip null partition when get base tablets for each be (for further usage in dedup updated row count in MV)
This may cause publish fail

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@freemandealer
[fix](statistics) NPE when drop partition during publish
86797b3 
skip null partition when get base tablets for each be (for further usage in dedup updated row count in MV)
@doris-robot
Copy link

doris-robot commented May 27, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@freemandealer
Copy link
Contributor Author

freemandealer commented May 27, 2024

run buildall

morningman
morningman approved these changes May 27, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented May 27, 2024

TPC-H: Total hot run time: 39751 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86797b3fcbf7c093225d8e314a2cb67174fce621, data reload: false

------ Round 1 ----------------------------------
q1	18091	4463	4300	4300
q2	2540	193	197	193
q3	11680	1149	1160	1149
q4	10706	748	795	748
q5	7905	2686	2690	2686
q6	235	130	135	130
q7	957	602	588	588
q8	9536	2051	2040	2040
q9	8819	6506	6460	6460
q10	8989	3702	3711	3702
q11	451	241	239	239
q12	386	218	222	218
q13	17750	2949	3015	2949
q14	260	216	225	216
q15	502	470	480	470
q16	530	387	378	378
q17	958	641	702	641
q18	7979	7389	7365	7365
q19	7820	1536	1494	1494
q20	658	305	308	305
q21	4936	3870	3206	3206
q22	332	274	276	274
Total cold run time: 122020 ms
Total hot run time: 39751 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4177	4185	4177
q2	364	263	264	263
q3	2987	2798	2763	2763
q4	1844	1534	1620	1534
q5	5271	5240	5267	5240
q6	214	124	130	124
q7	2110	1754	1768	1754
q8	3178	3310	3283	3283
q9	8390	8282	8309	8282
q10	3858	3711	3725	3711
q11	583	484	484	484
q12	751	614	606	606
q13	17252	2977	3008	2977
q14	299	267	260	260
q15	520	472	476	472
q16	479	430	408	408
q17	1744	1481	1465	1465
q18	7627	7538	7569	7538
q19	1635	1583	1540	1540
q20	1961	1787	1764	1764
q21	4793	4715	4695	4695
q22	542	492	475	475
Total cold run time: 70709 ms
Total hot run time: 53815 ms

@doris-robot
Copy link

doris-robot commented May 27, 2024

TPC-DS: Total hot run time: 168592 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86797b3fcbf7c093225d8e314a2cb67174fce621, data reload: false

query1	924	371	365	365
query2	6574	2504	2308	2308
query3	6649	202	201	201
query4	19845	17316	17226	17226
query5	4233	417	428	417
query6	256	156	148	148
query7	4582	298	294	294
query8	236	181	182	181
query9	8465	2341	2351	2341
query10	441	279	256	256
query11	10313	10049	10106	10049
query12	132	91	91	91
query13	1638	371	366	366
query14	9983	7457	6178	6178
query15	207	170	168	168
query16	7728	267	251	251
query17	1862	533	503	503
query18	1784	271	267	267
query19	202	150	157	150
query20	94	84	86	84
query21	204	138	135	135
query22	4154	4035	3953	3953
query23	33608	33062	33418	33062
query24	11993	2904	2894	2894
query25	700	361	368	361
query26	1751	159	157	157
query27	3059	330	314	314
query28	7566	2021	2018	2018
query29	1058	610	587	587
query30	296	149	151	149
query31	1012	742	771	742
query32	91	57	57	57
query33	765	266	261	261
query34	1033	495	478	478
query35	730	613	599	599
query36	1079	922	909	909
query37	284	68	68	68
query38	2892	2795	2781	2781
query39	878	796	781	781
query40	271	121	126	121
query41	46	44	44	44
query42	99	96	93	93
query43	579	554	562	554
query44	1228	731	739	731
query45	175	175	165	165
query46	1068	700	694	694
query47	1871	1765	1797	1765
query48	381	297	299	297
query49	1194	377	398	377
query50	760	391	384	384
query51	6941	6795	6818	6795
query52	108	89	85	85
query53	354	281	281	281
query54	947	431	420	420
query55	74	74	74	74
query56	255	235	245	235
query57	1139	1063	1047	1047
query58	232	210	208	208
query59	3505	3458	3377	3377
query60	283	254	259	254
query61	117	90	90	90
query62	651	455	441	441
query63	314	288	284	284
query64	9777	2200	1732	1732
query65	3182	3107	3119	3107
query66	1331	344	320	320
query67	15140	14791	15142	14791
query68	4590	525	521	521
query69	430	266	270	266
query70	1175	1116	1117	1116
query71	407	263	272	263
query72	7486	2703	2523	2523
query73	707	321	320	320
query74	5978	5679	5664	5664
query75	3414	2618	2603	2603
query76	2792	880	1046	880
query77	425	265	267	265
query78	10284	10203	9698	9698
query79	2362	518	501	501
query80	1005	430	488	430
query81	507	221	213	213
query82	693	90	92	90
query83	231	169	184	169
query84	239	81	88	81
query85	1604	278	257	257
query86	502	317	292	292
query87	3286	3094	3099	3094
query88	4109	2436	2406	2406
query89	469	374	376	374
query90	2004	189	182	182
query91	124	96	94	94
query92	63	50	51	50
query93	1478	498	477	477
query94	1258	187	184	184
query95	400	302	304	302
query96	588	277	270	270
query97	3195	3024	3004	3004
query98	238	219	218	218
query99	1122	871	869	869
Total cold run time: 274488 ms
Total hot run time: 168592 ms

@doris-robot
Copy link

doris-robot commented May 27, 2024

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86797b3fcbf7c093225d8e314a2cb67174fce621, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.49	0.49	0.50
query6	1.12	0.74	0.72
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.52	0.48	0.49
query10	0.52	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.61
query14	0.75	0.78	0.78
query15	0.83	0.82	0.80
query16	0.36	0.37	0.37
query17	1.02	1.00	0.94
query18	0.23	0.25	0.22
query19	1.82	1.79	1.69
query20	0.02	0.01	0.01
query21	15.48	0.68	0.67
query22	4.23	7.13	2.20
query23	18.26	1.48	1.35
query24	1.67	0.26	0.26
query25	0.16	0.08	0.08
query26	0.25	0.17	0.18
query27	0.08	0.08	0.08
query28	13.34	1.01	0.98
query29	12.70	3.31	3.32
query30	0.24	0.07	0.07
query31	2.84	0.39	0.38
query32	3.30	0.47	0.46
query33	2.94	2.86	2.86
query34	17.23	4.46	4.42
query35	4.49	4.51	4.45
query36	0.64	0.45	0.46
query37	0.17	0.17	0.16
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.26 s
Total hot run time: 30.9 s

@dataroaring dataroaring merged commit 02fd95a into apache:master May 28, 2024
dataroaring pushed a commit that referenced this pull request May 28, 2024
@freemandealer @dataroaring
[fix](statistics) NPE when drop partition during publish (#35475)
7136a03 
skip null partition when get base tablets for each be (for further usage
in dedup updated row count in MV)
This may cause publish fail
freemandealer added a commit to freemandealer/doris that referenced this pull request Jun 6, 2024
@freemandealer
[fix](statistics) NPE when drop partition during publish (pick apache…
c121481 
…#35475)

skip null partition when get base tablets for each be (for further usage in dedup updated row count in MV)
This may cause publish fail

cherry pick master apache#35475
@freemandealer freemandealer mentioned this pull request Jun 6, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 6, 2024
@freemandealer
[fix](statistics) NPE when drop partition during publish (pick #35475) (
fedb7e1 
#35977)

skip null partition when get base tablets for each be (for further usage
in dedup updated row count in MV) This may cause publish fail

cherry pick master #35475
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@freemandealer @doris-robot @morningman @gavinchou @yiguolei @dataroaring