Skip to content

branch-3.0: [improvement](statistics)Eliminate null values while sample analyzing ndv. #50574 #51575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 11, 2025
Merged

branch-3.0: [improvement](statistics)Eliminate null values while sample analyzing ndv. #50574 #51575

dataroaring merged 1 commit into from
Jun 11, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 9, 2025

Cherry-picked from #50574

@github-actions github-actions bot requested a review from dataroaring as a code owner June 9, 2025 03:12
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jun 9, 2025
@dataroaring dataroaring reopened this Jun 9, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 9, 2025

run buildall

dataroaring
dataroaring previously approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 11, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Jun 11, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented Jun 11, 2025

PR approved by anyone and no changes requested.

@Jibing-Li
[improvement](statistics)Eliminate null values while sample analyzing…
07e0458 
… ndv. (#50574)

1. Eliminate null values while sample analyzing ndv.
For example, if the values of col1 in table1 are all NULL, the ndv of
col1 should be 0, not 1.

2. If ndv=0 and min/max not null and null_num == rowCount, we keep this
stats info but set to unknown in stats calculator. This is probably
because sampled an all-null tablet.
@Jibing-Li Jibing-Li force-pushed the auto-pick-50574-branch-3.0 branch from f93b9be to 07e0458 Compare June 11, 2025 11:42
@Jibing-Li
Copy link
Contributor

Jibing-Li commented Jun 11, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 11, 2025
@doris-robot
Copy link

doris-robot commented Jun 11, 2025

TPC-H: Total hot run time: 39554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 07e0458e9d1bd2a0ed7fdced0c0e9283780fb40e, data reload: false

------ Round 1 ----------------------------------
q1	17615	6712	6594	6594
q2	2054	168	185	168
q3	10627	1097	1176	1097
q4	10575	763	741	741
q5	7732	2822	2834	2822
q6	217	133	134	133
q7	973	632	610	610
q8	9574	1929	1924	1924
q9	6798	6345	6378	6345
q10	6996	2283	2243	2243
q11	463	264	256	256
q12	395	214	211	211
q13	17787	2976	2966	2966
q14	233	202	207	202
q15	513	461	465	461
q16	483	369	371	369
q17	960	518	648	518
q18	7134	6738	6690	6690
q19	1394	1111	1015	1015
q20	472	201	203	201
q21	3879	3018	3052	3018
q22	1113	978	970	970
Total cold run time: 107987 ms
Total hot run time: 39554 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6534	6525	6484	6484
q2	322	230	228	228
q3	2912	2712	2814	2712
q4	2062	1778	1791	1778
q5	5749	5733	5731	5731
q6	213	132	130	130
q7	2192	1810	1845	1810
q8	3408	3606	3584	3584
q9	9073	8862	8995	8862
q10	3577	3520	3503	3503
q11	586	504	488	488
q12	795	592	601	592
q13	10428	3141	3176	3141
q14	296	288	268	268
q15	516	459	461	459
q16	481	447	429	429
q17	1835	1592	1599	1592
q18	8170	7648	7753	7648
q19	1672	1639	1585	1585
q20	2108	1891	1863	1863
q21	4988	4972	4948	4948
q22	1098	1029	1058	1029
Total cold run time: 69015 ms
Total hot run time: 58864 ms

@doris-robot
Copy link

doris-robot commented Jun 11, 2025

TPC-DS: Total hot run time: 195778 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 07e0458e9d1bd2a0ed7fdced0c0e9283780fb40e, data reload: false

query1	1289	910	926	910
query2	6390	1982	1877	1877
query3	10789	4359	4122	4122
query4	60995	28967	23345	23345
query5	5235	466	456	456
query6	409	179	187	179
query7	5541	324	314	314
query8	310	227	217	217
query9	8443	2620	2594	2594
query10	461	260	253	253
query11	17525	15150	15616	15150
query12	168	104	101	101
query13	1454	457	473	457
query14	10530	6791	6786	6786
query15	202	176	175	175
query16	7210	489	470	470
query17	1218	581	587	581
query18	1875	322	323	322
query19	241	156	153	153
query20	113	113	104	104
query21	212	106	98	98
query22	4789	4500	4587	4500
query23	34066	33868	33710	33710
query24	6229	2920	2852	2852
query25	508	426	408	408
query26	664	164	167	164
query27	1989	342	371	342
query28	4064	2175	2178	2175
query29	669	442	421	421
query30	257	164	151	151
query31	994	793	823	793
query32	75	54	57	54
query33	426	304	291	291
query34	915	499	515	499
query35	839	738	736	736
query36	1079	957	963	957
query37	111	68	64	64
query38	4044	3913	3976	3913
query39	1527	1476	1492	1476
query40	201	102	110	102
query41	51	50	48	48
query42	122	106	105	105
query43	543	494	486	486
query44	1199	830	831	830
query45	187	176	173	173
query46	1163	733	718	718
query47	2063	1902	1936	1902
query48	482	386	376	376
query49	758	438	401	401
query50	857	432	449	432
query51	7465	7113	7251	7113
query52	109	93	93	93
query53	262	193	204	193
query54	609	481	503	481
query55	79	79	82	79
query56	285	281	271	271
query57	1347	1205	1228	1205
query58	240	215	224	215
query59	3262	3106	3070	3070
query60	276	263	261	261
query61	116	106	107	106
query62	753	666	665	665
query63	214	187	179	179
query64	1362	668	619	619
query65	3263	3193	3164	3164
query66	709	290	288	288
query67	16177	15512	15560	15512
query68	4173	579	588	579
query69	414	267	259	259
query70	1128	1138	1086	1086
query71	333	251	259	251
query72	6362	4078	3992	3992
query73	749	345	357	345
query74	10175	9165	9002	9002
query75	3337	2657	2684	2657
query76	2171	1144	1131	1131
query77	481	282	260	260
query78	10581	9663	9581	9581
query79	2298	610	609	609
query80	1410	462	421	421
query81	516	218	216	216
query82	1256	92	87	87
query83	259	143	138	138
query84	285	86	76	76
query85	1025	302	290	290
query86	387	288	295	288
query87	4417	4215	4262	4215
query88	3888	2390	2361	2361
query89	410	290	290	290
query90	1944	187	184	184
query91	188	144	148	144
query92	61	48	48	48
query93	2957	556	553	553
query94	805	300	292	292
query95	354	260	254	254
query96	625	286	285	285
query97	3292	3143	3124	3124
query98	212	203	207	203
query99	1592	1336	1300	1300
Total cold run time: 316094 ms
Total hot run time: 195778 ms

@doris-robot
Copy link

doris-robot commented Jun 11, 2025

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 07e0458e9d1bd2a0ed7fdced0c0e9283780fb40e, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.51	0.51	0.49
query6	1.14	0.74	0.72
query7	0.04	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.54	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	0.78	0.78	0.79
query15	0.84	0.83	0.81
query16	0.39	0.38	0.39
query17	0.99	1.04	1.04
query18	0.24	0.23	0.21
query19	1.96	1.76	1.83
query20	0.01	0.02	0.01
query21	15.39	0.58	0.57
query22	2.53	1.78	1.78
query23	17.04	0.93	0.73
query24	3.26	1.23	1.00
query25	0.13	0.17	0.14
query26	0.49	0.14	0.15
query27	0.04	0.04	0.04
query28	10.02	0.56	0.48
query29	12.56	3.25	3.20
query30	0.24	0.06	0.06
query31	2.87	0.38	0.38
query32	3.26	0.46	0.46
query33	2.99	3.00	2.99
query34	16.64	4.45	4.49
query35	4.51	4.50	4.48
query36	0.67	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.05 s
Total hot run time: 29.99 s

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0039311 into branch-3.0 Jun 11, 2025
25 of 28 checks passed
@github-actions github-actions bot deleted the auto-pick-50574-branch-3.0 branch June 11, 2025 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @Jibing-Li @doris-robot @dataroaring