Skip to content

[opt](nereids) tabe row count priority: user injected > BE report > analyzed #40529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 1 commit into from
Sep 9, 2024
Merged

[opt](nereids) tabe row count priority: user injected > BE report > analyzed #40529

englefly merged 1 commit into from
Sep 9, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Sep 9, 2024

Proposed changes

there are 2 sources for table row count, one is analyzed result, another is from BE report.
But neither of them is accurate. the priorities of them from high to low:
user injected > BE report > analyzed

Issue Number: close #xxx

@englefly
there are 2 sources for table row count, one is analyzed result, anot…
d32586a 
…her is from BE report.

But neither of them is accurate. the priorities of them from high to low:
user injected > BE report > analyzed
@doris-robot
Copy link

doris-robot commented Sep 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

englefly commented Sep 9, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TPC-H: Total hot run time: 37840 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d32586a627281f8f5a98cf3ff31a383be73df61d, data reload: false

------ Round 1 ----------------------------------
q1	17638	4570	4299	4299
q2	2021	194	186	186
q3	11330	958	1176	958
q4	10416	692	691	691
q5	7740	2861	2805	2805
q6	229	142	141	141
q7	964	622	608	608
q8	9316	2058	2115	2058
q9	7307	6520	6564	6520
q10	6991	2205	2328	2205
q11	459	247	249	247
q12	394	231	225	225
q13	17811	3145	3125	3125
q14	286	235	235	235
q15	534	507	491	491
q16	519	428	427	427
q17	999	686	728	686
q18	7377	6849	6890	6849
q19	1390	1096	1103	1096
q20	673	329	325	325
q21	4400	3164	2691	2691
q22	1079	1017	972	972
Total cold run time: 109873 ms
Total hot run time: 37840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4294	4291	4291
q2	404	277	275	275
q3	2866	2728	2677	2677
q4	1925	1717	1637	1637
q5	5671	5734	5696	5696
q6	237	139	134	134
q7	2251	1840	1884	1840
q8	3329	3428	3447	3428
q9	8897	8876	8853	8853
q10	3556	3400	3353	3353
q11	641	550	520	520
q12	804	649	659	649
q13	14631	3352	3298	3298
q14	317	294	302	294
q15	558	488	497	488
q16	561	511	516	511
q17	1859	1575	1542	1542
q18	8268	7849	7829	7829
q19	1751	1656	1499	1499
q20	2176	1986	1909	1909
q21	5885	5590	5537	5537
q22	1144	1026	1035	1026
Total cold run time: 72056 ms
Total hot run time: 57286 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TPC-DS: Total hot run time: 192617 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d32586a627281f8f5a98cf3ff31a383be73df61d, data reload: false

query1	1255	888	916	888
query2	6371	1855	1972	1855
query3	10575	4004	3885	3885
query4	59330	26544	23140	23140
query5	5513	517	519	517
query6	408	176	162	162
query7	5892	292	286	286
query8	332	224	220	220
query9	9122	2499	2489	2489
query10	501	289	270	270
query11	18201	15264	15326	15264
query12	158	111	105	105
query13	1569	416	409	409
query14	11062	7336	7381	7336
query15	247	180	182	180
query16	7577	495	473	473
query17	1183	608	560	560
query18	2019	296	289	289
query19	300	152	147	147
query20	124	114	114	114
query21	207	103	103	103
query22	4593	4476	4890	4476
query23	34247	33322	33433	33322
query24	5904	2883	2893	2883
query25	533	398	389	389
query26	698	153	153	153
query27	1795	278	275	275
query28	3700	2042	2020	2020
query29	681	403	404	403
query30	232	155	156	155
query31	922	753	810	753
query32	94	55	61	55
query33	424	285	289	285
query34	882	485	468	468
query35	833	706	731	706
query36	1065	943	942	942
query37	147	89	84	84
query38	4031	3934	3998	3934
query39	1433	1423	1404	1404
query40	194	117	112	112
query41	47	45	45	45
query42	110	94	93	93
query43	529	475	475	475
query44	1089	777	722	722
query45	191	162	165	162
query46	1088	743	711	711
query47	1872	1772	1788	1772
query48	379	298	282	282
query49	760	450	446	446
query50	843	418	439	418
query51	7012	6889	6797	6797
query52	98	87	85	85
query53	246	176	177	176
query54	556	454	451	451
query55	74	75	73	73
query56	273	255	266	255
query57	1188	1075	1064	1064
query58	223	223	235	223
query59	3073	2806	3089	2806
query60	300	268	277	268
query61	117	103	94	94
query62	751	650	662	650
query63	219	192	184	184
query64	2801	677	644	644
query65	3298	3123	3167	3123
query66	619	330	339	330
query67	15428	15122	15174	15122
query68	3009	593	588	588
query69	396	291	267	267
query70	1196	1112	1121	1112
query71	346	288	273	273
query72	5238	4092	3922	3922
query73	774	328	325	325
query74	9135	8891	8799	8799
query75	3330	2671	2721	2671
query76	1409	928	1053	928
query77	541	334	322	322
query78	10563	9360	9165	9165
query79	1055	542	545	542
query80	714	514	507	507
query81	458	237	232	232
query82	244	140	145	140
query83	174	147	206	147
query84	254	78	77	77
query85	670	287	277	277
query86	306	295	301	295
query87	4348	4354	4299	4299
query88	3348	2277	2266	2266
query89	396	281	283	281
query90	1685	193	193	193
query91	125	99	97	97
query92	59	48	49	48
query93	1091	533	535	533
query94	549	291	317	291
query95	343	251	249	249
query96	585	261	268	261
query97	3266	3100	3059	3059
query98	225	204	194	194
query99	1488	1275	1277	1275
Total cold run time: 304858 ms
Total hot run time: 192617 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

ClickBench: Total hot run time: 30.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d32586a627281f8f5a98cf3ff31a383be73df61d, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.52	0.50	0.50
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.53	0.50	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	1.40	1.41	1.39
query15	0.83	0.82	0.82
query16	0.37	0.37	0.38
query17	1.07	1.04	1.02
query18	0.21	0.19	0.20
query19	1.81	1.78	1.85
query20	0.01	0.00	0.01
query21	15.41	0.67	0.67
query22	4.19	7.85	1.52
query23	18.30	1.41	1.23
query24	2.09	0.23	0.22
query25	0.14	0.08	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.07
query28	13.22	1.02	0.99
query29	12.60	3.35	3.30
query30	0.26	0.05	0.05
query31	2.88	0.41	0.39
query32	3.27	0.47	0.49
query33	2.99	3.00	3.03
query34	17.01	4.43	4.41
query35	4.42	4.46	4.45
query36	0.67	0.47	0.49
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.06
query42	0.06	0.04	0.04
query43	0.05	0.05	0.04
Total cold run time: 109.98 s
Total hot run time: 30.99 s

Jibing-Li
Jibing-Li approved these changes Sep 9, 2024
View reviewed changes
Copy link
Contributor

@Jibing-Li Jibing-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by anyone and no changes requested.

@englefly englefly merged commit 0d3374b into apache:master Sep 9, 2024
@seawinde seawinde mentioned this pull request Sep 18, 2024
Merged
englefly added a commit to englefly/incubator-doris that referenced this pull request Sep 26, 2024
@englefly
[opt](nereids) tabe row count priority: user injected > BE report > a…
8bc977d 
…nalyzed apache#40529'
This was referenced Sep 26, 2024
Closed
Merged
englefly added a commit to englefly/incubator-doris that referenced this pull request Sep 29, 2024
@englefly
[opt](nereids) tabe row count priority: user injected > BE report > a…
1fdd098 
…nalyzed apache#40529'
englefly added a commit that referenced this pull request Sep 29, 2024
@englefly
[opt](nereids) tabe row count priority: user injected > BE report > …
3cd8da3 
…analyzed (3.0) (#41347)

## Proposed changes
pick #40529
Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly mentioned this pull request Sep 29, 2024
Merged
englefly added a commit that referenced this pull request Oct 8, 2024
@englefly
[fix](nereids)do not add delta row count to BE reported row count (#4…
958d638 
…1464)

## Proposed changes
the table row count is in 3 cases:
1. injected row count
2. analyzed row count + delta row count
3. BE reported row count.

in previous pr #40529, we added delta row count in all 3 cases
Issue Number: close #xxx

<!--Describe your changes.-->
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Oct 10, 2024
@englefly @eldenmoon
[fix](nereids)do not add delta row count to BE reported row count (ap…
d1437e6 
…ache#41464)

## Proposed changes
the table row count is in 3 cases:
1. injected row count
2. analyzed row count + delta row count
3. BE reported row count.

in previous pr apache#40529, we added delta row count in all 3 cases
Issue Number: close #xxx

<!--Describe your changes.-->
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@englefly @cjj2010
[fix](nereids)do not add delta row count to BE reported row count (ap…
c0ea4d8 
…ache#41464)

## Proposed changes
the table row count is in 3 cases:
1. injected row count
2. analyzed row count + delta row count
3. BE reported row count.

in previous pr apache#40529, we added delta row count in all 3 cases
Issue Number: close #xxx

<!--Describe your changes.-->
amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
@englefly
[fix](nereids)do not add delta row count to BE reported row count (ap…
cf96a5f 
…ache#41464)

## Proposed changes
the table row count is in 3 cases:
1. injected row count
2. analyzed row count + delta row count
3. BE reported row count.

in previous pr apache#40529, we added delta row count in all 3 cases
Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly added not-merge/2.0 do not merge into 2.0 branch not-merge/2.1 and removed dev/2.0.x labels Oct 18, 2024
This was referenced Oct 18, 2024
Merged
Merged
englefly added a commit that referenced this pull request Oct 22, 2024
@englefly
[Pick](nereids) pick 40529 41464 40349 (#42073)
9c4ce73 
## Proposed changes
pick  #40529 #41464 #40349

Issue Number: close #xxx

<!--Describe your changes.-->
englefly added a commit that referenced this pull request Oct 22, 2024
@englefly
[Pick](nereids)table row related Prs 3.0 (#42237)
e3efb14 
## Proposed changes
pick #41464 #40529 #40349 #39222
Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@englefly @doris-robot @starocean999 @Jibing-Li @morrySnow