Skip to content

[fix](load) fix no error url if no partition can be found #36831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 26, 2024
Merged

[fix](load) fix no error url if no partition can be found #36831

dataroaring merged 1 commit into from
Jun 26, 2024

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jun 25, 2024

Proposed changes

before

Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0
}

after

Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3"
}

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Jun 25, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@sollhui
Copy link
Contributor Author

sollhui commented Jun 25, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-H: Total hot run time: 39502 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a839716e886c1b56f131bac8e31d816e423facdc, data reload: false

------ Round 1 ----------------------------------
q1	17605	4318	4243	4243
q2	2015	195	188	188
q3	10504	1103	1156	1103
q4	10175	716	843	716
q5	7468	2689	2633	2633
q6	220	137	138	137
q7	955	612	624	612
q8	9248	2063	2079	2063
q9	9017	6515	6454	6454
q10	8966	3742	3717	3717
q11	459	241	241	241
q12	465	225	223	223
q13	17756	2999	2943	2943
q14	272	231	224	224
q15	530	481	475	475
q16	521	379	385	379
q17	973	700	665	665
q18	7917	7465	7305	7305
q19	5371	1362	1550	1362
q20	658	315	332	315
q21	4891	3165	3890	3165
q22	401	344	339	339
Total cold run time: 116387 ms
Total hot run time: 39502 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4504	4394	4281	4281
q2	368	273	272	272
q3	2956	2860	2890	2860
q4	1990	1700	1660	1660
q5	5634	5467	5486	5467
q6	217	130	136	130
q7	2251	1902	1860	1860
q8	3324	3435	3744	3435
q9	8727	8647	8812	8647
q10	4078	3870	3668	3668
q11	590	507	519	507
q12	836	675	640	640
q13	17126	3181	3172	3172
q14	326	300	282	282
q15	527	504	492	492
q16	475	436	444	436
q17	1811	1520	1523	1520
q18	8131	7931	8023	7931
q19	1811	1665	1593	1593
q20	3118	1851	1885	1851
q21	5024	4985	4801	4801
q22	626	536	589	536
Total cold run time: 74450 ms
Total hot run time: 56041 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-DS: Total hot run time: 173865 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a839716e886c1b56f131bac8e31d816e423facdc, data reload: false

query1	922	387	366	366
query2	6461	2421	2405	2405
query3	6626	205	206	205
query4	21610	17283	17262	17262
query5	3618	468	508	468
query6	237	165	160	160
query7	4596	296	293	293
query8	310	294	295	294
query9	8602	2404	2391	2391
query10	591	302	263	263
query11	10484	10145	9988	9988
query12	108	84	80	80
query13	1632	384	385	384
query14	10227	6925	7645	6925
query15	239	195	195	195
query16	7719	284	262	262
query17	1778	539	507	507
query18	1963	275	275	275
query19	200	146	170	146
query20	89	86	80	80
query21	208	127	127	127
query22	4348	3994	4075	3994
query23	33926	33608	33830	33608
query24	11147	2957	2890	2890
query25	625	379	390	379
query26	726	158	162	158
query27	2323	327	323	323
query28	5803	2167	2163	2163
query29	870	647	638	638
query30	252	154	153	153
query31	1001	764	738	738
query32	92	57	55	55
query33	664	290	293	290
query34	911	478	505	478
query35	746	667	620	620
query36	1111	936	1003	936
query37	150	75	83	75
query38	2952	2788	2847	2788
query39	922	855	852	852
query40	221	128	137	128
query41	56	57	52	52
query42	109	103	104	103
query43	605	552	555	552
query44	1102	733	731	731
query45	191	164	168	164
query46	1080	723	703	703
query47	1797	1746	1759	1746
query48	385	297	308	297
query49	835	405	409	405
query50	763	413	404	404
query51	6945	6851	6804	6804
query52	104	93	89	89
query53	364	295	291	291
query54	894	458	444	444
query55	76	76	75	75
query56	300	278	280	278
query57	1152	1064	1056	1056
query58	262	242	250	242
query59	3511	3211	3037	3037
query60	306	282	288	282
query61	155	89	91	89
query62	596	435	439	435
query63	320	289	288	288
query64	8517	2269	1760	1760
query65	3164	3120	3146	3120
query66	800	334	333	333
query67	15366	14981	14934	14934
query68	4739	542	554	542
query69	636	464	436	436
query70	1201	1108	1147	1108
query71	462	273	275	273
query72	7094	5515	5520	5515
query73	759	324	335	324
query74	5823	5479	5457	5457
query75	3598	2694	2622	2622
query76	3366	945	912	912
query77	607	294	291	291
query78	10693	9767	9652	9652
query79	2662	524	536	524
query80	1219	467	477	467
query81	604	220	220	220
query82	1405	110	106	106
query83	260	184	173	173
query84	226	84	86	84
query85	1362	296	275	275
query86	474	328	337	328
query87	3286	3100	3088	3088
query88	4085	2370	2355	2355
query89	479	373	384	373
query90	1754	186	186	186
query91	126	100	167	100
query92	59	51	48	48
query93	2231	520	515	515
query94	1081	189	186	186
query95	415	313	319	313
query96	604	264	264	264
query97	3265	3043	3082	3043
query98	222	208	194	194
query99	1160	837	835	835
Total cold run time: 272722 ms
Total hot run time: 173865 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

ClickBench: Total hot run time: 30.58 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a839716e886c1b56f131bac8e31d816e423facdc, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.21	0.05	0.05
query4	1.68	0.09	0.08
query5	0.51	0.49	0.49
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.55	0.49	0.50
query10	0.55	0.55	0.54
query11	0.14	0.11	0.12
query12	0.15	0.12	0.12
query13	0.58	0.58	0.58
query14	0.75	0.78	0.80
query15	0.84	0.81	0.81
query16	0.36	0.37	0.37
query17	0.97	1.04	1.00
query18	0.21	0.25	0.25
query19	1.81	1.65	1.72
query20	0.01	0.01	0.01
query21	15.43	0.66	0.65
query22	4.31	7.32	1.91
query23	18.32	1.41	1.24
query24	2.12	0.23	0.22
query25	0.16	0.09	0.08
query26	0.26	0.17	0.17
query27	0.09	0.08	0.08
query28	13.16	1.01	1.00
query29	12.64	3.35	3.33
query30	0.26	0.06	0.06
query31	2.86	0.40	0.39
query32	3.27	0.47	0.49
query33	2.85	2.89	2.93
query34	16.86	4.47	4.42
query35	4.49	4.46	4.47
query36	0.66	0.45	0.46
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.08	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.05 s
Total hot run time: 30.58 s

dataroaring
dataroaring approved these changes Jun 26, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 8170df9 into apache:master Jun 26, 2024
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@sollhui @dataroaring
[fix](load) fix no error url if no partition can be found (#36831)
e945a1b 
## Proposed changes

before
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0
}
```

after
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3"
}
```
@sollhui sollhui mentioned this pull request Jul 7, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 8, 2024
@sollhui
[branch-2.1](load) fix no error url if no partition can be found (#36831
c66df8d 
) (#37401)

## Proposed changes

pick #36831

before
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0
}
```

after
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3"
}
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@kaijchen kaijchen kaijchen approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sollhui @doris-robot @kaijchen @dataroaring @liaoxin01