Skip to content

[fix](load) return DataQualityError when filtered rows exceeds limit (#47617) #49289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 5 commits into from
Apr 2, 2025
Merged

[fix](load) return DataQualityError when filtered rows exceeds limit (#47617) #49289

yiguolei merged 5 commits into from
Apr 2, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Mar 20, 2025

backport #47617

@kaijchen
[fix](load) return DataQualityError when filtered rows exceeds limit (a…
21715ea 
…pache#47617)

Issue Number: DORIS-17408

Problem Summary:

When the load is canceled due to an excessive number of filtered rows in
`CsvReader`, the end of stream flag is set, but an error is not
returned. As a result, the following confusing message is shown to the
user:

```json
"Message": "[CANCELLED]cancelled: closed",
```

This PR addresses the issue by returning a `DataQualityError` in such
cases. The updated message will be:

```json
"Message": "[CANCELLED]cancelled: [DATA_QUALITY_ERROR]Encountered unqualified data, stop processing. cur path: .",
```
@kaijchen kaijchen requested a review from yiguolei as a code owner March 20, 2025 03:18
@kaijchen
Copy link
Member Author

kaijchen commented Mar 20, 2025

run buildall

@kaijchen
Copy link
Member Author

kaijchen commented Mar 24, 2025

run buildall

@kaijchen
Copy link
Member Author

kaijchen commented Mar 24, 2025

run buildall

@kaijchen
Copy link
Member Author

kaijchen commented Mar 25, 2025

run buildall

@kaijchen
Copy link
Member Author

kaijchen commented Apr 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

BE UT Coverage Report

Increment line coverage 19.75% (16/81) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 37.40% (9855/26350)
Line Coverage 29.12% (82653/283834)
Region Coverage 27.72% (42376/152893)
Branch Coverage 24.46% (21569/88182)

@yiguolei yiguolei merged commit cf847b8 into apache:branch-2.1 Apr 2, 2025
19 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kaijchen @doris-robot @yiguolei