Skip to content

[bug](GC)the issue of incorrect disk usage #18397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 5 commits into from
Apr 8, 2023
Merged

[bug](GC)the issue of incorrect disk usage #18397

morningman merged 5 commits into from
Apr 8, 2023

Conversation

@chenlinzhong
Copy link
Contributor

@chenlinzhong chenlinzhong commented Apr 5, 2023

Proposed changes

disk used capacity is incorrect：
DataUsedCapacity+AvailCapacity !=TotalCapacity
image

Problem summary

some invalid rowset rowset not be deleted for ever due to the bug:
image
when key_may_exists() return true not means it is real exists

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@chenlinzhong chenlinzhong requested a review from morningman April 5, 2023 03:17
@chenlinzhong chenlinzhong changed the title [bugfix] Fix the issue of incorrect disk usage [bug] (fix) the issue of incorrect disk usage Apr 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhong chenlinzhong changed the title [bug] (fix) the issue of incorrect disk usage [bug] (GC) the issue of incorrect disk usage Apr 5, 2023
@chenlinzhong chenlinzhong changed the title [bug] (GC) the issue of incorrect disk usage [bug](GC)the issue of incorrect disk usage Apr 5, 2023
@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 5, 2023

run all

@morningman morningman self-assigned this Apr 6, 2023
@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

run p0

1 similar comment
@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

run p0

@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

run beut

@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

run feut

@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

run p0

@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

ru beut

@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 6, 2023

run clickbench

morningman
morningman reviewed Apr 6, 2023
View reviewed changes
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 6, 2023 •
edited
Loading

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 36.12 seconds
stream load tsv: 443 seconds loaded 74807831229 Bytes, about 161 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 72 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230407160549_clickbench_pr_126080.html

@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2023

clang-tidy review says "All clean, LGTM! 👍"

3 similar comments
@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2023

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2023

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2023

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhong
Copy link
Contributor Author

chenlinzhong commented Apr 7, 2023

run buildall

morningman
morningman approved these changes Apr 8, 2023
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 1616783 into apache:master Apr 8, 2023
@morningman morningman mentioned this pull request Apr 8, 2023
Merged
5 tasks
morningman added a commit that referenced this pull request Apr 10, 2023
@morningman
[fix](hdfs) not setting hadoop username when kerberos enabled (#18485)
ea47a6a 
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
morningman added a commit that referenced this pull request Apr 11, 2023
@morningman
[fix](hdfs) not setting hadoop username when kerberos enabled (#18485)
62eb640 
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
morningman pushed a commit that referenced this pull request Apr 12, 2023
@chenlinzhong @morningman
[bug](GC)the issue of incorrect disk usage (#18397)
1af97cd
morningman pushed a commit that referenced this pull request Apr 13, 2023
@chenlinzhong @morningman
[bug](GC)the issue of incorrect disk usage (#18397)
c54dacf
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
@chenlinzhong @gnehil
[bug](GC)the issue of incorrect disk usage (apache#18397)
13318ca
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
@morningman @gnehil
[fix](hdfs) not setting hadoop username when kerberos enabled (apache…
1e73966 
…#18485)

1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up apache#18397
3. Add KW_HOSTNAME to keywords region, follow up apache#17329
4. Fix tvf not working with pipeline engine, follow up apache#18376
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
@morningman
[fix](hdfs) not setting hadoop username when kerberos enabled (apache…
b54bba0 
…#18485)

1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up apache#18397
3. Add KW_HOSTNAME to keywords region, follow up apache#17329
4. Fix tvf not working with pipeline engine, follow up apache#18376
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
@chenlinzhong
[bug](GC)the issue of incorrect disk usage (apache#18397)
4ef2c7c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

Assignees

@morningman morningman

Labels

dev/1.2.4-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@chenlinzhong @hello-stephen @morningman @yangzhg