[Feature] Support for querying the trash used capacity by BiteTheDDDDt · Pull Request #6247 · apache/doris

BiteTheDDDDt · 2021-07-15T06:31:22Z

Proposed changes

Support for querying the trash used capacity.
Now user can proactively scan trash directory.

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)
Optimization. Including functional usability improvements and performance improvements.
Dependency. Such as changes related to third-party components.
Other.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have created an issue on (Fix [Feature] Support for querying the trash used capacity #6246) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If these changes need document changes, I have updated the document
Any dependent changes have been merged

…djust query define at 'sql_parser' && remove duplicates input backend

yangzhg · 2021-07-15T06:54:27Z

fe/fe-core/src/main/jflex/sql_scanner.flex

        keywordMap.put("array", new Integer(SqlParserSymbols.KW_ARRAY));
        keywordMap.put("backend", new Integer(SqlParserSymbols.KW_BACKEND));
        keywordMap.put("backends", new Integer(SqlParserSymbols.KW_BACKENDS));
+        keywordMap.put("trash", new Integer(SqlParserSymbols.KW_TRASH));


keep the alpha order

yangzhg · 2021-07-15T06:58:42Z

fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java

+                trashUsedCapacityB, System.currentTimeMillis() - start);
+                ok = true;
+            } catch (Exception e) {
+                LOG.warn("task exec error. backend[{}]", backend.getId(), e);


Just a log without further processing?

Just a log without further processing?

now will display null when program throw error

be/src/service/backend_service.cpp

EmmyMiao87 · 2021-07-15T07:03:08Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTrashStmt.java

+
+    public ShowTrashStmt(List<String> backends) {
+        ImmutableMap<Long, Backend> backendsInfo = Catalog.getCurrentSystemInfo().getIdToBackend();
+        Map<String, Long> backendsID = new HashMap<String, Long>();


Why not use backend id directly？

Do you mean to directly use 'backend id' query instead of 'backend host':'backend heart beat port'?

EmmyMiao87 · 2021-07-15T07:03:40Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTrashStmt.java

+
+public class ShowTrashStmt extends ShowStmt {
+    public static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>().add("BackendId")
+            .add("Backend").add("TrashUsedCapacity").build();


Would it be better to add the trash information to show proc as well?

Would it be better to add the trash information to show proc as well?

now support 'show proc '/trash''

EmmyMiao87 · 2021-07-15T07:03:58Z

fe/fe-core/src/main/cup/sql_parser.cup

    {:
        RESULT = new ShowBackendsStmt();
    :}
+    | KW_TRASH KW_ON LPAREN string_list:backends RPAREN


Please add document of show trash stmt

Please add document of show trash stmt

done.

be/src/olap/storage_engine.cpp

2. display null when connect fail to client 3. trash information to show proc, and support show proc '/trash' 4. add document of show trash stmt

Co-authored-by: Zhengguo Yang <780531911@qq.com>

yangzhg · 2021-07-16T01:44:17Z

fe/fe-core/src/main/java/org/apache/doris/common/proc/TrashProcNode.java

+                Pair<Double, String> trashUsedCapacity = DebugUtil.getByteUint(trashUsedCapacityB);
+                backendInfo.add(DebugUtil.DECIMAL_FORMAT_SCALE_3.format(trashUsedCapacity.first) + " " + trashUsedCapacity.second);
+            } else {
+                backendInfo.add("null");


not literal null

2. Support for querying the specific disk information of a single backend. 3. Updated related interfaces and documents.

yangzhg

LGTM

github-actions · 2021-08-03T11:47:14Z

PR approved by at least one committer and no changes requested.

github-actions · 2021-08-03T11:47:16Z

PR approved by anyone and no changes requested.

morningman

ShowTrashDiskStmt and ShowTrashStmt can be unified

morningman · 2021-08-05T06:08:41Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTrashDiskStmt.java

+        if (ConnectContext.get().getSessionVariable().getForwardToMaster()) {
+            return RedirectStatus.FORWARD_NO_SYNC;
+        } else {
+            return RedirectStatus.NO_FORWARD;


No need to forward

morningman · 2021-08-05T06:12:13Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTrashStmt.java

+        if (ConnectContext.get().getSessionVariable().getForwardToMaster()) {
+            return RedirectStatus.FORWARD_NO_SYNC;
+        } else {
+            return RedirectStatus.NO_FORWARD;


No need to forward

No need to forward

I fix it at c3f7c7e

morningman

LGTM

github-actions · 2021-08-07T14:30:39Z

PR approved by at least one committer and no changes requested.

BiteTheDDDDt added 2 commits July 14, 2021 20:31

add query 'show trash' for display backends trash used capacity

a4111d2

merge column 'BackendHost','BackendHeartBeatPort' into 'Backend' && a…

80ba2c3

…djust query define at 'sql_parser' && remove duplicates input backend

yangzhg reviewed Jul 15, 2021

View reviewed changes

be/src/service/backend_service.cpp Outdated Show resolved Hide resolved

EmmyMiao87 reviewed Jul 15, 2021

View reviewed changes

yangzhg reviewed Jul 15, 2021

View reviewed changes

be/src/olap/storage_engine.cpp Outdated Show resolved Hide resolved

BiteTheDDDDt and others added 5 commits July 15, 2021 19:15

1. change sql_scannner.flex to keep the alpha order.

f92938d

2. display null when connect fail to client 3. trash information to show proc, and support show proc '/trash' 4. add document of show trash stmt

fix bug about document of show trash stmt

b89cd13

Update be/src/olap/storage_engine.cpp

ec5c79e

Co-authored-by: Zhengguo Yang <780531911@qq.com>

Update be/src/service/backend_service.cpp

7a410fe

Co-authored-by: Zhengguo Yang <780531911@qq.com>

remove useless TITLENAME define

f84e758

yangzhg reviewed Jul 16, 2021

View reviewed changes

BiteTheDDDDt added 4 commits July 16, 2021 11:01

change literal null to empty string

2440f2e

add support for 'SHOW PROC /trash/backendId'

a2b9718

add limit about query format like 'trash/backendId/backendId'

0293c53

1. Removed the syntax for querying multiple backends.

c484ec8

2. Support for querying the specific disk information of a single backend. 3. Updated related interfaces and documents.

yangzhg previously approved these changes Aug 3, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 3, 2021

github-actions bot added the reviewed label Aug 3, 2021

morningman reviewed Aug 5, 2021

View reviewed changes

change Redirect to 'NO_FORWARD' at ShowTrashStmt/ShowTrashDiskStmt

c3f7c7e

BiteTheDDDDt dismissed yangzhg’s stale review via c3f7c7e August 5, 2021 06:54

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 5, 2021

morningman approved these changes Aug 7, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 7, 2021

morningman merged commit 236e0f1 into apache:master Aug 10, 2021

morningman mentioned this pull request Oct 10, 2021

Release note 0.15.0 #6806

Closed

BiteTheDDDDt deleted the dev_show branch January 20, 2025 06:45

Conversation

BiteTheDDDDt commented Jul 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Types of changes

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 3, 2021

Uh oh!

github-actions bot commented Aug 3, 2021

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BiteTheDDDDt Aug 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

BiteTheDDDDt commented Jul 15, 2021 •

edited

Loading

BiteTheDDDDt Aug 5, 2021 •

edited

Loading