[Enhancement](metrics) Avoid update_process_fd_num reporting no such file error indiscriminately. #26013

kaka11chen · 2023-10-27T06:02:55Z

Proposed changes

Avoid update_process_fd_num reporting no such file error indiscriminately.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

kaka11chen · 2023-10-27T06:10:40Z

run buildall

github-actions · 2023-10-27T06:12:01Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-27T06:44:13Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.3 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162138648 Bytes

…e error indiscriminately.

kaka11chen · 2023-10-29T07:25:26Z

run buildall

morningman

LGTM

github-actions · 2023-10-29T07:29:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-29T07:29:47Z

PR approved by anyone and no changes requested.

github-actions · 2023-10-29T07:32:35Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-29T07:57:52Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.39 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162140363 Bytes

JackDrogon · 2023-10-30T02:36:39Z

be/src/util/doris_metrics.cpp

+
    process_fd_num_used->set_value(count);

+    auto limits = fmt::format("/proc/{}/limits", pid);


can use ‘/proc/self/limits’

JackDrogon · 2023-10-30T02:39:19Z

be/src/util/doris_metrics.cpp

        process_fd_num_used->set_value(0);
        return;
    }
+    for (const auto& entry : dict_iter) {


use count_if to replace this for loop

JackDrogon · 2023-10-30T02:39:47Z

be/src/util/doris_metrics.cpp

-    if (!st.ok()) {
-        LOG(WARNING) << "failed to count fd: " << st;
+    std::error_code ec;
+    std::filesystem::directory_iterator dict_iter(fmt::format("/proc/{}/fd/", pid), ec);


use ‘/proc/self/fd/'

github-actions · 2023-10-30T03:18:40Z

clang-tidy review says "All clean, LGTM! 👍"

kaka11chen · 2023-10-30T03:22:48Z

run buildall

JackDrogon · 2023-10-30T03:29:18Z

be/src/util/doris_metrics.cpp

 void DorisMetrics::_update_process_thread_num() {
-    int64_t pid = getpid();
-    std::stringstream ss;
    ss << "/proc/" << pid << "/task/";


remove this dead code

JackDrogon · 2023-10-30T03:31:03Z

be/src/util/doris_metrics.cpp

        return;
    }
+    int64_t count = std::count_if(dict_iter, fs::directory_iterator(),
+                                  [](const fs::directory_entry& entry) {


in lambda , use const auto & entry to replace const fs::directory_entry& entry

github-actions · 2023-10-30T03:40:28Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-30T03:43:46Z

clang-tidy review says "All clean, LGTM! 👍"

kaka11chen · 2023-10-30T04:58:43Z

run buildall

kaka11chen · 2023-10-30T05:18:43Z

run buildall

github-actions · 2023-10-30T05:27:22Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-30T05:45:53Z

clang-tidy review says "All clean, LGTM! 👍"

JackDrogon · 2023-10-30T06:32:35Z

be/src/util/doris_metrics.cpp

-    Status st = io::global_local_filesystem()->iterate_directory(ss.str(), cb);
-    if (!st.ok()) {
-        LOG(WARNING) << "failed to count fd: " << st;
+    std::error_code ec;


remove these unused shadow variable

JackDrogon · 2023-10-30T06:35:23Z

be/src/util/doris_metrics.cpp

        return;
    }
+    int64_t count = std::count_if(dict_iter, std::filesystem::fs::directory_iterator(), [](const auto& entry) {
+        std::error_code ec;


not use shadow variable, just use dec or something else

JackDrogon · 2023-10-30T06:39:52Z

be/src/util/doris_metrics.cpp

+        process_fd_num_used->set_value(0);
        return;
    }
+    int64_t count = std::count_if(dict_iter, std::filesystem::directory_iterator(), [](const auto& entry) {


use std::filesystem::end(dict_iter) to replace std::filesystem::directory_iterator(), std::filesystem::directory_iterator() is ub in end

github-actions · 2023-10-30T07:01:16Z

clang-tidy review says "All clean, LGTM! 👍"

kaka11chen · 2023-10-30T07:29:13Z

run buildall

doris-robot · 2023-10-30T07:48:28Z

TeamCity be ut coverage result:
Function Coverage: 37.37% (8475/22676)
Line Coverage: 29.66% (68516/230968)
Region Coverage: 28.24% (35584/126018)
Branch Coverage: 25.10% (18214/72564)
Coverage Report: http://coverage.selectdb-in.cc/coverage/533fe3c91acf73113bf20b82f301dfd6f5efe9d3_533fe3c91acf73113bf20b82f301dfd6f5efe9d3/report/index.html

doris-robot · 2023-10-30T08:08:08Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.06 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162068629 Bytes

JackDrogon

LGTM

morningman

LGTM

github-actions · 2023-10-31T09:20:50Z

PR approved by at least one committer and no changes requested.

…file error indiscriminately #26013 (#26373)

…file error indiscriminately. (apache#26013) Avoid `update_process_fd_num` reporting `no such file error `indiscriminately.

…file error indiscriminately apache#26013 (apache#26373)

…file error indiscriminately. (apache#26013) Avoid `update_process_fd_num` reporting `no such file error `indiscriminately.

kaka11chen changed the title ~~[Enhancement](metrics) Avoid update_process_fd_num report no such fil…~~ [Enhancement](metrics) Avoid update_process_fd_num reporting no such file error indiscriminately. Oct 27, 2023

[Enhancement](metrics) Avoid update_process_fd_num report no such fil…

0836acb

…e error indiscriminately.

kaka11chen force-pushed the update_process_fd_num_error branch from 1c29115 to 0836acb Compare October 29, 2023 07:25

morningman previously approved these changes Oct 29, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2023

github-actions bot added the reviewed label Oct 29, 2023

JackDrogon suggested changes Oct 30, 2023

View reviewed changes

kaka11chen dismissed morningman’s stale review via 1960ef8 October 30, 2023 03:10

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 30, 2023

JackDrogon suggested changes Oct 30, 2023

View reviewed changes

kaka11chen force-pushed the update_process_fd_num_error branch 2 times, most recently from 5b2bb09 to 3fd5472 Compare October 30, 2023 03:34

kaka11chen force-pushed the update_process_fd_num_error branch from 3fd5472 to 0de5285 Compare October 30, 2023 05:18

kaka11chen force-pushed the update_process_fd_num_error branch from 0de5285 to bc5ec47 Compare October 30, 2023 05:38

JackDrogon suggested changes Oct 30, 2023

View reviewed changes

Fix comments.

533fe3c

kaka11chen force-pushed the update_process_fd_num_error branch from bc5ec47 to 533fe3c Compare October 30, 2023 06:52

JackDrogon approved these changes Oct 30, 2023

View reviewed changes

morningman approved these changes Oct 31, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 31, 2023

morningman added the dev/2.0.3 label Oct 31, 2023

morningman approved these changes Oct 31, 2023

View reviewed changes

morningman merged commit b619982 into apache:master Oct 31, 2023

kaka11chen mentioned this pull request Nov 3, 2023

[Enhancement](metrics) Avoid update_process_fd_num reporting no such file error indiscriminately. #26373

Merged

xiaokang pushed a commit that referenced this pull request Nov 3, 2023

[Enhancement](metrics) Avoid update_process_fd_num reporting no such …

6a5cfc4

…file error indiscriminately #26013 (#26373)

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 3, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023

[Enhancement](metrics) Avoid update_process_fd_num reporting no such …

5444f68

…file error indiscriminately. (apache#26013) Avoid `update_process_fd_num` reporting `no such file error `indiscriminately.

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023

[Enhancement](metrics) Avoid update_process_fd_num reporting no such …

2ff1e00

…file error indiscriminately apache#26013 (apache#26373)

dataroaring mentioned this pull request Aug 19, 2024

[bug](metrics) fix be metric doris_be_process_thread_num is zero #35511

Merged


		process_fd_num_used->set_value(count);

		auto limits = fmt::format("/proc/{}/limits", pid);

[Enhancement](metrics) Avoid update_process_fd_num reporting no such file error indiscriminately. #26013

[Enhancement](metrics) Avoid update_process_fd_num reporting no such file error indiscriminately. #26013

Uh oh!

Conversation

kaka11chen commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

kaka11chen commented Oct 27, 2023

Uh oh!

github-actions bot commented Oct 27, 2023

Uh oh!

doris-robot commented Oct 27, 2023

Uh oh!

kaka11chen commented Oct 29, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 29, 2023

Uh oh!

github-actions bot commented Oct 29, 2023

Uh oh!

github-actions bot commented Oct 29, 2023

Uh oh!

doris-robot commented Oct 29, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

kaka11chen commented Oct 30, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

kaka11chen commented Oct 30, 2023

Uh oh!

kaka11chen commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

kaka11chen commented Oct 30, 2023

Uh oh!

doris-robot commented Oct 30, 2023

Uh oh!

doris-robot commented Oct 30, 2023

Uh oh!

JackDrogon left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

kaka11chen commented Oct 27, 2023 •

edited

Loading