[Improve](start-shell)Optimize fe&be startup #24556

CalvinKirs · 2023-09-18T10:36:39Z

Added startup --console option
Modify the original default startup input stream to avoid log loss
--console is more suitable for development scenarios. When the user does not add any startup parameters, it is better to retain logs.

Proposed changes

sh start_fe/start_be --console is used to instruct the program to run in console mode.
sh start_fe/start_be --daemon is used to instruct the program to run in daemon mode.
sh start_fe/start_be used starts as a background execution, records output and error logs to the specified file

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-09-18T10:37:34Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/start_be.sh.orig
+++ bin/start_be.sh
@@ -51,7 +51,7 @@
     --console)
         RUN_CONSOLE=1
         shift
-        ;;      
+        ;;
     --)
         shift
         break
@@ -345,6 +345,6 @@
 elif [[ "${RUN_CONSOLE}" -eq 1 ]]; then
     export DORIS_LOG_TO_STDERR=1
     ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" 2>&1 </dev/null
-else 
+else
     ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" >>"${LOG_DIR}/be.out" 2>&1 </dev/null &
 fi
--- bin/start_fe.sh.orig
+++ bin/start_fe.sh
@@ -54,7 +54,7 @@
     --console)
         RUN_CONSOLE=1
         shift
-        ;;      
+        ;;
     --version)
         OPT_VERSION="--version"
         shift
@@ -235,7 +235,7 @@
     nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
 elif [[ "${RUN_CONSOLE}" -eq 1 ]]; then
     export DORIS_LOG_TO_STDERR=1
-     ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
+    ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
 else
     ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
 fi
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

Added startup --console option Modify the original default startup input stream to avoid log loss

CalvinKirs · 2023-09-18T13:05:34Z

run buildall

doris-robot · 2023-09-18T13:34:02Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7981/21669)
Line Coverage: 28.86% (64054/221931)
Region Coverage: 27.75% (33234/119781)
Branch Coverage: 24.38% (17041/69910)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8b21d84edf008ef7e0a215686e295f36f7f2b153_8b21d84edf008ef7e0a215686e295f36f7f2b153/report/index.html

doris-robot · 2023-09-18T13:48:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.1 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162433318 Bytes

CalvinKirs · 2023-09-19T02:35:21Z

run p1

CalvinKirs · 2023-09-19T04:00:30Z

run p1

yiguolei

LGTM

github-actions · 2023-09-19T04:04:20Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-19T04:04:22Z

PR approved by anyone and no changes requested.

CalvinKirs · 2023-09-19T06:32:06Z

run buildall

doris-robot · 2023-09-19T06:56:15Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7981/21667)
Line Coverage: 28.86% (64050/221940)
Region Coverage: 27.75% (33243/119779)
Branch Coverage: 24.37% (17038/69910)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43974da97822d9038c483e29065df7c0ea9bd62f_43974da97822d9038c483e29065df7c0ea9bd62f/report/index.html

doris-robot · 2023-09-19T07:02:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.84 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162435940 Bytes

yiguolei

LGTM

github-actions · 2023-09-19T08:07:07Z

PR approved by at least one committer and no changes requested.

CalvinKirs · 2023-09-19T08:29:48Z

run p0

CalvinKirs · 2023-09-19T08:30:00Z

run buildall

doris-robot · 2023-09-19T08:57:15Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7979/21667)
Line Coverage: 28.85% (64026/221940)
Region Coverage: 27.74% (33228/119779)
Branch Coverage: 24.36% (17030/69910)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43974da97822d9038c483e29065df7c0ea9bd62f_43974da97822d9038c483e29065df7c0ea9bd62f/report/index.html

doris-robot · 2023-09-19T09:07:18Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162504059 Bytes

yiguolei · 2023-09-19T11:09:45Z

run buildall

doris-robot · 2023-09-19T11:25:44Z

TeamCity be ut coverage result:
Function Coverage: 36.76% (7981/21713)
Line Coverage: 28.83% (64061/222192)
Region Coverage: 27.73% (33255/119938)
Branch Coverage: 24.36% (17047/69992)
Coverage Report: http://coverage.selectdb-in.cc/coverage/083b05582a916cdc4c631d41154c436b06dc8b7d_083b05582a916cdc4c631d41154c436b06dc8b7d/report/index.html

doris-robot · 2023-09-19T11:41:57Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.49 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162238576 Bytes

xinyiZzz

LGTM

morningman

LGTM
please update doc in next PR

- sh start_fe/start_be --console is used to instruct the program to run in console mode. - sh start_fe/start_be --daemon is used to instruct the program to run in daemon mode. - sh start_fe/start_be used starts as a background execution, records output and error logs to the specified file

xiaokang · 2023-09-20T10:30:52Z

add some document @CalvinKirs

- sh start_fe/start_be --console is used to instruct the program to run in console mode. - sh start_fe/start_be --daemon is used to instruct the program to run in daemon mode. - sh start_fe/start_be used starts as a background execution, records output and error logs to the specified file

Followup #35679 and #24556 # Background Previously, the cloud logs were written to files. The main cloud logs include meta_service.INFO and meta_service.WARNING, doris_cloud.out. In a K8s deployment environment, logs usually must be output to standard output, and other components process the log stream. # Solution This PR made the following changes: Modified the glog config: - When started with --daemon, logs are still written to various files, and the format remains unchanged. - When started with --console, meta_service.INFO's log is output to standard output and marked with the prefix RuntimeLogger. Examples are as follows: ``` RuntimeLogger I20240605 23:41:20.426553 4137369 runtime_query_statistics_mgr.cpp:245] Report profile thread stopped ``` Added a new cloud config: `enable_file_logger`, which defaults to true. This indicates that logs will be recorded in files regardless of the startup method. For example, if it is started with `--console`, the log will be output to both the file and the standard output. If it is false, the log will not be recorded in the file regardless of the startup method.

[Improve](start-shell)Optimize fe&be startup

8b21d84

Added startup --console option Modify the original default startup input stream to avoid log loss

CalvinKirs force-pushed the run-console branch from 63fe2e7 to 8b21d84 Compare September 18, 2023 13:05

yiguolei previously approved these changes Sep 19, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

github-actions bot added the reviewed label Sep 19, 2023

CalvinKirs closed this Sep 19, 2023

fix script error

43974da

CalvinKirs reopened this Sep 19, 2023

CalvinKirs dismissed yiguolei’s stale review via 43974da September 19, 2023 06:30

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

yiguolei approved these changes Sep 19, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

Merge branch 'master' into run-console

083b055

xinyiZzz approved these changes Sep 19, 2023

View reviewed changes

morningman approved these changes Sep 19, 2023

View reviewed changes

morningman merged commit 1a553f7 into apache:master Sep 19, 2023

yiguolei added the dev/2.0.2 label Sep 20, 2023

xiaokang added the usercase Important user case type label label Sep 20, 2023

xiaokang added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 20, 2023

CalvinKirs deleted the run-console branch September 29, 2023 13:52

gavinchou mentioned this pull request Aug 19, 2024

[opt](log) refine the cloud logger #39488

Merged

[Improve](start-shell)Optimize fe&be startup #24556

[Improve](start-shell)Optimize fe&be startup #24556

Uh oh!

Conversation

CalvinKirs commented Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actions bot commented Sep 18, 2023

sh-checker report

Uh oh!

CalvinKirs commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

CalvinKirs commented Sep 19, 2023

Uh oh!

CalvinKirs commented Sep 19, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 19, 2023

Uh oh!

github-actions bot commented Sep 19, 2023

Uh oh!

CalvinKirs commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 19, 2023

Uh oh!

CalvinKirs commented Sep 19, 2023

Uh oh!

CalvinKirs commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

yiguolei commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

xinyiZzz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

xiaokang commented Sep 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CalvinKirs commented Sep 18, 2023 •

edited

Loading

`sh-checker report`

xinyiZzz left a comment •

edited

Loading