[Feature][Config] Support persistence of configuration items modified at runtime #4704

morningman · 2020-10-07T07:10:16Z

Proposed changes

Support persistence of configuration items modified at runtime via HTTP API.

FE:
GET /api/_set_config?key=value&persist=true

BE
POST /api/update_config?key=value&persist=true

The modified config will be saved in fe_custom.conf or be_custom.conf.
And when process starts, it will load fe.conf/be.conf first, then fe_custom.conf/be_custom.conf.

Types of changes

New feature (non-breaking change which adds functionality)

Checklist

I have create an issue on (Fix [Proposal] Support persistence of configuration items modified at runtime #4703), and have described the bug/feature there in detail
If this change need a document change, I have updated the document

acelyc111 · 2020-10-09T02:30:57Z

be/src/common/configbase.cpp

+    std::ofstream out(conffile);
+    out << "# THIS IS AN AUTO GENERATED CONFIG FILE.\n";
+    out << "# You can modify this file manually, and the configurations in this file\n";
+    out << "# will overwrite the configurations in fe.conf\n";


Suggested change

out << "# will overwrite the configurations in fe.conf\n";

out << "# will overwrite the configurations in be.conf\n";

acelyc111 · 2020-10-09T02:39:29Z

be/src/http/action/update_config_action.cpp

+    // So the number of query params should at most be 2.
+    if (req->params()->size() > 2 || req->params()->size() < 1) {
        s = Status::InvalidArgument("");
        msg = "Now only support to set a single config once, via 'config_name=new_value'";


I think you should also describe the optional parameter persist in error message.

acelyc111 · 2020-10-09T02:39:54Z

be/src/http/action/update_config_action.cpp

+        } else if (req->params()->size() == 2) {
+            if (req->params()->find(PERSIST_PARAM) == req->params()->end()) {
+                s = Status::InvalidArgument("");
+                msg = "Now only support to set a single config once, via 'config_name=new_value'";


acelyc111 · 2020-10-09T02:41:40Z

docs/en/administrator-guide/config/be_config.md


 This document mainly introduces the relevant configuration items of BE.

+The BE configuration file `be.conf` is usually stored in the `conf/` directory of the BE deployment path. In version 0.14, another configuration file `be_custom.conf` will be introduced. The configuration file is used to record the configuration items that are dynamically configured and persisted by the user during operation.


... is used for recording ... ?

This is translated by Google Translator... So, I think it is correct ^^

acelyc111 · 2020-10-09T02:43:19Z

docs/en/administrator-guide/config/be_config.md


-         **Configuration items modified in this way will become invalid after the BE process restarts. **
+    ```
+    curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persis=true'


Suggested change

curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persis=true'

curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persist=true'

xy720 · 2020-10-09T13:04:33Z

docs/zh-CN/administrator-guide/config/be_config.md

+	在 0.13 版本及之前，通过该方式修改的配置项将在 BE 进程重启后失效。在 0.14 及之后版本中，可以通过以下命令持久化修改后的配置。修改后的配置项存储在 `be_custom.conf` 文件中。
+
+	```
+	curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persis=true'


Suggested change

curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persis=true'

curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persist=true'

xy720 · 2020-10-09T13:08:23Z

fe/fe-core/src/main/java/org/apache/doris/common/ConfigBase.java

+        }
+
+        try (FileOutputStream fos = new FileOutputStream(file)) {
+            props.store(fos, "THIS IS AN AUTO GENERATED CONFIG FILE. DO NOT EDIT IT!\n\n" +


I think users can edit this file if they need to.

yangzhg

+1

yangzhg

+1

morningman added kind/feature Categorizes issue or PR as related to a new feature. area/config Issues or PRs related to configuration labels Oct 7, 2020

morningman self-assigned this Oct 7, 2020

morningman force-pushed the persist_config branch from 19f2256 to 02e5de2 Compare October 8, 2020 13:58

acelyc111 reviewed Oct 9, 2020

View reviewed changes

xy720 reviewed Oct 9, 2020

View reviewed changes

morningman force-pushed the persist_config branch from a0393d9 to 330e656 Compare October 12, 2020 04:29

yangzhg previously approved these changes Oct 20, 2020

View reviewed changes

morningman added the approved Indicates a PR has been approved by one committer. label Oct 20, 2020

morningman and others added 6 commits October 20, 2020 11:59

first

a67bb89

second

f43137e

be config

3e81d43

doc

e976d46

fix ut

ebf4216

fix by review

4c71bf5

morningman dismissed yangzhg’s stale review via 4c71bf5 October 20, 2020 04:07

morningman force-pushed the persist_config branch from 330e656 to 4c71bf5 Compare October 20, 2020 04:07

morningman and others added 2 commits October 20, 2020 12:24

add custom config dir config

3c9190c

add custom_conf_dir config

88a5b19

morningman requested a review from yangzhg October 20, 2020 16:02

yangzhg approved these changes Oct 21, 2020

View reviewed changes

morningman merged commit 0213e93 into apache:master Oct 22, 2020

yangzhg mentioned this pull request Feb 9, 2021

Release Notes 0.14.0 #5374

Closed

Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Dec 29, 2025

[enhance](mtls)support thrift reload certificate (apache#4704)

543f4a6

	out << "# will overwrite the configurations in fe.conf\n";
	out << "# will overwrite the configurations in be.conf\n";


		This document mainly introduces the relevant configuration items of BE.

		The BE configuration file `be.conf` is usually stored in the `conf/` directory of the BE deployment path. In version 0.14, another configuration file `be_custom.conf` will be introduced. The configuration file is used to record the configuration items that are dynamically configured and persisted by the user during operation.

	curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persis=true'
	curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persist=true'

[Feature][Config] Support persistence of configuration items modified at runtime #4704

[Feature][Config] Support persistence of configuration items modified at runtime #4704

Uh oh!

Conversation

morningman commented Oct 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Types of changes

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

morningman commented Oct 7, 2020 •

edited

Loading