diff --git a/docs/.vuepress/public/images/image-20210618091006146.png b/docs/.vuepress/public/images/image-20210618091006146.png new file mode 100644 index 00000000000000..d70ae5b9866409 Binary files /dev/null and b/docs/.vuepress/public/images/image-20210618091006146.png differ diff --git a/docs/.vuepress/public/images/image-20210618095240216.png b/docs/.vuepress/public/images/image-20210618095240216.png new file mode 100644 index 00000000000000..54f548862c1430 Binary files /dev/null and b/docs/.vuepress/public/images/image-20210618095240216.png differ diff --git a/docs/.vuepress/public/images/image-20210618104042192.png b/docs/.vuepress/public/images/image-20210618104042192.png new file mode 100644 index 00000000000000..486e3c1766080e Binary files /dev/null and b/docs/.vuepress/public/images/image-20210618104042192.png differ diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js index 75cfe7148a9bbd..f2f1bb021ce388 100644 --- a/docs/.vuepress/sidebar/en.js +++ b/docs/.vuepress/sidebar/en.js @@ -541,6 +541,7 @@ module.exports = [ "debug-tool", "fe-eclipse-dev", "fe-idea-dev", + "be-vscode-dev", "format-code", ], }, diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js index cfbe0a4e7db8ce..a075a35707f342 100644 --- a/docs/.vuepress/sidebar/zh-CN.js +++ b/docs/.vuepress/sidebar/zh-CN.js @@ -545,7 +545,8 @@ module.exports = [ children: [ "debug-tool", "fe-eclipse-dev", - "fe-idea-dev", + "fe-idea-dev", + "be-vscode-dev", "format-code", ], }, diff --git a/docs/en/developer-guide/be-vscode-dev.md b/docs/en/developer-guide/be-vscode-dev.md new file mode 100644 index 00000000000000..a40aca25865bfc --- /dev/null +++ b/docs/en/developer-guide/be-vscode-dev.md @@ -0,0 +1,280 @@ +--- +{ + "title": "BE development and debugging environment under Linux", + "language": "en" +} +--- + + + +# Apache Doris Be development and debugging + +## initial preparation work + +**This tutorial was conducted under Ubuntu 20.04** + +1. Download the doris source code + + URL：[apache/incubator-doris: Apache Doris (Incubating) (github.com)](https://github.com/apache/incubator-doris) + +2. Install GCC 8.3.1+, Oracle JDK 1.8+, Python 2.7+, confirm that the gcc, java, python commands point to the correct version, and set the JAVA_HOME environment variable + +3. Install other dependent packages + +``` +sudo apt install build-essential openjdk-8-jdk maven cmake byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python brotli +sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa +sudo apt update +sudo apt install gcc-10 g++-10 +sudo apt-get install autoconf automake libtool autopoint +``` + +4. install : openssl-devel +``` +sudo apt install -y openssl-devel +``` + +## Compile + +The following steps are carried out in the /home/workspace directory + +1. dowload source + +``` +git clone https://github.com/apache/incubator-doris.git +``` + +2. Compile third-party dependency packages + +``` + cd /home/workspace/incubator-doris/thirdparty + ./build-thirdparty.sh +``` + +3. Compile doris product code + +``` +cd /home/workspace/incubator-doris +./build.sh +``` + +Note: This compilation has the following instructions: + +```shell +./build.sh #Compile be and fe at the same time +./build.sh --be #Only compile be +./build.sh --fe #Only compilefe +./build.sh --fe --be --clean#Delete and compile be fe at the same time +./build.sh --fe --clean#Delete and compile fe +./build.sh --be --clean#Delete and compile be +./build.sh --be --fe --clean#Delete and compile be fe at the same time +``` + +If nothing happens, the compilation should be successful, and the final deployment file will be output to the /home/zhangfeng/incubator-doris/output/ directory. If you still encounter other problems, you can refer to the doris installation document http://doris.apache.org. + +## Deployment and debugging + +1. Authorize be compilation result files + +``` +chmod /home/workspace/incubator-doris/output/be/lib/palo_be +``` + +Note: /home/workspace/incubator-doris/output/be/lib/palo_be is the executable file of be. + +2. Create a data storage directory + +By viewing /home/workspace/incubator-doris/output/be/conf/be.conf + +``` +# INFO, WARNING, ERROR, FATAL +sys_log_level = INFO +be_port = 9060 +be_rpc_port = 9070 +webserver_port = 8040 +heartbeat_service_port = 9050 +brpc_port = 8060 + +# Note that there should at most one ip match this list. +# If no ip match this rule, will choose one randomly. +# use CIDR format, e.g. 10.10.10.0/ +# Default value is empty. +priority_networks = 192.168.59.0/24 # data root path, seperate by ';' +storage_root_path = /soft/be/storage +# sys_log_dir = ${PALO_HOME}/log +# sys_log_roll_mode = SIZE-MB- +# sys_log_roll_num = +# sys_log_verbose_modules = +# log_buffer_level = - +# palo_cgroups +``` + +Need to create this folder, this is where the be data is stored + +``` +mkdir -p /soft/be/storage +``` + +3. Open vscode, and open the directory where the be source code is located. In this case, open the directory as **/home/workspace/incubator-doris/**，For details on how to vscode, refer to the online tutorial + +4. Install the vscode ms c++ debugging plug-in, the plug-in identified by the red box in the figure below + +![](/images/image-20210618104042192.png) + +5. Create a launch.json file, the content of the file is as follows: + +``` +{ + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "launch", + "program": "/home/workspace/incubator-doris/output/be/lib/palo_be", + "args": [], + "stopAtEntry": false, + "cwd": "/home/workspace/incubator-doris/", + "environment": [{"name":"PALO_HOME","value":"/home/workspace/incubator-doris/output/be/"}, + {"name":"UDF_RUNTIME_DIR","value":"/home/workspace/incubator-doris/output/be/lib/udf-runtime"}, + {"name":"LOG_DIR","value":"/home/workspace/incubator-doris/output/be/log"}, + {"name":"PID_DIR","value":"/home/workspace/incubator-doris/output/be/bin"} + ], + "externalConsole": true, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + } + ] +} +``` + +Among them, environment defines several environment variables DORIS_HOME UDF_RUNTIME_DIR LOG_DIR PID_DIR, which are the environment variables needed when palo_be is running. If it is not set, the startup will fail + +**Note: If you want attach (additional process) debugging, the configuration code is as follows:** + +``` +{ + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "attach", + "program": "/home/workspace/incubator-doris/output/lib/palo_be", + "processId":, + "MIMode": "gdb", + "internalConsoleOptions":"openOnSessionStart", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + } + ] +} +``` + +In the configuration **"request": "attach", "processId": PID**, these two configurations are the key points: set the debug mode of gdb to attach and attach the processId of the process, otherwise it will fail. To find the process id, you can enter the following command in the command line: + +``` +ps -ef | grep palo* +``` + +As shown in the figure: + +![](/images/image-20210618095240216.png) + +Among them, 15200 is the process id of the currently running be. + +An example of a complete lainch.json is as follows: + +``` + { + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Attach", + "type": "cppdbg", + "request": "attach", + "program": "/home/workspace/incubator-doris/output/be/lib/palo_be", + "processId": 17016, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + }, + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "launch", + "program": "/home/workspace/incubator-doris/output/be/lib/palo_be", + "args": [], + "stopAtEntry": false, + "cwd": "/home/workspace/incubator-doris/output/be", + "environment": [ + { + "name": "DORIS_HOME", + "value": "/home/workspace/incubator-doris/output/be" + }, + { + "name": "UDF_RUNTIME_DIR", + "value": "/home/workspace/incubator-doris/output/be/lib/udf-runtime" + }, + { + "name": "LOG_DIR", + "value": "/home/workspace/incubator-doris/output/be/log" + }, + { + "name": "PID_DIR", + "value": "/home/workspace/incubator-doris/output/be/bin" + } + ], + "externalConsole": false, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + } + ] +} +``` + +6. Click to debug + + You can start your debugging journey with the rest, + +![](/images/image-20210618091006146.png) + diff --git a/docs/en/installing/compilation.md b/docs/en/installing/compilation.md index 8dc4adee1e6261..0e09ba3bf48345 100644 --- a/docs/en/installing/compilation.md +++ b/docs/en/installing/compilation.md @@ -119,6 +119,7 @@ You can try to compile Doris directly in your own Linux environment. sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa sudo apt update sudo apt install gcc-10 g++-10 + sudo apt-get install autoconf automake libtool autopoint ``` If you are using CentOS you can use the following command to install the dependencies @@ -137,9 +138,9 @@ You can try to compile Doris directly in your own Linux environment. gpgcheck=1 enabled=1 ``` - After installation, set environment variables `PATH`, `JAVA_HOME`, etc. - Doris 0.14.0 will use gcc7 env to compile. - + After installation, set environment variables `PATH`, `JAVA_HOME`, etc. + Doris 0.14.0 will use gcc7 env to compile. + 2. Compile Doris ``` @@ -152,7 +153,6 @@ You can try to compile Doris directly in your own Linux environment. 1. `Could not transfer artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh from/to xxx` If you encounter the above error, please refer to [PR #4769](https://github.com/apache/incubator-doris/pull/4769/files) to modify the cloudera-related repo configuration in `fe/pom.xml`. - ## Special statement Starting from version 0.13, the dependency on the two third-party libraries [1] and [2] will be removed in the default compiled output. These two third-party libraries are under [GNU General Public License V3](https://www.gnu.org/licenses/gpl-3.0.en.html). This license is incompatible with [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), so it should not appear in the Apache release by default. diff --git a/docs/zh-CN/developer-guide/be-vscode-dev.md b/docs/zh-CN/developer-guide/be-vscode-dev.md new file mode 100644 index 00000000000000..d3e0086267e87a --- /dev/null +++ b/docs/zh-CN/developer-guide/be-vscode-dev.md @@ -0,0 +1,283 @@ +--- +{ + "title": "Doris BE开发调试环境 -- vscode", + "language": "zh-CN" +} + +--- + + + +# Apache Doris Be 开发调试 + +## 前期准备工作 + +**本教程是在 Ubuntu 20.04 下进行的** + +1. 下载 doris 源代码 + + 下载地址为：[apache/incubator-doris: Apache Doris (Incubating) (github.com)](https://github.com/apache/incubator-doris) + +2. 安装 GCC 8.3.1+，Oracle JDK 1.8+，Python 2.7+，确认 gcc, java, python 命令指向正确版本, 设置 JAVA_HOME 环境变量 + +3. 安装其他依赖包 + +``` +sudo apt install build-essential openjdk-8-jdk maven cmake byacc flex automake libtool-bin bison binutils-dev libiberty-dev zip unzip libncurses5-dev curl git ninja-build python brotli +sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa +sudo apt update +sudo apt install gcc-10 g++-10 +sudo apt-get install autoconf automake libtool autopoint +``` + +4. 安装 openssl-devel + +``` +sudo apt install -y openssl-devel +``` + +## 编译 + +以下操作步骤在 /home/workspace 目录下进行 + +1. 下载源码 + +``` +git clone https://github.com/apache/incubator-doris.git +``` + +2. 编译第三方依赖包 + +``` + cd /home/workspace/incubator-doris/thirdparty + ./build-thirdparty.sh +``` + +3. 编译doris产品代码 + +``` +cd /home/workspace/incubator-doris +./build.sh +``` + +注意：这个编译有以下几条指令： + +``` +./build.sh #同时编译be 和fe +./build.sh --be #只编译be +./build.sh --fe #只编译fe +./build.sh --fe --be#同时编译be fe +./build.sh --fe --be --clean#删除并同时编译be fe +./build.sh --fe --clean#删除并编译fe +./build.sh --be --clean#删除并编译be +./build.sh --be --fe --clean#删除并同时编译be fe +``` + +如果不出意外，应该会编译成功，最终的部署文件将产出到 /home/workspace/incubator-doris/output/ 目录下。如果还遇到其他问题，可以参照 doris 的安装文档 http://doris.apache.org。 + +## 部署调试 + +1. 给be编译结果文件授权 + +``` +chmod /home/workspace/incubator-doris/output/be/lib/palo_be +``` + +注意： /home/workspace/incubator-doris/output/be/lib/palo_be为be的执行文件。 + +2. 创建数据存放目录 + +通过查看/home/workspace/incubator-doris/output/be/conf/be.conf + +``` +# INFO, WARNING, ERROR, FATAL +sys_log_level = INFO +be_port = 9060 +be_rpc_port = 9070 +webserver_port = 8040 +heartbeat_service_port = 9050 +brpc_port = 8060 + +# Note that there should at most one ip match this list. +# If no ip match this rule, will choose one randomly. +# use CIDR format, e.g. 10.10.10.0/ +# Default value is empty. +priority_networks = 192.168.59.0/24 # data root path, seperate by ';' +storage_root_path = /soft/be/storage +# sys_log_dir = ${PALO_HOME}/log +# sys_log_roll_mode = SIZE-MB- +# sys_log_roll_num = +# sys_log_verbose_modules = +# log_buffer_level = - +# palo_cgroups +``` + +需要创建一个文件夹，这是be数据存放的地方 + +``` +mkdir -p /soft/be/storage +``` + +3. 打开 vscode，并打开 be 源码所在目录，在本案例中打开目录为 **/home/workspace/incubator-doris/** + +4. 安装 vscode ms c++ 调试插件 + +![](/images/image-20210618104004956.png) + +5. 创建 launch.json 文件，文件内容如下： + +``` +{ + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "launch", + "program": "/home/workspace/incubator-doris/output/be/lib/palo_be", + "args": [], + "stopAtEntry": false, + "cwd": "/home/workspace/incubator-doris/", + "environment": [{"name":"PALO_HOME","value":"/home/workspace/incubator-doris/output/be/"}, + {"name":"UDF_RUNTIME_DIR","value":"/home/workspace/incubator-doris/output/be/lib/udf-runtime"}, + {"name":"LOG_DIR","value":"/home/workspace/incubator-doris/output/be/log"}, + {"name":"PID_DIR","value":"/home/workspace/incubator-doris/output/be/bin"} + ], + "externalConsole": true, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + } + ] +} +``` + +其中，environment 定义了几个环境变量 DORIS_HOME UDF_RUNTIME_DIR LOG_DIR PID_DIR，这是 palo_be 运行时需要的环境变量，如果没有设置，启动会失败。 + +**注意：如果希望是attach(附加进程）调试，配置代码如下：** + +``` +{ + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "attach", + "program": "/home/workspace/incubator-doris/output/lib/palo_be", + "processId":, + "MIMode": "gdb", + "internalConsoleOptions":"openOnSessionStart", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + } + ] +} +``` + +配置中 **"request": "attach"， "processId":PID**，这两个配置是重点： 分别设置 gdb 的调试模式为 attach，附加进程的processId，否则会失败。如何查找进程id，可以在命令行中输入以下命令： + +``` +ps -ef | grep palo* +``` + +如图： + +![](/images/image-20210618095240216.png) + +其中的15200即为当前运行的 be 的进程 id. + +一个完整的 lainch.json 的例子如下： + +``` + { + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Attach", + "type": "cppdbg", + "request": "attach", + "program": "/home/workspace/incubator-doris/output/be/lib/palo_be", + "processId": 17016, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + }, + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "launch", + "program": "/home/workspace/incubator-doris/output/be/lib/palo_be", + "args": [], + "stopAtEntry": false, + "cwd": "/home/workspace/incubator-doris/output/be", + "environment": [ + { + "name": "DORIS_HOME", + "value": "/home/workspace/incubator-doris/output/be" + }, + { + "name": "UDF_RUNTIME_DIR", + "value": "/home/workspace/incubator-doris/output/be/lib/udf-runtime" + }, + { + "name": "LOG_DIR", + "value": "/home/workspace/incubator-doris/output/be/log" + }, + { + "name": "PID_DIR", + "value": "/home/workspace/incubator-doris/output/be/bin" + } + ], + "externalConsole": false, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + } + ] + } + ] +} +``` + +6. 点击调试即可 + + 下面就可以开始你的 Doris DEBUG 之旅了 + +![](/images/image-20210618091006146.png) + diff --git a/docs/zh-CN/installing/compilation.md b/docs/zh-CN/installing/compilation.md index 2c8ff52ebce4d1..e3e5f79bba8158 100644 --- a/docs/zh-CN/installing/compilation.md +++ b/docs/zh-CN/installing/compilation.md @@ -43,7 +43,6 @@ under the License. REPOSITORY TAG IMAGE ID CREATED SIZE apache/incubator-doris build-env-1.3 ca207367c09f 21 hours ago 3.28GB ``` - 注: 针对不同的 Doris 版本，需要下载对应的镜像版本 @@ -85,7 +84,6 @@ under the License. ``` 编译完成后，产出文件在 `output/` 目录中。 - ### 自行编译开发环境镜像 你也可以自己创建一个 Doris 开发环境镜像，具体可参阅 `docker/README.md` 文件。 @@ -96,7 +94,7 @@ under the License. 你可以在自己的 linux 环境中直接尝试编译 Doris。 1. 系统依赖 -不同的版本依赖也不相同 + 不同的版本依赖也不相同 * 在 [ad67dd3](https://github.com/apache/incubator-doris/commit/ad67dd34a04c1ca960cff38e5b335b30fc7d559f) 之前版本依赖如下： `GCC 7.3+, Oracle JDK 1.8+, Python 2.7+, Apache Maven 3.5+, CMake 3.11+ Bison 3.0+` @@ -119,8 +117,9 @@ under the License. sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa sudo apt update sudo apt install gcc-10 g++-10 + sudo apt-get install autoconf automake libtool autopoint ``` - + 如果是CentOS 可以执行以下命令 ``` sudo yum groupinstall 'Development Tools' && sudo yum install maven cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk @@ -137,7 +136,7 @@ under the License. gpgcheck=1 enabled=1 ``` - + 安装完成后，自行设置环境变量 `PATH`, `JAVA_HOME` 等。 注意： Doris 0.14.0 的版本仍然使用gcc7 的依赖编译，之后的代码将使用gcc10 的依赖 2. 编译 Doris