Data export function, add export to specify certain columns by hf200012 · Pull Request #5689 · apache/doris

hf200012 · 2021-04-22T02:17:50Z

EXPORT TABLE db.tbl
TO "hdfs://namenode:8020/tmp/doris_20213"
PROPERTIES
(
"columns"="city_name,date",
"column_separator"=",",
"exec_mem_limit"="2147483648",
"timeout" = "3600"
)
WITH BROKER "broker_name_2"
(
"username" = "",
"password" = ""
);

The data export function adds a parameter “columns”, which is used to specify the column names in the export table, which can be multiple columns, separated by commas, and the column names are not case sensitive
If this parameter is not filled in, all columns of the table will be exported by default

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If these changes need document changes, I have updated the document
Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

… table Data export function, add certain columns that can be exported to the table

Modify the data export usage document

EmmyMiao87 · 2021-04-22T04:16:47Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java


 package org.apache.doris.analysis;

+import com.google.common.base.Splitter;


please pay attention to import sequence

EmmyMiao87 · 2021-04-22T04:51:13Z

fe/fe-core/src/main/java/org/apache/doris/analysis/LoadStmt.java

    public static final String KEY_IN_PARAM_BACKEND_ID = "backend_id";

+    //export
+    public static final String EXPORT_KEY_IN_PARAM_COLUMNS = "columns";


use KEY_IN_PARAM_COLUMNS instead

EmmyMiao87 · 2021-04-22T04:51:39Z

fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java

                properties, ExportStmt.DEFAULT_COLUMN_SEPARATOR));
        this.lineDelimiter = Separator.convertSeparator(PropertyAnalyzer.analyzeLineDelimiter(
                properties, ExportStmt.DEFAULT_LINE_DELIMITER));
+        if(properties.containsKey(LoadStmt.EXPORT_KEY_IN_PARAM_COLUMNS)){


code format

EmmyMiao87 · 2021-04-22T10:45:44Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

-            slot.setIsMaterialized(true);
-            slot.setColumn(col);
-            slot.setIsNullable(col.isAllowNull());
+            if(!this.exportColumns.isEmpty() && this.exportColumns.contains(col.getName().toLowerCase())) {


code format

EmmyMiao87 · 2021-04-22T10:46:00Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

        exportTupleDesc = desc.createTupleDescriptor();
        exportTupleDesc.setTable(exportTable);
        exportTupleDesc.setRef(tableRef);
+        this.exportColumns = stmt.getColumns();


please set it in public void setJob(ExportStmt stmt) throws UserException

EmmyMiao87 · 2021-04-22T10:46:19Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java


-    private void genExecFragment() throws UserException {
-        registerToDesc();
+    private void genExecFragment(ExportStmt stmt) throws UserException {


Suggested change

private void genExecFragment(ExportStmt stmt) throws UserException {

private void genExecFragment() throws UserException {

EmmyMiao87 · 2021-04-22T10:46:28Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

    }

-    private void registerToDesc() {
+    private void registerToDesc(ExportStmt stmt) {


Suggested change

private void registerToDesc(ExportStmt stmt) {

private void registerToDesc() {

EmmyMiao87 · 2021-04-22T10:46:36Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

            this.tableId = exportTable.getId();
            this.tableName = stmt.getTblName();
-            genExecFragment();
+            genExecFragment(stmt);


Suggested change

genExecFragment(stmt);

genExecFragment();

EmmyMiao87 · 2021-04-22T10:54:50Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

    private OriginStatement origStmt;
    protected Map<String, String> sessionVariables = Maps.newHashMap();

+    private List<String> exportColumns ;


If you store columns as a separate attribute in the export job from properties, you need to consider persistence.
Either reload the columns attribute during replay.
Or just persist the columns object directly.
My suggestion is not to modify the persistence logic. Re-parse columns after persistence.

Code format modify

EmmyMiao87 · 2021-04-22T11:36:18Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

        for (Column col : exportTable.getBaseSchema()) {
-            if(!this.exportColumns.isEmpty() && this.exportColumns.contains(col.getName().toLowerCase())) {
+            String colName = col.getName().toLowerCase();
+            if (!this.exportColumns.isEmpty() && this.exportColumns.contains(colName)) {


Suggested change

if (!this.exportColumns.isEmpty() && this.exportColumns.contains(colName)) {

if (this.exportColumns !=null && this.exportColumns.contains(colName)) {

There may be a null problem in the modification

columns persistence

EmmyMiao87 · 2021-04-23T03:04:14Z

docs/zh-CN/administrator-guide/export-manual.md

 ```

 * `column_separator`：列分隔符。默认为 `\t`。支持不可见字符，比如 '\x07'。
+* columns：要导出的列，使用英文状态逗号隔开，如果不填这个参数默认是导出表的所有列


可以把英文注释也加一下~

EmmyMiao87 · 2021-04-23T03:06:36Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

        Text.writeString(out, exportPath);
        Text.writeString(out, columnSeparator);
        Text.writeString(out, lineDelimiter);
+        Text.writeString(out, columns);


In fact, there is no need to modify the logic here. You only need to initialize the columns after reading the properties.
Also, even if columns are to be persisted, they cannot actually be placed in this position.

modify columns Persistence

EmmyMiao87 · 2021-04-23T09:11:32Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

-import org.apache.doris.common.Pair;
-import org.apache.doris.common.Status;
-import org.apache.doris.common.UserException;
+import org.apache.doris.common.*;


code style

remove import *

…to dev

EmmyMiao87 · 2021-04-26T02:29:13Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

                this.properties.put(propertyKey, propertyValue);
            }
        }
+        this.columns = this.properties.get(LoadStmt.KEY_IN_PARAM_COLUMNS);


Suggested change

this.columns = this.properties.get(LoadStmt.KEY_IN_PARAM_COLUMNS);

this.columns = this.properties.get(LoadStmt.KEY_IN_PARAM_COLUMNS);

if (!Strings.isNullOrEmpty(this.columns)) {

Splitter split = Splitter.on(',').trimResults().omitEmptyStrings();

this.exportColumns = split.splitToList(stmt.getColumns().toLowerCase());

}

modify readFields exportColumns

EmmyMiao87

LGTM

Data export function, add certain columns that can be exported to the table. Export stmt properties ("columns" = "k1, k2, k3");

## Proposed changes Issue Number: close #xxx

hf200012 added 2 commits April 22, 2021 10:12

Data export function, add certain columns that can be exported to the…

7ccff6a

… table Data export function, add certain columns that can be exported to the table

Modify the data export usage document

4a8ee26

Modify the data export usage document

EmmyMiao87 requested changes Apr 22, 2021

View reviewed changes

EmmyMiao87 reviewed Apr 22, 2021

View reviewed changes

Code format modify

87e38f9

Code format modify

EmmyMiao87 reviewed Apr 22, 2021

View reviewed changes

hf200012 added 2 commits April 22, 2021 19:40

There may be a null problem in the modification

6c73808

There may be a null problem in the modification

columns persistence

f0158a9

columns persistence

EmmyMiao87 reviewed Apr 23, 2021

View reviewed changes

modify columns Persistence

62f70a7

modify columns Persistence

EmmyMiao87 reviewed Apr 23, 2021

View reviewed changes

hf200012 added 3 commits April 23, 2021 17:14

code style

7e12a83

code style

remove import *

fa97865

remove import *

Merge branch 'master' of https://github.com/apache/incubator-doris in…

19b698b

…to dev

EmmyMiao87 self-assigned this Apr 26, 2021

EmmyMiao87 reviewed Apr 26, 2021

View reviewed changes

modify readFields exportColumns

f5afa19

modify readFields exportColumns

EmmyMiao87 approved these changes Apr 26, 2021

View reviewed changes

EmmyMiao87 added area/backup Issues of PRS related to backup and restore kind/improvement api-review Categorizes an issue or PR as actively needing an API review. approved Indicates a PR has been approved by one committer. labels Apr 26, 2021

EmmyMiao87 merged commit c7af83b into apache:master Apr 27, 2021

EmmyMiao87 mentioned this pull request May 7, 2021

[Bug-fix] Export specified column #5759

Merged

10 tasks

morningman mentioned this pull request Oct 10, 2021

Release note 0.15.0 #6806

Closed

yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Dec 30, 2025

[fix](case) external_table_p0/paimon (apache#5689)

046230d

## Proposed changes Issue Number: close #xxx


		package org.apache.doris.analysis;

		import com.google.common.base.Splitter;

	private void genExecFragment(ExportStmt stmt) throws UserException {
	private void genExecFragment() throws UserException {

	private void registerToDesc(ExportStmt stmt) {
	private void registerToDesc() {

	if (!this.exportColumns.isEmpty() && this.exportColumns.contains(colName)) {
	if (this.exportColumns !=null && this.exportColumns.contains(colName)) {

-        this.columns = this.properties.get(LoadStmt.KEY_IN_PARAM_COLUMNS);
+        this.columns = this.properties.get(LoadStmt.KEY_IN_PARAM_COLUMNS);
+       if (!Strings.isNullOrEmpty(this.columns)) {
+            Splitter split = Splitter.on(',').trimResults().omitEmptyStrings();
+            this.exportColumns = split.splitToList(stmt.getColumns().toLowerCase());
+        }

Conversation

hf200012 commented Apr 22, 2021

Proposed changes

Types of changes

Checklist

Further comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants