Skip to content

fix #9351 can't load parquet file with column name case sensitive with Doris column #9358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morningman merged 4 commits into from
May 12, 2022
Merged

fix #9351 can't load parquet file with column name case sensitive with Doris column #9358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ad80db5
fix #9351 can't load parquet file with column name case sensitive wit…
deardeng May 3, 2022
9fb63e7
#9351 add some comments
May 7, 2022
ddad03a
#9351 add some comments
May 7, 2022
fd52520
#9351 fix reviewed
deardeng May 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions fe/fe-core/src/main/java/org/apache/doris/load/Load.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1049,8 +1049,12 @@ private static void initColumns(Table tbl, List<ImportColumnDesc> columnExprs,
for (ImportColumnDesc importColumnDesc : copiedColumnExprs) {
// make column name case match with real column name
String columnName = importColumnDesc.getColumnName();
String realColName = tbl.getColumn(columnName) == null ? columnName
: tbl.getColumn(columnName).getName();
String realColName;
if (tbl.getColumn(columnName) == null || importColumnDesc.getExpr() == null) {
realColName = columnName;
} else {
realColName = tbl.getColumn(columnName).getName();
}
if (importColumnDesc.getExpr() != null) {
Expr expr = transformHadoopFunctionExpr(tbl, realColName, importColumnDesc.getExpr());
exprsByName.put(realColName, expr);
Expand Down
4 changes: 2 additions & 2 deletions fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,8 +247,8 @@ private void initParams(ParamCreateContext context)
*/
private void initColumns(ParamCreateContext context) throws UserException {
context.srcTupleDescriptor = analyzer.getDescTbl().createTupleDescriptor();
context.slotDescByName = Maps.newHashMap();
context.exprMap = Maps.newHashMap();
context.slotDescByName = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);
context.exprMap = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);

// for load job, column exprs is got from file group
// for query, there is no column exprs, they will be got from table's schema in "Load.initColumns"
Expand Down
7 changes: 5 additions & 2 deletions fe/fe-core/src/main/java/org/apache/doris/planner/StreamLoadScanNode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,11 @@ public class StreamLoadScanNode extends LoadScanNode {
private TupleDescriptor srcTupleDesc;
private TBrokerScanRange brokerScanRange;

private Map<String, SlotDescriptor> slotDescByName = Maps.newHashMap();
private Map<String, Expr> exprsByName = Maps.newHashMap();
// If use case sensitive map, for example,
// the column name 「A」 in the table and the mapping '(a) set (A = a)' in load sql，
// Slotdescbyname stores「a」, later will use 「a」to get table's 「A」 column info, will throw exception.
private final Map<String, SlotDescriptor> slotDescByName = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);
private final Map<String, Expr> exprsByName = Maps.newTreeMap(String.CASE_INSENSITIVE_ORDER);

// used to construct for streaming loading
public StreamLoadScanNode(
Expand Down