Skip to content

[fix](outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #1370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BePPPower wants to merge 2 commits into from
Closed

[fix](outfile) Fix the data type mapping for complex types in Doris to the ORC and Parquet file formats. #1370

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
19c07e9
fix 1
BePPPower Nov 18, 2024
f52e442
fix 2
BePPPower Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs/data-operate/export/export-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,11 @@ The following table shows the mapping between Doris data types and Parquet, ORC
|struct|struct|
|map|map|
|array|array|
|json| Not supported|
|json | string|
|variant | string|
|bitmap | binary | binary
|quantile_state | binary | binary
|hll | binary | binary

2. When Doris exports to Parquet file format, it first converts Doris in-memory data to Arrow in-memory data format, then writes out to Parquet file format. The mapping relationship between Doris data types and Arrow data types is:

Expand All @@ -129,4 +133,8 @@ The following table shows the mapping between Doris data types and Parquet, ORC
| struct | struct |
| map | map |
| array | list |
|json| utf8 |
| json | utf8 |
| variant | utf8|
| bitmap | binary | binary
| quantile_state | binary | binary
| hll | binary | binary
10 changes: 9 additions & 1 deletion ...N/docusaurus-plugin-content-docs/current/data-operate/export/export-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,11 @@ Parquet、ORC 文件格式拥有自己的数据类型。Apache Doris 的导出
|struct|struct|
|map|map|
|array|array|
|json|不支持|
|json|string|
|variant | string|
|bitmap | binary | binary
|quantile_state | binary | binary
|hll | binary | binary


2. Apache Doris 导出到 Parquet 文件格式时，会先将 Apache Doris 内存数据转换为 Arrow 内存数据格式，然后由 Arrow 写出到 Parquet 文件格式。Apache Doris 数据类型到 Arrow 数据类的映射关系为：
Expand All @@ -131,3 +135,7 @@ Parquet、ORC 文件格式拥有自己的数据类型。Apache Doris 的导出
|map|map|
|array|list|
|json|utf8|
|variant | utf8 |
|bitmap | binary | binary
|quantile_state | binary | binary
|hll | binary | binary