[feature-wip](parquet-vec) Support parquet scanner in vectorized engine #9231

yinzhijian · 2022-04-26T07:48:33Z

Proposed changes

Issue Number: close #xxx

Problem Summary:

Optimization Todo list：

FE generates the corresponding src slot desc and Expr through the parquet schema
BE supports direct conversion of arrow type into dest primitive type of similar type. For example, arrow type is INT32, and dest type is TYPE_BIGINT (int64), INT32=>TYPE_BIGINT. Instead of the current way: INT32=> TYPE_INT => TYPE_BIGINT

Performance Testing:

load parquet file in vec version almost 1x faster than rowset version.
rows num:300k
test table schema:
CREATE TABLE parquet (
id int(11) NOT NULL COMMENT "",
email varchar(26) NOT NULL COMMENT "",
c_date32 DATE NOT NULL COMMENT "",
c_date64 DATETIME NOT NULL COMMENT "",
c_timestamp DATETIME NOT NULL COMMENT "",
c_decimal128 DECIMAL(27, 9) NULL COMMENT "",
c_bool BOOLEAN NULL COMMENT "",
c_float FLOAT NULL COMMENT "",
c_double DOUBLE NULL COMMENT "",
c_fixed_size_binary CHAR(20) NULL COMMENT "",
c_binary VARCHAR(32) NULL COMMENT "",
c_uint64 BIGINT NULL COMMENT ""
)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
"replication_num" = "1"
);

Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

be/src/vec/utils/arrow_column_to_doris_column.h

be/src/http/action/stream_load.cpp

be/src/vec/exec/vparquet_scanner.cpp

…-doris into vec-parquet

HappenLee · 2022-05-05T02:14:43Z

be/src/vec/CMakeLists.txt

  runtime/vdata_stream_mgr.cpp
  runtime/vpartition_info.cpp
+  runtime/vsorted_run_merger.cpp
+  runtime/vload_channel.cpp


the code not add in this PR？

HappenLee · 2022-05-05T02:18:11Z

be/src/vec/utils/arrow_column_to_doris_column.cpp

+#define FOR_ARROW_TYPES(M) \
+    M(::arrow::Type::BOOL, TYPE_BOOLEAN) \
+    M(::arrow::Type::INT8, TYPE_TINYINT) \
+    M(::arrow::Type::UINT8, TYPE_TINYINT) \


UINT8 bigger than tinyint ？may overflow

HappenLee · 2022-05-05T02:22:38Z

be/src/vec/utils/arrow_column_to_doris_column.cpp

+    PaddedPODArray<UInt8> & column_chars_t = assert_cast<ColumnString &>(*data_column).get_chars();
+    PaddedPODArray<UInt32> & column_offsets = assert_cast<ColumnString &>(*data_column).get_offsets();
+
+    const auto & concrete_array = dynamic_cast<const arrow::BinaryArray &>(*array);


code format： auto&. same to other.

HappenLee · 2022-05-05T02:28:34Z

be/src/vec/utils/arrow_column_to_doris_column.cpp

+    PaddedPODArray<UInt8> & column_chars_t = assert_cast<ColumnString &>(*data_column).get_chars();
+    PaddedPODArray<UInt32> & column_offsets = assert_cast<ColumnString &>(*data_column).get_offsets();
+
+    const auto & concrete_array = dynamic_cast<const arrow::BinaryArray &>(*array);


Rethink here really need dynamic_cast ?

dynamic_cast means bad design of class or code, replace with virtual function

this code may throw exception if cast faild. it is danger here.

HappenLee · 2022-05-05T02:42:55Z

be/src/vec/utils/arrow_column_to_doris_column.cpp

+    return Status::OK();
+}
+
+Status arrow_column_to_doris_column(const arrow::Array* arrow_column,


Add UT for this class

HappenLee · 2022-05-05T02:55:30Z

be/src/vec/exec/vparquet_scanner.cpp

+    // init arrow batch
+    {
+        Status st = init_arrow_batch_if_necessary();
+        if (!st.ok()) {


the code is weird? rethink the logic?

HappenLee · 2022-05-05T02:57:21Z

be/src/vec/exec/vparquet_scanner.cpp

+
+// eval conjuncts, for example: t1 > 1
+Status VParquetScanner::eval_conjunts(Block* block) {
+    for (auto& vctx : _pre_filter_vctxs) {


may should change _pre_filter_vctxs to a tree to avoid the each time of copy operation

yinzhijian · 2022-05-07T07:57:34Z

close this pr, use this #9433

[feature-wip](parquet-vec) Support parquet scanner in vectorized engine

7d346e9

github-actions bot added area/sql/function Issues or PRs related to the SQL functions area/vectorization labels Apr 26, 2022