[Feature](Variant) support variant sparse feature and schema template with multi indexes #51623

eldenmoon · 2025-06-10T16:28:43Z

What problem does this PR solve?

Introduced variant sparse columns
Implement Variant with schema template feature
Implement multi inverted index feature

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-06-10T16:28:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2025-06-11T10:37:54Z

run buildall

eldenmoon · 2025-06-11T11:07:48Z

run buildall

hello-stephen · 2025-06-11T11:30:48Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.32% (1119/1343)
Line Coverage	66.77% (19184/28732)
Region Coverage	66.45% (9504/14302)
Branch Coverage	56.35% (5148/9136)

eldenmoon · 2025-06-11T14:22:04Z

run buildall

doris-robot · 2025-06-11T14:38:59Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.32% (1119/1343)
Line Coverage	66.81% (19196/28732)
Region Coverage	66.44% (9502/14302)
Branch Coverage	56.41% (5154/9136)

eldenmoon · 2025-06-12T04:01:38Z

run buildall

hello-stephen · 2025-06-12T04:19:54Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.32% (1119/1343)
Line Coverage	66.80% (19192/28732)
Region Coverage	66.49% (9509/14302)
Branch Coverage	56.40% (5153/9136)

eldenmoon · 2025-06-13T03:06:15Z

run buildall

hello-stephen · 2025-06-13T03:34:25Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.33% (1120/1344)
Line Coverage	66.85% (19325/28909)
Region Coverage	66.60% (9584/14390)
Branch Coverage	56.52% (5209/9216)

…ma template with multi indexes #51623 (#52172) Cherry-pick from #51623

eldenmoon · 2025-07-23T15:19:27Z

run buildall

doris-robot · 2025-07-23T15:38:37Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.42% (1302/1619)
Line Coverage	65.77% (21797/33140)
Region Coverage	67.04% (10946/16328)
Branch Coverage	56.59% (5759/10176)

zzzxl1993

LGTM

github-actions · 2025-07-24T03:21:03Z

PR approved by anyone and no changes requested.

yiguolei · 2025-07-26T08:58:25Z

be/src/agent/be_exec_version_manager.cpp

 */

-const int BeExecVersionManager::max_be_exec_version = 8;
+const int BeExecVersionManager::max_be_exec_version = 9;


没必要有这个，直接做不兼容的改动就好了，如果有这个，我们说明有一些遗留的代码，是可以删除的

yiguolei · 2025-07-26T09:29:03Z

be/src/olap/key_coder.cpp

        add_mapping<FieldType::OLAP_FIELD_TYPE_DECIMAL256>();
        add_mapping<FieldType::OLAP_FIELD_TYPE_IPV4>();
        add_mapping<FieldType::OLAP_FIELD_TYPE_IPV6>();
+        add_mapping<FieldType::OLAP_FIELD_TYPE_FLOAT>();


为什么要改这个？

yiguolei · 2025-07-26T09:32:11Z

be/src/olap/rowset/beta_rowset.cpp

-                if (!st.ok()) {
-                    return st;
-                }
+                auto index_metes = _rowset_meta->tablet_schema()->inverted_indexs(*column);


typo error？

yiguolei · 2025-07-26T09:32:11Z

be/src/olap/rowset/beta_rowset.cpp

-                auto status = process_files(*index_meta, indices, index);
-                if (!status.ok()) {
-                    return status;
+                    auto status = process_files(*index_meta, indices, index);


return if error

yiguolei · 2025-07-26T09:32:36Z

be/src/olap/rowset/beta_rowset_writer.cpp

    _rowset_meta->set_tablet_uid(_context.tablet_uid);
-    _rowset_meta->set_tablet_schema(_context.tablet_schema);
+    auto schema = _context.tablet_schema->need_record_variant_extended_schema()
+                          ? _context.tablet_schema


这些地方得有一些注释

yiguolei · 2025-07-26T09:39:02Z

be/src/olap/rowset/segment_creator.cpp


 bool SegmentFlusher::need_buffering() {
    // buffering variants for schema change
    return _context.write_type == DataWriteType::TYPE_SCHEMA_CHANGE &&


variant 支持啥schema change啊？

yiguolei · 2025-07-26T09:41:50Z

be/src/olap/rowset/segment_v2/column_reader.cpp

-    //bool has_null = true;
-    //RETURN_IF_ERROR(index_file_reader->has_null(index_meta, &has_null));
-    //_inverted_index->set_has_null(has_null);
+    _index_readers[index_meta->index_id()] = index_reader;


我们当前是每一个column reader 里缓存这个column 相关的所有的index readers 吗？

yiguolei · 2025-07-26T09:44:19Z

be/src/olap/rowset/segment_v2/column_writer.cpp

+                opts, column, std::unique_ptr<Field>(FieldFactory::create(*column))));
+        return Status::OK();
+    }
+    *writer = std::unique_ptr<ColumnWriter>(new VariantColumnWriter(


std::make_unique

yiguolei · 2025-07-26T09:44:58Z

be/src/olap/rowset/segment_v2/column_writer.cpp

+                // Use ScalarColumnWriter to write it's only root data
+                std::unique_ptr<ColumnWriter> writer_local = std::unique_ptr<ColumnWriter>(
+                        new ScalarColumnWriter(opts, std::move(field), file_writer));
+                *writer = std::move(writer_local);


std::make_unique

yiguolei · 2025-07-26T09:45:57Z

be/src/olap/rowset/segment_v2/column_writer.cpp

+                // Use ScalarColumnWriter to write it's only root data
+                std::unique_ptr<ColumnWriter> writer_local = std::unique_ptr<ColumnWriter>(
+                        new ScalarColumnWriter(opts, std::move(field), file_writer));
+                *writer = std::move(writer_local);


std::make_unique

yiguolei · 2025-07-26T10:02:14Z

be/src/olap/tablet_schema.h

    const std::vector<TabletColumnPtr>& sparse_columns() const;
    size_t num_sparse_columns() const { return _num_sparse_columns; }

+    void set_precision_frac(int32_t precision, int32_t frac) {


为什么会有set 方法？这个不应该是从元数据中读取出来是啥就是啥吗？

yiguolei · 2025-07-26T10:04:07Z

be/src/olap/tablet_schema.h

    const std::vector<TabletColumnPtr>& sparse_columns() const;
    size_t num_sparse_columns() const { return _num_sparse_columns; }

+    void set_precision_frac(int32_t precision, int32_t frac) {


为什么会有set 方法？这个不应该是从元数据中读取出来是啥就是啥吗？

yiguolei · 2025-07-26T10:04:49Z

be/src/olap/rowset/segment_v2/variant/hierarchical_data_iterator.h

+    vectorized::PathInData path;
+    vectorized::ColumnPtr column;
+    vectorized::DataTypePtr type;
+};


所有的新文件，默认都要加上编译检查

yiguolei · 2025-07-26T10:24:44Z

be/src/olap/tablet_schema.h

        return Status::OK();
    }

+    void set_variant_max_subcolumns_count(int32_t variant_max_subcolumns_count) {


为什么要提供set 方法

yiguolei · 2025-07-26T10:47:08Z

be/src/vec/common/string_buffer.hpp

+  */
+#define writeCString(s, buf) (buf).write((s), strlen(s))
+
+inline void writeJSONString(const char* begin, const char* end, BufferWritable& buf) {


这个方法应该是bufferwriteable的成员函数

yiguolei · 2025-07-26T10:54:29Z

be/src/vec/common/string_buffer.hpp

+}
+
+/** Writes a C-string without creating a temporary object. If the string is a literal, then `strlen` is executed at the compilation stage.
+  * Use when the string is a literal.


这个代码需要单独看看！！

yiguolei · 2025-07-26T10:57:40Z

be/src/vec/functions/round.h

            return col_res;
        } else {
-            static_assert(false);
+            // static_assert(false);


为啥注释了？

yiguolei · 2025-07-26T10:58:59Z

gensrc/proto/data.proto

@@ -66,6 +66,7 @@ message PColumnMeta {
    optional string function_name = 7;
    optional int32 be_exec_version = 8;
    optional segment_v2.ColumnPathInfo column_path = 9;
+    optional int32 variant_max_subcolumns_count = 10 [default = 0];


default 0 是啥含义？

yiguolei · 2025-07-26T10:59:28Z

gensrc/proto/olap_file.proto

    // this field is only used during flexible partial update load
-    optional bool is_on_update_current_timestamp = 25 [default = false];
+    optional bool is_on_update_current_timestamp = 28 [default = false];


你这改一下顺序，是不是启动的时候存储就错了啊？

eldenmoon marked this pull request as draft June 10, 2025 16:28

eldenmoon force-pushed the variant-sparse-merge branch 2 times, most recently from b0f6c1e to ac71c3d Compare June 10, 2025 16:33

eldenmoon force-pushed the variant-sparse-merge branch from 84ceebf to 808f525 Compare June 11, 2025 11:07

eldenmoon added the dev/3.1.x label Jun 18, 2025

eldenmoon changed the title ~~Variant sparse merge~~ [Feature](Variant) support variant sparse feature and schema template with multi indexes Jun 25, 2025

eldenmoon marked this pull request as ready for review June 26, 2025 03:42

eldenmoon requested review from BiteTheDDDDt, CalvinKirs, dataroaring, gavinchou, morningman and yiguolei as code owners June 26, 2025 03:42

morrySnow pushed a commit that referenced this pull request Jun 26, 2025

Branch 3.1 [Feature](Variant) support variant sparse feature and sche…

c302687

…ma template with multi indexes #51623 (#52172) Cherry-pick from #51623

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jun 26, 2025

eldenmoon force-pushed the variant-sparse-merge branch 3 times, most recently from 852cdbc to c46cc63 Compare July 23, 2025 12:20

[feature](semi-structure) support variant and index with many features

86f19be

eldenmoon force-pushed the variant-sparse-merge branch from 953d0c0 to 2a7fc31 Compare July 23, 2025 15:19

zzzxl1993 approved these changes Jul 24, 2025

View reviewed changes

github-actions bot added the reviewed label Jul 24, 2025

yiguolei reviewed Jul 26, 2025

View reviewed changes

be/src/vec/functions/round.h

return col_res;

} else {

static_assert(false);

// static_assert(false);

Copy link

Contributor

yiguolei Jul 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为啥注释了？

yiguolei reviewed Jul 26, 2025

View reviewed changes

pick some PRs from branch-3.1 (#53975)

c268637

eldenmoon requested a review from w41ter as a code owner July 29, 2025 04:03

eldenmoon force-pushed the variant-sparse-merge branch from e53d779 to c268637 Compare July 29, 2025 06:38

[enhance](variant) typed_paths_to sparse column (#54078)

2d718b7

eldenmoon closed this Aug 26, 2025

[Feature](Variant) support variant sparse feature and schema template with multi indexes #51623

[Feature](Variant) support variant sparse feature and schema template with multi indexes #51623

Uh oh!

Conversation

eldenmoon commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jun 10, 2025

Uh oh!

eldenmoon commented Jun 11, 2025

Uh oh!

eldenmoon commented Jun 11, 2025

Uh oh!

hello-stephen commented Jun 11, 2025

Cloud UT Coverage Report

Uh oh!

eldenmoon commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Cloud UT Coverage Report

Uh oh!

eldenmoon commented Jun 12, 2025

Uh oh!

hello-stephen commented Jun 12, 2025

Cloud UT Coverage Report

Uh oh!

eldenmoon commented Jun 13, 2025

Uh oh!

hello-stephen commented Jun 13, 2025

Cloud UT Coverage Report

Uh oh!

eldenmoon commented Jul 23, 2025

Uh oh!

doris-robot commented Jul 23, 2025

Cloud UT Coverage Report

Uh oh!

zzzxl1993 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

eldenmoon commented Jun 10, 2025 •

edited

Loading