[Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 #37257

kaka11chen · 2024-07-04T02:22:10Z

Proposed changes

backport #37234

doris-robot · 2024-07-04T02:22:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-04T02:28:53Z

be/src/vec/exec/format/orc/orc_memory_pool.h

+
+#pragma once
+
+#include "orc/MemoryPool.hh"


warning: 'orc/MemoryPool.hh' file not found [clang-diagnostic-error]

#include "orc/MemoryPool.hh" ^

github-actions · 2024-07-04T02:28:54Z

be/src/vec/exec/format/orc/orc_memory_pool.h

+#endif // #if defined(USE_JEMALLOC) && defined(USE_MEM_TRACKER)
+    }
+
+    void free(char* p) override {


warning: pointer parameter 'p' can be pointer to const [readability-non-const-parameter]

Suggested change

void free(char* p) override {

void free(const char* p) override {

github-actions · 2024-07-04T02:28:54Z

be/src/vec/runtime/vorc_transformer.cpp

    set_compression_type(compress_type);
 }

+VOrcTransformer::~VOrcTransformer() {


warning: use '= default' to define a trivial destructor [modernize-use-equals-default]

VOrcTransformer::~VOrcTransformer() { ^

github-actions · 2024-07-04T02:28:54Z

be/src/vec/runtime/vorc_transformer.h

                    const iceberg::Schema* iceberg_schema = nullptr);

-    ~VOrcTransformer() = default;
+    ~VOrcTransformer();


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]

Suggested change

~VOrcTransformer();

~VOrcTransformer() override;

kaka11chen · 2024-07-04T05:49:13Z

run buildall

doris-robot · 2024-07-04T06:53:21Z

TeamCity be ut coverage result:
Function Coverage: 36.32% (9150/25194)
Line Coverage: 27.88% (74698/267971)
Region Coverage: 26.77% (38518/143874)
Branch Coverage: 23.47% (19520/83168)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9d35789ab994b03d9c4e892ca6ab5befaa2d0260_9d35789ab994b03d9c4e892ca6ab5befaa2d0260/report/index.html

zgxme · 2024-07-05T03:54:29Z

run buildall

doris-robot · 2024-07-05T04:58:56Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (9151/25199)
Line Coverage: 27.87% (74696/268060)
Region Coverage: 26.76% (38517/143945)
Branch Coverage: 23.45% (19518/83230)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb95ae7145370a6f5f37949e561d0a51e23dfe73_bb95ae7145370a6f5f37949e561d0a51e23dfe73/report/index.html

github-actions

clang-tidy made some suggestions

be/src/vec/common/allocator.cpp

be/src/vec/common/allocator.h

kaka11chen · 2024-07-09T07:22:05Z

run buildall

kaka11chen · 2024-07-09T08:10:58Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/common/allocator.cpp

kaka11chen · 2024-07-09T08:32:14Z

run buildall

kaka11chen · 2024-07-09T08:59:59Z

run buildall

doris-robot · 2024-07-09T10:24:07Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (9158/25219)
Line Coverage: 27.87% (74767/268237)
Region Coverage: 26.77% (38577/144101)
Branch Coverage: 23.47% (19548/83298)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a879062fdfbff971bd9172f6d62c09943acb09c6_a879062fdfbff971bd9172f6d62c09943acb09c6/report/index.html

github-actions

clang-tidy made some suggestions

be/src/vec/common/allocator.cpp

kaka11chen · 2024-07-11T07:49:40Z

run buildall

kaka11chen · 2024-07-11T08:49:22Z

run buildall

kaka11chen · 2024-07-19T10:11:22Z

run buildall

doris-robot · 2024-07-19T11:51:48Z

TeamCity be ut coverage result:
Function Coverage: 36.41% (9245/25393)
Line Coverage: 27.98% (75548/270023)
Region Coverage: 26.80% (38835/144923)
Branch Coverage: 23.55% (19717/83734)
Coverage Report: http://coverage.selectdb-in.cc/coverage/608f36c6fc49b4c63eccc19ac4ba38acccfb2bc3_608f36c6fc49b4c63eccc19ac4ba38acccfb2bc3/report/index.html

be/src/vec/common/allocator.h

be/src/vec/exec/format/parquet/arrow_memory_pool.h

kaka11chen · 2024-07-23T17:27:45Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-23T17:29:42Z

be/src/vec/exec/format/parquet/arrow_memory_pool.cpp

+    return arrow::Status::OK();
+}
+
+void ArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) {


warning: pointer parameter 'ptr' can be pointer to const [readability-non-const-parameter]

Suggested change

void ArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) {

void ArrowAllocator::deallocate_aligned(const uint8_t* ptr, int64_t size, int64_t alignment) {

be/src/vec/exec/format/parquet/arrow_memory_pool.h:47:

- void deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment); + void deallocate_aligned(const uint8_t* ptr, int64_t size, int64_t alignment);

kaka11chen · 2024-07-23T17:57:25Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/common/allocator.h

doris-robot · 2024-07-23T19:27:19Z

TeamCity be ut coverage result:
Function Coverage: 36.39% (9242/25396)
Line Coverage: 27.97% (75531/270063)
Region Coverage: 26.79% (38835/144955)
Branch Coverage: 23.54% (19715/83762)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf1a1a9416d265e469de8b9021ee6d004adbe8ef_bf1a1a9416d265e469de8b9021ee6d004adbe8ef/report/index.html

xinyiZzz

LGTM

github-actions · 2024-07-24T03:42:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-24T03:42:58Z

PR approved by anyone and no changes requested.

be/src/vec/common/allocator.h

kaka11chen · 2024-07-24T06:23:48Z

run buildall

doris-robot · 2024-07-24T08:01:32Z

TeamCity be ut coverage result:
Function Coverage: 36.40% (9243/25396)
Line Coverage: 27.97% (75534/270071)
Region Coverage: 26.79% (38830/144960)
Branch Coverage: 23.53% (19712/83764)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8f3272d573f85eac5bf4ac38c04a24caaa92e5bf_8f3272d573f85eac5bf4ac38c04a24caaa92e5bf/report/index.html

kaka11chen · 2024-07-24T09:47:42Z

run buildall

doris-robot · 2024-07-24T11:29:09Z

TeamCity be ut coverage result:
Function Coverage: 36.40% (9243/25396)
Line Coverage: 27.97% (75533/270094)
Region Coverage: 26.78% (38821/144981)
Branch Coverage: 23.53% (19711/83784)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d746ddc44603b8c12ecf6a09c79c8f1681a95dbf_d746ddc44603b8c12ecf6a09c79c8f1681a95dbf/report/index.html

Backport #38562. Fix allocator.h compiling failed on mac which introduced by #37257.

github-actions bot reviewed Jul 4, 2024

View reviewed changes

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch 3 times, most recently from cf824f8 to 9d35789 Compare July 4, 2024 05:48

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 9d35789 to bb95ae7 Compare July 5, 2024 03:36

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from bb95ae7 to 4ebdd75 Compare July 9, 2024 07:13

github-actions bot reviewed Jul 9, 2024

View reviewed changes

be/src/vec/common/allocator.cpp Show resolved Hide resolved

be/src/vec/common/allocator.h Show resolved Hide resolved

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch 2 times, most recently from 37921e7 to 7df36ce Compare July 9, 2024 08:08

github-actions bot reviewed Jul 9, 2024

View reviewed changes

be/src/vec/common/allocator.cpp Show resolved Hide resolved

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 7df36ce to fa2511b Compare July 9, 2024 08:32

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from fa2511b to a879062 Compare July 9, 2024 08:57

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from a879062 to 8d619bf Compare July 11, 2024 07:31

github-actions bot reviewed Jul 11, 2024

View reviewed changes

be/src/vec/common/allocator.cpp Show resolved Hide resolved

[Feature](multi-catalog) Add memory tracker for orc reader and writer.

c28752d

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 8d619bf to 78d16de Compare July 11, 2024 07:49

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 78d16de to 9b362e4 Compare July 11, 2024 08:48

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 9b362e4 to ed40639 Compare July 11, 2024 09:24

xinyiZzz reviewed Jul 23, 2024

View reviewed changes

be/src/vec/common/allocator.h Outdated Show resolved Hide resolved

xinyiZzz reviewed Jul 23, 2024

View reviewed changes

be/src/vec/common/allocator.h Outdated Show resolved Hide resolved

xinyiZzz reviewed Jul 23, 2024

View reviewed changes

be/src/vec/common/allocator.h Outdated Show resolved Hide resolved

xinyiZzz reviewed Jul 23, 2024

View reviewed changes

be/src/vec/exec/format/parquet/arrow_memory_pool.h Show resolved Hide resolved

github-actions bot reviewed Jul 23, 2024

View reviewed changes

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch 2 times, most recently from 64a4a7e to bf1a1a9 Compare July 23, 2024 17:56

github-actions bot reviewed Jul 23, 2024

View reviewed changes

be/src/vec/common/allocator.h Show resolved Hide resolved

be/src/vec/common/allocator.h Show resolved Hide resolved

xinyiZzz approved these changes Jul 24, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 24, 2024

yiguolei reviewed Jul 24, 2024

View reviewed changes

be/src/vec/common/allocator.h Show resolved Hide resolved

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from bf1a1a9 to 8f3272d Compare July 24, 2024 06:19

Fix review comments.

d746ddc

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 8f3272d to d746ddc Compare July 24, 2024 08:29

morningman merged commit a751372 into apache:branch-2.1 Jul 25, 2024

kaka11chen mentioned this pull request Aug 1, 2024

[Fix](memory) Fix allocator.h compiling failed on mac. #38646

Merged

yiguolei pushed a commit that referenced this pull request Aug 1, 2024

[Fix](memory) Fix allocator.h compiling failed on mac. (#38646)

4042cdf

Backport #38562. Fix allocator.h compiling failed on mac which introduced by #37257.

yiguolei mentioned this pull request Sep 5, 2024

Release Note 2.1.6 #40443

Closed

3 tasks

	void free(char* p) override {
	void free(const char* p) override {

	void ArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) {
	void ArrowAllocator::deallocate_aligned(const uint8_t* ptr, int64_t size, int64_t alignment) {

[Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 #37257

[Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 #37257

Uh oh!

Conversation

kaka11chen commented Jul 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

zgxme commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaka11chen commented Jul 11, 2024

Uh oh!

kaka11chen commented Jul 11, 2024

Uh oh!

kaka11chen commented Jul 19, 2024

Uh oh!

doris-robot commented Jul 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaka11chen commented Jul 23, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jul 23, 2024

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented Jul 23, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

kaka11chen commented Jul 4, 2024 •

edited

Loading