Skip to content

[fix](cloud-mow)Full compaction no need to sync delete bitmap #43613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dataroaring merged 1 commit into from
Nov 11, 2024
Merged

[fix](cloud-mow)Full compaction no need to sync delete bitmap #43613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions be/src/cloud/cloud_compaction_action.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,9 @@ Status CloudCompactionAction::_handle_run_compaction(HttpRequest* req, std::stri
compaction_type != PARAM_COMPACTION_FULL) {
return Status::NotSupported("The compaction type '{}' is not supported", compaction_type);
}

CloudTabletSPtr tablet = DORIS_TRY(_engine.tablet_mgr().get_tablet(tablet_id));
bool sync_delete_bitmap = compaction_type != PARAM_COMPACTION_FULL;
CloudTabletSPtr tablet =
DORIS_TRY(_engine.tablet_mgr().get_tablet(tablet_id, false, sync_delete_bitmap));
if (tablet == nullptr) {
return Status::NotFound("Tablet not found. tablet_id={}", tablet_id);
}
Expand Down
5 changes: 3 additions & 2 deletions be/src/cloud/cloud_meta_mgr.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,8 @@ Status CloudMetaMgr::get_tablet_meta(int64_t tablet_id, TabletMetaSharedPtr* tab
return Status::OK();
}

Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data) {
Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
Copy link
Contributor

@github-actions github-actions bot Nov 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'sync_tablet_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]

Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
                     ^
Additional context

be/src/cloud/cloud_meta_mgr.cpp:384: 181 lines including whitespace and comments (threshold 80)

Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
                     ^

bool sync_delete_bitmap) {
using namespace std::chrono;

TEST_SYNC_POINT_RETURN_WITH_VALUE("CloudMetaMgr::sync_tablet_rowsets", Status::OK(), tablet);
Expand Down Expand Up @@ -463,7 +464,7 @@ Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_

// If is mow, the tablet has no delete bitmap in base rowsets.
// So dont need to sync it.
if (tablet->enable_unique_key_merge_on_write() &&
if (sync_delete_bitmap && tablet->enable_unique_key_merge_on_write() &&
tablet->tablet_state() == TABLET_RUNNING) {
DeleteBitmap delete_bitmap(tablet_id);
int64_t old_max_version = req.start_version() - 1;
Expand Down
3 changes: 2 additions & 1 deletion be/src/cloud/cloud_meta_mgr.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ class CloudMetaMgr {

Status get_tablet_meta(int64_t tablet_id, std::shared_ptr<TabletMeta>* tablet_meta);

Status sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data = false);
Status sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data = false,
bool sync_delete_bitmap = true);

Status prepare_rowset(const RowsetMeta& rs_meta,
std::shared_ptr<RowsetMeta>* existed_rs_meta = nullptr);
Expand Down
11 changes: 6 additions & 5 deletions be/src/cloud/cloud_tablet_mgr.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,8 @@ void set_tablet_access_time_ms(CloudTablet* tablet) {
tablet->last_access_time_ms = now;
}

Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_id,
bool warmup_data) {
Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_id, bool warmup_data,
bool sync_delete_bitmap) {
// LRU value type. `Value`'s lifetime MUST NOT be longer than `CloudTabletMgr`
class Value : public LRUCacheValueBase {
public:
Expand All @@ -168,8 +168,8 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
CacheKey key(tablet_id_str);
auto* handle = _cache->lookup(key);
if (handle == nullptr) {
auto load_tablet = [this, &key,
warmup_data](int64_t tablet_id) -> std::shared_ptr<CloudTablet> {
auto load_tablet = [this, &key, warmup_data,
sync_delete_bitmap](int64_t tablet_id) -> std::shared_ptr<CloudTablet> {
TabletMetaSharedPtr tablet_meta;
auto st = _engine.meta_mgr().get_tablet_meta(tablet_id, &tablet_meta);
if (!st.ok()) {
Expand All @@ -180,7 +180,8 @@ Result<std::shared_ptr<CloudTablet>> CloudTabletMgr::get_tablet(int64_t tablet_i
auto tablet = std::make_shared<CloudTablet>(_engine, std::move(tablet_meta));
auto value = std::make_unique<Value>(tablet, *_tablet_map);
// MUST sync stats to let compaction scheduler work correctly
st = _engine.meta_mgr().sync_tablet_rowsets(tablet.get(), warmup_data);
st = _engine.meta_mgr().sync_tablet_rowsets(tablet.get(), warmup_data,
sync_delete_bitmap);
if (!st.ok()) {
LOG(WARNING) << "failed to sync tablet " << tablet_id << ": " << st;
return nullptr;
Expand Down
3 changes: 2 additions & 1 deletion be/src/cloud/cloud_tablet_mgr.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ class CloudTabletMgr {

// If the tablet is in cache, return this tablet directly; otherwise will get tablet meta first,
// sync rowsets after, and download segment data in background if `warmup_data` is true.
Result<std::shared_ptr<CloudTablet>> get_tablet(int64_t tablet_id, bool warmup_data = false);
Result<std::shared_ptr<CloudTablet>> get_tablet(int64_t tablet_id, bool warmup_data = false,
bool sync_delete_bitmap = true);

void erase_tablet(int64_t tablet_id);

Expand Down