[Proposal] The execution model of compaction needs to improve

[vagetablechicken]

Current model

Each dir creates M threads for base compaction, N threads for cumulative compaction. And compaction threads execute one compaction in one cycle(may skip execution because no best tablet).
Too many compaction tasks may run out of memory, so we limit the max concurrency of running compaction tasks by semaphore.

Problem

It only limits the thread number. If the running threads cost too much memory, we can't defense it.
If we reduce concurrency to avoid OOM, we can't do compaction in time. We may meet more heavy compactions.
So concurrency limitation is not enough.

Proposal

The most desirable solution is limiting the memory. But this solution assumes that we can estimate the memory usage of one compaction. It's diffcult.
So we can only refer to the tablet score(the segments num). It has positive correlation with memory, but can't simply estimate the mem usage by a scale factor.
What about a model of scores limitation?
A compaction needs to acquire the permits(equals to it's score), and release the permits when it finished. So it wiil be low concurrency when high score compactions running, and high concurrency when low score compactions running.

[morningman]

Good proposal.

However, the compaction strategy is difficult to meet all needs. In the case of inaccurate memory statistics, manual intervention may still be required. Looking forward to a more detailed description of permit-related strategies.

[kangkaisen]

Good proposal.

However, the compaction strategy is difficult to meet all needs. In the case of inaccurate memory statistics, manual intervention may still be required. Looking forward to a more detailed description of permit-related strategies.

+1. If we could abstract the compaction strategies to an interface, which is much better.

[vagetablechicken]

Before a new compaction strategy, we should add compaction memory usage monitoring. It'll help us adjust the model parameters. We could use MemTracker, not all but easy to track.

[vagetablechicken]

We can add a PermitLimiter to do the limitation, and use single-producer/multi-consumer model.
Producer will try to acquire permits, if ok, add the compaction task to consumer pool.
So we need a PermitLimiter member function:

bool PermitLimiter::block_acquire(int permits);
// return false: drop this task
// return true: block waiting for permits succeed, task can start now

pseudocode

Producer:

generate tasks
for task: tasks
    if PermitLimiter.block_acquire(task.permits)
        add to consumer pool
    else
        drop this task
// one cycle end

Consumers:

PermitLimiter.release(permits) when task finished(whether successful or not)

PermitLimiter:

bool PermitLimiter::block_acquire(int permits):
    if permits > total_permits
        if over_sold
            wait_until used_permits == 0 // only allow this task to run
        else
            return false
      else
        wait_until (total_permits - used_permits) >= permits
    
    used_permits += permits
    return true

void PermitLimiter::release(int permits):
    used_permits -= permits

Droping a compaction task is not a good idea, that means this tablet can't do compaction anymore(rowsets will incr over time).
So I use over_sold, which can be modified in runtime. If memory usage is too high, we can disable over_sold for a while.
And I don't use free_permtis, use (total_permits - used_permits) instread. Thus, the total_permits can be modified in runtime too.

[morningman]

That is more clear.
And we need to think about

1. what strategy the producer use to produce the tablet for compaction? by the score?
2. If we use the score, does it use overall sorting, or does each data dir sort individually?
3. What is the relationship between consumer threads and data directories? If it is no relation, it may happen that all consumer threads are processing compaction on the same disk, resulting in greater IO pressure.

[vagetablechicken]

@morningman
The problems can be divided into two separate parts

1. Generation strategy：
   Highest permit task first, but we need to consider about dirs. So
   1. Skip the dirs which are doing compaction
   2. For one dir, we only gen one or X tasks in one round, so it's probably better to gen CUMU 9 times out of 10 rounds, only 1 round to gen BASE. The number can be adjusted.
2. Permits calc strategy：
   I only tested about AlphaRowsetReader's MemPool. The memory estimate model is not perfect now. (Without considering the elusive memory and BetaRowsetReader)
   So the memory estimate model is temporarily unavailable.
   We can use scores as permit on the first step.
   (cumulative compaction has config::max_cumulative_compaction_num_singleton_deltas. So the score of cumu compaction task is greater than max_deltas, the permits can be set to max_deltas.)

[vagetablechicken]

To add compaction memory usage monitoring, we need connect the MemTrackers in compaction. So we could get the total mem consumption of compactions.
Then register metrics of total & log thread level info(one compaction's info).