[Proposal] Execute 'pick rowsets' before applying for permits for a compaction task

[weizuo93]

The current compaction mechanism is that there is a producer thread that has been producing compaction tasks, and the selected tablet must apply for permits. When a tablet could hold permits, compaction task for this tablet will be submitted to thread pool. We take compaction score as permits which is used for limiting memory consumption. However, pick_rowset_to_compaction() will be executed before the file merge in compaction thread, and the number of segment files that actually perform the merge operation is smaller than compaction score. In addition, it is also possible that compaction task exits directly because the tablet doesn't meet the requirements of compaction.

Can we execute 'pick rowsets' before applying for permits for a compaction task, calculate the number of segment files that actually participate in the merge operation, and take this number as permits.

Based on test results, we found thatpick_rowset_to_compaction() execution time for a tablet is on the time scale of milliseconds. Therefore, there is little pressure on the producer thread.

1112 16:00:08.686122 (+   982us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:08.355522 (+  1033us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:08.355372 (+   930us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:09.707385 (+   990us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:32.218772 (+  1305us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:32.212823 (+  1212us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:32.077322 (+  2411us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:33.534394 (+  2484us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:52.350023 (+  1181us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:54.011435 (+  1212us) cumulative_compaction.cpp:50] rowsets picked
1112 16:01:14.845957 (+   294us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:17.865887 (+    97us) base_compaction.cpp:44] rowsets picked
1112 16:01:35.193204 (+   206us) cumulative_compaction.cpp:50] rowsets picked
1112 16:01:39.244388 (+   198us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:18.931984 (+  1627us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:19.014592 (+  1394us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:03.405753 (+   160us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:04.777374 (+   350us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:07.016017 (+   164us) cumulative_compaction.cpp:50] rowsets picked
1112 15:59:12.573256 (+   205us) base_compaction.cpp:44] rowsets picked
1112 16:02:18.932822 (+  2213us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:08.975090 (+   339us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:18.932387 (+  1885us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:11.285206 (+   402us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:14.799639 (+   119us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:20.540975 (+   148us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:23.908919 (+   198us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:32.000026 (+   133us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:13.347364 (+  2796us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:13.722429 (+  2046us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:43.050516 (+   410us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:43.849807 (+   387us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:17.914220 (+   469us) base_compaction.cpp:44] rowsets picked
1112 16:03:48.663824 (+   336us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:49.778246 (+   241us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:52.141355 (+   468us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:20.670042 (+  1564us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:38.078296 (+   264us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:36.673882 (+  1177us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:37.060312 (+  1139us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:55.154987 (+  1215us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:55.403399 (+  1071us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:02.536426 (+  1551us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:32.090439 (+  1024us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:18.661826 (+  1306us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:18.990078 (+  1241us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:42.798424 (+   143us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:46.082527 (+   176us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:41.727956 (+  1584us) cumulative_compaction.cpp:50] rowsets picked
1112 16:06:01.083588 (+  1123us) cumulative_compaction.cpp:50] rowsets picked
1112 16:06:26.622230 (+   632us) cumulative_compaction.cpp:50] rowsets picked
1112 16:06:35.897463 (+    92us) cumulative_compaction.cpp:50] rowsets picked