Auto compaction can get stuck

[jihoonson]

Affected Version

0.13.x, 0.14.x, 0.15.x

Description

The auto compaction can pick up the same interval over again if there are more than 1 segment in that interval after compaction. This was introduced in #6407 and is because the auto compaction is not considering targetCompactionSizeBytes or maxRowsPerSegment while searching for segments to compact.

[jihoonson]

I think this should be a release blocker since it potentially breaks the auto compaction.

[clintropolis]

removing from 0.16 milestone since partial fix #8495 was merged and backported

[jihoonson]

Fixed in #8573.