Segments created by an overwriting task with dynamic partitioning doesn't atomically overshadow old segments in the overlapping time chunk

[jihoonson]

Affected Version

All versions

Description

The overwriting task such as the compaction task with the dynamic partitioning creates segments with NumberedShardSpec. This NumberedShardSpec has the partitions field indicating the core partition size. However, it is always set to 0 for batch tasks. Since the core partition size is 0, the segment update in the broker/coordinator server view is not guaranteed to be done atomically even in the overlapping time chunk. Instead, a new segment will overshadow immediately when a first historical announces a new segment. As a result, some queries can return less results than normal until all new segments are available in historicals.

This issue doesn't exist with hash or range partitioning.