Skip to content

Synchronized segment allocation #6087

@gianm

Description

@gianm

Today, when SegmentAllocateAction is run by multiple threads at once, it can end up doing a lot of internal retries due to its SELECT -> INSERT logic (see allocatePendingSegment in IndexerSQLMetadataStorageCoordinator). There isn't much point in running concurrent allocations for the same datasource anyway, and the contention causes retries which add delay due to sleeps. It might work better to synchronize allocations on a per-datasource basis, so only one can happen at a time per datasource.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions