Skip to content

Speed-up video processing when including Youtube videos #97

@rgaudin

Description

@rgaudin

--without-videos only prevent the Youtube ones from being processed and included but there are many regular videos anyway. Those regular videos are short and small-sized and easy to process.

Youtube ones can be huge and are retrieved from Youtube where exceeding a certain number of resource usage (requests? bandwidth ?) results in being banned for a period of time.

We thus need to process Youtube videos one by one but others can be handled concurrently.

Current code specify the nb of videos worker based on --without-videos but ideally, we should treat both sources differently no matter the option as including Youtube videos means processing all the videos on a single worker.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions