-
-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
Description
--without-videos only prevent the Youtube ones from being processed and included but there are many regular videos anyway. Those regular videos are short and small-sized and easy to process.
Youtube ones can be huge and are retrieved from Youtube where exceeding a certain number of resource usage (requests? bandwidth ?) results in being banned for a period of time.
We thus need to process Youtube videos one by one but others can be handled concurrently.
Current code specify the nb of videos worker based on --without-videos but ideally, we should treat both sources differently no matter the option as including Youtube videos means processing all the videos on a single worker.