Implement distributed rate limiting for downloads

[Amdrel]

Description

This patch aims to help reduce the risk of blocks from YouTube (see #320) by restricting video2commons from performing concurrent downloads of videos from YouTube and introducing a delay between requests. Concurrent downloads are restricted by using a lock stored in Redis that allows only one worker at a time to download videos.

Note: It seems the block is active again so we won't be able to see how successful this patch is until the existing one expires.

Changes

• Added a new context manager named YoutubeDLRateLimited that wraps yt_dlp.YoutubeDL
  • This applies rate-limiting parameters and acquires a distributed lock in Redis that prevents concurrent requests to rate-limited sites between multiple workers
  • The frontend (flask) and backend (workers) use separate locks since the blocks on the backend don't affect the frontend due to having different public IPs (see the source param)
• Only YouTube domains are rate-limited currently and other URLs bypass locking entirely
  • The code can be easily extended to work with other sites if needed, though I'm doubtful this will happen anytime soon as we've only have had blocking issues with YouTube from what I've seen
• Once the lock is acquired, rate-limiting is done via sleep delays that vary
  • Media downloads have a delay of 10 seconds
  • Extraction requests wait 3 seconds
  • Subtitle downloads wait 6 seconds
  • The frontend uses a shorter lock timeout of 60 seconds and an extraction delay of 1 second as it seems doing those kinds of requests in isolation hasn't caused us any issue yet
• Added logging.basicConfig(level=logging.INFO) to the worker and flask app since logging wasn't setup and having the diagnostic information for concurrent locks might be useful if we run into any issues

Deployment

This change affects both the workers and the flask application, so both would need to be redeployed.

Testing

I've tested this patch against a couple of playlists (no larger than a dozen videos) and files with large amounts of subtitles using multiple workers, and the lock seemed to work fine. You can confirm the lock is working from the frontend if you see tasks being held up by "Creating YoutubeDL instance".

In the screenshot the first task is downloading a video (with the lock), the second is transcoding one, and the third is waiting for the lock to be released. It's important to note that the lock doesn't prevent other encoding tasks from running as it's released the moment the YouTube API is no longer in use.

[Amdrel]

I couldn't find a way to use the extractor since that appears to be determined after the params are created and sent to yt-dlp, so this is a bit of a hack. I'm open to replacing this if there's a better way.

[don-vip]

looks fine to me

[don-vip]

Nice! Let's see if it helps. At least it won't hurt :)