Set time limit and interrupt crawling that takes too long

[ryangawei]

Hi,

I am crawling 10K URLs using crawler.arun_many with batch size 100. In some batches, some URLs takes extremely long (a few hours) and blocks the entire process. I want to know if it is possible to add time limit so that, the URL job that takes more than X seconds to crawl will be canceled, or keep only the content that have been crawled. I'm willing to sacrifice that URL to ensure the time of entire dataset.

Thank you.