How can I avoid rate limit when using Deep Crawl?

[nguyenthengocdev]

How can I avoid rate limit when using Deep Crawl?

• Any native way to do that instead of proxy when i crawl for depth urls

[Mohammedrefaat98]

I advise you to select the needed elements you need to scrape by using the css_selector attribute inside CrawlerRunConfig method. Otherwise, for more rate limit, use Google Gemeni LLM as it can handle larger tokens than any LLMs.

Resources:

https://brightdata.com/blog/web-data/crawl4ai-and-deepseek-web-scraping (to see the css_selector)