[Feature Request]: Basic Health Check Endpoint to verify if a URL is accessible before running a full crawl operation.

[PATAKAMURIVENKATAGANESH]

What needs to be done?

Add a simple URL health check method to AsyncWebCrawler that allows users to quickly verify if a URL is accessible before running a full crawl operation.

What problem does this solve?

Current Pain Points:

1. Users waste time and resources crawling inaccessible URLs
2. No quick way to validate URLs before expensive crawling operations
3. Failed crawls only discovered after full browser startup and page load attempts
4. Batch crawling operations fail silently on dead URLs
5. No pre-crawl validation for URL lists

All of the above can solved as it does:

1. Pre-crawl validation: Check URL accessibility before launching resource-intensive crawling
2. Batch processing: Filter out dead URLs from large URL lists before crawling
3. URL list cleanup: Validate sitemaps or URL collections quickly
4. Network troubleshooting: Diagnose connectivity issues vs. crawling issues
5. Resource optimization: Avoid wasting browser resources on unreachable sites

Target users/beneficiaries

1. Developers building web scraping applications who need to validate URLs
2. Data engineers processing large URL datasets who want to filter invalid URLs
3. QA teams testing URL accessibility before running crawling scripts
4. DevOps teams monitoring website availability as part of crawling pipelines
5. Research teams working with large URL collections from academic datasets

Current alternatives/workarounds

No response

Proposed approach

class AsyncWebCrawler:
async def health_check(self, url: str, timeout: float = 10.0) -> Dict[str, Any]:
"""
Perform quick accessibility check on URL

    Args:
        url: URL to check
        timeout: Request timeout in seconds
        
    Returns:
        Dict with accessibility status and metadata
    """
    start_time = time.time()
    
    try:
        # Use same session/config as main crawler for consistency
        async with aiohttp.ClientSession(
            timeout=aiohttp.ClientTimeout(total=timeout),
            headers=self._get_health_check_headers()
        ) as session:
            async with session.head(url, allow_redirects=True) as response:
                response_time = (time.time() - start_time) * 1000
                
                return {
                    "accessible": 200 <= response.status < 400,
                    "status_code": response.status,
                    "response_time_ms": round(response_time, 2),
                    "content_type": response.headers.get('content-type', ''),
                    "final_url": str(response.url),
                    "redirected": str(response.url) != url
                }
                
    except Exception as e:
        return {
            "accessible": False,
            "status_code": None,
            "response_time_ms": round((time.time() - start_time) * 1000, 2),
            "error": str(e),
            "error_type": type(e).__name__
        }