scrapper-tool

A reusable Python web-scraping toolkit — production-grade primitives, anti-bot ladder, fixture-replay testing.

Built from the scraping core behind PartsPilot, extracted as an open-source library so other projects (and LLM agents) can pick up the same patterns without redoing the reverse-engineering work.

Quickstart · All docs · Settings · MCP · Docker · Changelog

---

Status (2026-08-29): stable (v3.1.1). The public Python API and MCP tool surface are SemVer-stable.

v3.1.0 migrates the MCP server to the 2.x SDK (FastMCP — MCPServer), lifting the mcp<2 cap deferred from 3.0.0. No public Python API or MCP tool changed; the breaking part is the dependency floor, mcp>=2.1.1,<3 on the [agent] extra, which will not co-install with a mcp 1.x pin. It also fixes both docker-compose MCP services, which inherited the REST entrypoint and so never spoke MCP at all.

v3.0.0 adds a target URL guard that vets every URL before a request is issued — private, loopback and cloud-metadata targets are refused, and that is on by default, which is the breaking part. It also gives the captcha grid tier its own vision model, blocks page-initiated SSRF in the render tier, promotes the impersonation ladder to chrome150, and stops overriding the impersonated User-Agent (which had been advertising scrapper-tool/0.1 beside a Chrome TLS handshake — a self-identifying mismatch).

Read the breaking-change table before upgrading. Every number in the docs is measured, including the ones that did not work; see docs/TESTING.md.

What it does

Web scraping is mostly the same work every time: pick the extraction method the site actually needs, survive the TLS fingerprint, retry sanely, and write tests that do not break the moment the vendor ships CSS. scrapper-tool packages the parts that do not change per vendor.

One call does the escalation for you — cheapest method first, climbing only when the site forces it:

from scrapper_tool import scrape

data = await scrape("https://vendor.example/product/123")

Behind that call: a TLS-impersonation ladder, a stealth browser, a local LLM, and a captcha cascade — in that order, and only as far as the site makes necessary.

Install

uv pip install "scrapper-tool[full,agent]"    # all five patterns + MCP server
camoufox fetch                                # ~300 MB, best-stealth browser

pip works too, but [full] needs uv — Scrapling and Crawl4AI pin incompatible lxml ranges and only uv honours the override that reconciles them. Lighter installs and the pip escape hatch: Install guide.

Check what actually works on your machine:

scrapper-tool doctor

It reports every tier as ok / degraded / missing with the exact command to fix each one, and exits non-zero so it works as a CI or container healthcheck.

The five patterns

Pick the one DevTools points at, or let scrape() choose.

Pattern	When	Cost
A — JSON API	An XHR returns the data	Lowest
B — Embedded JSON	ld+json, __NEXT_DATA__, __NUXT__	Low
C — CSS / microdata	Price is in the HTML, no JSON	Medium
D — Hostile	Cloudflare Turnstile, Akamai, DataDome	High — real browser
E — LLM agent	D is still blocked, or the page needs interaction	Highest — local LLM

Full guides: patterns A–E.

Security: targets are vetted before they are fetched

Every surface checks a URL before issuing a request. Private, loopback, link-local and cloud-metadata targets are refused, along with non-http(s) schemes and hostnames that resolve into private space.

This is on by default, and it matters most if you run the REST sidecar: without it, anything that can reach the sidecar can make it fetch 169.254.169.254 and read your cloud credentials back.

To reach a legitimate internal target, allowlist it rather than turning the guard off:

SCRAPPER_TOOL_URL_GUARD_ALLOW=127.0.0.1,10.0.0.0/8

What is covered, what is not, and the fully-closed ..._STRICT mode: Target URL guard.

Run it as a service

Mode	Command	Docs
MCP server (Claude, Cursor, any MCP client)	scrapper-tool-mcp	docs/mcp.md
REST sidecar (any language, plain HTTP)	scrapper-tool-serve	docs/http-sidecar.md
Docker (all five patterns in one image)	docker compose up	docs/docker.md

Use it as an MCP server

Exposes the whole toolkit to any MCP client — Claude Code, Claude Desktop, Cursor, mcp-use, AutoGen, LangChain. Needs the [agent] extra:

uv pip install "scrapper-tool[full,agent]"

Wire it up

Add to your client's MCP config (.mcp.json for Claude Code, claude_desktop_config.json for Claude Desktop):

{
  "mcpServers": {
    "scrapper-tool": {
      "command": "scrapper-tool-mcp",
      "args": [],
      "env": {}
    }
  }
}

Restart the client and all nine tools appear, plus a skill://scrapper-tool resource carrying the tool's own operating manual. That is the whole setup for the default transport — stdio, which the client spawns and talks to over stdin/stdout.

The nine tools

Tool	Use it for
auto_scrape	Start here. Escalates A/B/C → D → E1 → E2 by itself and reports which tier won.
fetch_with_ladder	One fetch through the TLS-impersonation ladder. extract_structured=True also parses JSON-LD.
extract_product	schema.org Product+Offer out of HTML you already have.
extract_microdata_price	<meta itemprop="price"> anchors out of HTML you already have.
map_site	List a site's URLs from sitemaps + page links. No browser, no LLM, so it is cheap.
crawl_site	Breadth-first crawl running the full cascade per page. Honours robots.txt.
agent_extract	Pattern E1 — stealth render plus one LLM call. Needs [llm-agent].
agent_browse	Pattern E2 — multi-step agent for logins, pagination, forms. Needs [llm-agent].
canary	Which TLS fingerprint a site accepts. Diagnostics.

docs/mcp-tools.json is the generated, CI-enforced copy of this list.

Over HTTP instead of stdio

For a long-lived server that clients reach by URL:

scrapper-tool-mcp --transport streamable-http --host 0.0.0.0 --port 8000

Then point the client at http://localhost:8000/mcp. --transport sse is also supported. Each flag has an env var (SCRAPPER_TOOL_MCP_TRANSPORT, _HOST, _PORT).

In Docker, use the bundled service rather than the bare image — the image's default entrypoint is the REST sidecar, and the compose service overrides it:

docker compose --profile http up -d scrapper-tool-mcp-http

For the stdio spawn pattern in Docker (docker compose run --rm -T scrapper-tool), see docs/mcp.md.

Check it works

scrapper-tool doctor

Reports every tier as ok / degraded / missing. For a real end-to-end session that opens a JSON-RPC connection and calls all nine tools:

uv run python scripts/e2e/test_mcp_session.py

Two things that will otherwise cost you an hour: agent_browse needs a CDP-capable browser, so set SCRAPPER_TOOL_AGENT_BROWSER=patchright (the Camoufox default is Firefox, which has no CDP and fails deliberately rather than silently dropping stealth); and in Docker a host-local LLM URL must be host.docker.internal, not 127.0.0.1, which inside the container means the container.

Full reference: docs/mcp.md. Framework-specific wiring: docs/agent-integration.md.

Settings

Every knob is an env var, a constructor argument, or a per-call keyword — in that order of precedence. docs/SETTINGS.md is the canonical reference: if a setting is not there, it is not a public knob. .env.example is a drop-in starter with every variable annotated.

Architecture

flowchart TD
    A[Your scraper code or LLM agent] --> B[vendor_client / request_with_retry]
    B --> C{TLS-sensitive?}
    C -- no --> D[httpx]
    C -- yes --> E[curl_cffi ladder]
    E --> E1[chrome150] --> E2[chrome146] --> E3[safari2601] --> E4[firefox147]
    D --> F[Response]
    E4 --> F
    F --> G{Pattern}
    G -- A --> H[JSON API model]
    G -- B --> I[extruct: ld+json / next_data / nuxt]
    G -- C --> J[selectolax: microdata / CSS]
    G -- D --> K["Scrapling (Playwright + Turnstile)"]
    G -- "BlockedError + interactive" --> M["Pattern E: agent_extract / agent_browse"]
    M --> M1["Stealth browser (Camoufox / Patchright / Obscura)"]
    M1 --> M2["Local LLM (Ollama, qwen3-vl:8b)"]
    M2 --> M3["Captcha cascade (Camoufox auto → Theyka → paid)"]
    M3 --> L[Validated product data]
    H --> L
    I --> L
    J --> L
    K --> L

Loading

Documentation

Quickstart	5-minute on-ramp.
Settings reference	Every env var, default, choice list. (v1.0.0+)
.env.example	Drop-in starter file with every variable annotated.
E2E test plan	Operator-runnable end-to-end suite — library / Docker / MCP modes against LM Studio. (v1.0.0+)
scripts/e2e/	Runnable test scripts referenced by the E2E plan.
Recon playbook	DevTools-driven reverse-engineering of a new vendor site.
Pattern A — JSON API	Vendor exposes an XHR / JSON endpoint.
Pattern B — Embedded JSON	ld+json, __NEXT_DATA__, __NUXT__, RSC payloads.
Pattern C — CSS / microdata	itemprop="price", fallback selectors.
Pattern D — Hostile	Cloudflare Turnstile, Akamai EVA.
Pattern E — LLM agent	Local-LLM-driven scraping for any protected site. (v1.0.0+)
Anti-bot ladder reference	How the ladder walks, when to bump the primary profile.
Test helpers	FakeCurlSession, replay_fixture, golden-snapshot pattern.
Agent integration	MCP wiring for Claude, OpenClaw, Hermes Agent, AutoGen, LangChain. (v0.2.0+)
2026-04-30 landscape research	Why these tools, sourced.

Why this exists

Most scrapers are written from scratch every time, even though 90% of the work is the same: pick the right extraction pattern, survive the TLS fingerprint, retry/backoff sanely, and write tests that don't drift the moment a site updates.

scrapper-tool packages the parts that don't change per vendor, so you only write the parts that do.

• Pattern-first design. Five named, documented extraction patterns (A–E) — pick the one DevTools points at, skip the rest.
• Anti-bot ladder built in. Auto-walks chrome150 → chrome146 → safari2601 → firefox147 → chrome133a when a profile gets fingerprinted.
• Deterministic tests. Fixture-replay (FakeCurlSession, replay_fixture, golden snapshots) — no live HTTP in CI.
• Optional hostile mode. Cloudflare Turnstile / Akamai EVA defeat path via Scrapling — opt-in extra, no Playwright bloat by default.
• LLM-agent ready. v0.2.0+ ships an MCP server so Claude, AutoGen, LangChain, etc. can drive the scraper directly.
• Local-LLM scraping for any protected site (v1.0.0+). Pattern E adds Camoufox + browser-use + Crawl4AI + Ollama — zero API cost, two modes (agent_extract for fast 1-call extraction, agent_browse for interactive multi-step tasks). Humanlike-behavior layer defeats DataDome.
• Captchas solved on the way past (v2.2.0+). Five tiers, cheapest first: settle → click the checkbox → align the slider (pure geometry, no model) → read the image grid with a local VLM → paid solver. Measured live: reCAPTCHA v2 grids 3/4–4/5 with a ~27B VLM, GeeTest sliders ~20% with no model at all. reCAPTCHA v3 and AWS WAF are not solvable — they are risk scores, not puzzles, and the docs say so.
• Clearance cookies are kept, not thrown away (v2.2.0+). A solve costs ~70 s of local inference or a paid API call; the cf_clearance it buys now survives to the next tier, and to the next run via a persisted browser profile.
• Boring stack. httpx, curl_cffi, selectolax, extruct. No managed SaaS bundled — your code, your egress.

Roadmap

• v0.1.0 — Core HTTP client, retry/backoff, anti-bot ladder, patterns A–D, fixture-replay test helpers.
• v0.2.0 — MCP server for LLM agents; canary CLI for nightly fingerprint-health probes.
• v1.0.0 — Pattern E: local-LLM-driven scraping (Camoufox + browser-use + Crawl4AI + Ollama), captcha cascade, humanlike-behavior layer, full Docker stack. Public API + MCP tool surface stable under SemVer.
• v1.1.0 — Pluggable rate-limit / robots.txt policies; per-vendor profile presets; agent_session() warm-browser pooling; broader Pattern E backends.

See CHANGELOG.md for landed changes and open issues for what's in flight.

Contributing

PRs and issues are welcome. Every PR that meaningfully changes how we scrape lands a CHANGELOG.md row.

• Read CONTRIBUTING.md for the maintenance contract.
• Read CODE_OF_CONDUCT.md before opening a discussion.
• Good first issues live under the good first issue label.

Contributors

Want to see your avatar here? Check CONTRIBUTING.md and open a PR.

Acknowledgements

scrapper-tool stands on the shoulders of these projects:

• httpx — async HTTP client
• curl_cffi — TLS / JA3 impersonation
• selectolax — fast HTML parsing
• extruct — ld+json, microdata, RDFa extraction
• Scrapling — Playwright-based hostile-site backend

License

MIT © scrapper-tool contributors.

If scrapper-tool saves you time, consider starring the repo — it helps others find it.