feat(crawl): add the zenrows crawl command and its acceptance test

[StefanoGuerrini]

Summary

This PR adds zenrows crawl, a command group for the new Crawl API (https://api.zenrows.com/v1/crawls). Crawl follows links from one start URL, on the registrable domain of that URL. depth is optional: with it the crawl stops after that many link hops; without it there is no depth limit, and max_pages or max_items (both default 100) stops the crawl. It returns the URLs it keeps, with optional page HTML. stop retries 429, 502, 503 and 504, not only a 503 crawl_busy: up to 3 retries, each waiting Retry-After (at most 30 s) or 1 s. The CLI uses the same operation and parameter names as the Node, Python and Go SDKs and the MCP server. The PR also adds an acceptance test that installs the release artifact in a clean container and runs every Crawl feature against the live API.

This PR also carries #33 and #34, folded into this branch.

Squash commit message

feat(crawl): add the `zenrows crawl` command and its acceptance test

Add the crawl subcommands: create, get, list, results, content,
download, stop and wait. Crawl follows the links from one start URL
and returns the URLs it keeps, with optional page HTML. --depth is
optional: without it the crawl has no depth limit, and --max-pages or
--max-items stops it. Both default to 100 in the API.

Add a Crawl API client. It sends the X-API-Key header and a
zenrows-cli User-Agent, and ZENROWS_CRAWL_API_BASE overrides the base.
It retries transient failures as the Crawl SDKs do: up to 3 retries,
with Retry-After capped at 30 s. A create retries only with an
Idempotency-Key and never on 429. A stop retries 429, 502, 503 and
504, waiting Retry-After or 1 s, since repeating a stop is safe.

Map Crawl errors to CRAWL_* codes that carry server_code, crawl_id
and, for 429 and crawl_busy, retry_after. A 429 is
CRAWL_TOO_MANY_CRAWLS only when the API sends too_many_crawls; any
other 429 is CRAWL_RATE_LIMITED. Print snake_case JSON under
--json; create, get, wait and stop print {ok, crawl_id, status,
crawl}. A wait that runs out exits 0 with the crawl still running and
a resume hint. Ctrl-C stops the wait, not the crawl, and exits 130.

Stream the download export into <file>.part and rename it into
place, so a failed export keeps any earlier file. A content id with
a bad % escape fails with INVALID_USAGE before any request.

Check the domain policy and max_pages_per_run before a create, with
an unset --max-pages counted as 100, and write a run artifact. Add
the crawl capability and skill, README and docs sections, unit tests,
and an opt-in e2e test (npm run test:e2e).

Add acceptance/crawl/run.sh. It installs the npm pack tarball in a
clean, digest-pinned container that gets the API key by name only,
runs the README Crawl quickstart and checks 1 to 21 plus CLI extras
against the live API, and writes a report to acceptance/crawl/out/.
A run starts five or six crawls at depth 1, keeps at most 2 active,
stops every crawl it started, and scans out/ for the key. Clean-up
retries a stop that gets crawl_busy. The package files list keeps
the harness out of the published package.

Squash-merge with this message: the branch commits carry trailer lines that should not land.

This PR replaces #30. The review of the first version is on #30. The code changed after that review, and this description matches the current head. This PR also carries #32, merged into this branch.

What changes

Commands

zenrows crawl create <url> [--depth N] [--max-items N] [--max-pages N]
                     [--include-patterns P]... [--exclude-patterns P]... [--output-format html]
                     [--follow] [--timeout s] [--idempotency-key K] [--no-signup]
zenrows crawl get <id> [--cursor C] [--limit N]
zenrows crawl list [--cursor C] [--limit N]
zenrows crawl results <id> [--limit N] [--out file]
zenrows crawl content <id> <content_id|content_url> [--out file]
zenrows crawl download <id> [--out file]
zenrows crawl stop <id>
zenrows crawl wait <id> [--timeout s]

Subcommand	Behavior
create	Sends only the fields the caller set. --depth (1 to 100000) is optional: without it the CLI sends no depth, the crawl has no depth limit, and the crawl it prints has no depth field. --max-items and --max-pages default to 100 in the API. --include-patterns and --exclude-patterns are repeatable substring filters that map to include_patterns and exclude_patterns. --output-format takes only html; each kept page is then one more fetch that --max-pages counts.
get	Reads one API page: the crawl, one page of results, and next_cursor.
list	Reads one page of crawls.
results	Follows every page; --limit is the page size. A running crawl gives the URLs kept so far with partial: true. An ended crawl is complete, also a stopped one with next_cursor still set.
content	Takes a content_id or the whole content_url. A ref with a bad % escape fails with INVALID_USAGE before any request. Under --json it prints content_id, content_type, bytes, and content or, with --out, file.
download	Streams the NDJSON export into <file>.part and renames it to --out (default <id>.jsonl), so the export never sits whole in memory. A failed export keeps any earlier file. It reports status from X-Crawl-Status (null when absent) and partial.
stop	On a crawl that already ended, prints the end status, not "Stopped".
create --follow, wait	Poll with limit=1, from 2 s, x1.5, capped at 15 s, until the crawl ends or --timeout runs out (default 600 s).

When the wait runs out, the command exits 0. It prints the crawl with status running and the hint zenrows crawl wait <id>; under --json the hint is in note. A crawl that ends failed exits 1. Ctrl-C stops the wait, not the crawl, prints the resume hint, and exits 130.

JSON output

Every subcommand prints JSON with --json, in the {ok, ..., error} shape of the other commands. All keys are snake_case, as in the MCP server. create, get, wait and stop print {ok, crawl_id, status, crawl}, and the crawl object holds coverage, stop_reason and error. get adds results and next_cursor. With results --out --json, the envelope carries file instead of results. The run artifact keeps crawlId, as Batch keeps jobId.

Client and retries (src/core/crawl-api.ts)

Request	Retries on	Wait
GET	429, 502, 503, 504, network error or timeout	Retry-After, else 250 ms x 2^n (+/-20% jitter, max 10 s)
create with --idempotency-key	502, 503, 504, network error; never 429	same
create without a key	none	—
stop	429, 502, 503 (any code, crawl_busy included), 504	Retry-After, else 1 s

Each request retries at most 3 times. The client caps Retry-After at 30 s, so a large server value cannot hang a command. A stop is safe to repeat: a stop on an ended crawl answers 200 with its status. A crawl_busy means the server did not save the stop, and the crawl still runs. A stop error that remains after the retries maps as the error table says. A download that breaks mid-stream retries as any GET, and the retry writes the file and the count again from the start. An error from writing the file gets no retry. Batch does not change.

Error codes

Code	Cause
CRAWL_NOT_ENABLED	403 REQS008
AUTH_INVALID	401
CRAWL_KEY_CAP_REACHED	402 AUTH014 or api_key_cap_reached
CRAWL_INSUFFICIENT_CREDITS	any other 402; keeps the claim-URL and top-up advice
CRAWL_INVALID_REQUEST	400, 422; for idempotency_key_reused, send a new key or no key
CRAWL_REQUEST_IN_FLIGHT	409; retry when the first request ends
CRAWL_NOT_FOUND, CRAWL_CONTENT_NOT_FOUND	404
CRAWL_TOO_MANY_CRAWLS	429 too_many_crawls, which the API sends only on create; active-job limit (3 by default, shared with Batch), with retry_after
CRAWL_RATE_LIMITED	any other 429, such as a stop that still gets 429 after its retries; "The Crawl API is rate limiting requests.", with retry_after
CRAWL_BUSY	503 crawl_busy after the stop retries, with retry_after; running the stop again is safe
CRAWL_FAILED	a crawl that ended failed (advice keyed on error.code, such as seed_unreachable or no_items_found), any other 403, or another error, such as a 502, 504 or other 503 that remains after the retries
BACKEND_UNAVAILABLE	network failure

Every error carries the API code as server_code when the API sends one. A call about one crawl also carries crawl_id.

Policy, docs and skills

create checks the domain policy and max_pages_per_run (default 1000) before any network call, and writes a run artifact. An unset --max-pages counts as the API default of 100, so a policy below 100 refuses it with POLICY_LIMIT_EXCEEDED. The help says that this local policy caps --max-pages.

The PR adds the crawl capability and a crawl skill, both available, with a registry entry. It adds crawl lines to the zenrows and trace-debug skills, a read-me (README) section and table row, and a docs/capabilities.md row. The help, README and skills call Crawl "new", with one line that says Crawl is still evolving.

Test guards changed on purpose

File	Before	Now
tests/production-surface.test.ts	crawl is an unknown command	checks only discover and monitor; the registry lists crawl
tests/registry.test.ts	no crawl skill exists	same as above
tests/capabilities.test.ts	—	pins the crawl status to available

Acceptance test (acceptance/crawl/)

run.sh installs the release artifact (npm install -g of the npm pack tarball) in a clean container with digest-pinned images. The container gets no source, no build caches and no tokens, and it gets the key by name only. The run follows the README Crawl quickstart, then runs checks 1 to 21 and the CLI extras. Each check writes one JSON result: PASS, FAIL or SKIP. acceptance/crawl/README.md gives the expected results. The offline half of check 1 also creates without --depth and checks that no depth is sent. Check 21 expects the documented default of 100 for max_pages and max_items; it passes once the API makes depth optional and moves those defaults to 100. The report verdict is "ready" when it has no blocker or major finding.

#	Feature	#	Feature	#	Feature
1	Create	8	Results	15	Bad key
2	Create with html	9	Content	16	Errors carry the server code
3	Idempotency	10	Download	17	Naming
4	Wait	11	Stop	18	Copy
5	Wait runs out	12	Not found	19	Failed crawl
6	Get	13	Invalid input	20	Results after stop
7	List	14	Gated option	21	Documented defaults

The CLI extras check --json on every subcommand, the Ctrl-C exit 130 with the resume hint, and that --wait and --html are unknown flags. The harness also runs on macOS. The files list in package.json is an allow-list without acceptance/, so the published package does not contain the harness.

Cost and safety	Value
Crawl size	depth 1, max_pages 10 or less, except check 19's crawl, which leaves max_pages and max_items at the API default (100) for check 21 and fetches one page
Crawls per run	5; check 9 starts a sixth (at most 10 pages) when the stop in check 20 lands before the crawl fetches a page
Pages per run	about 10 on the default target, at most 50 (60 with the sixth crawl)
Active crawls	at most 2 (the account limit is 3 active jobs, shared with Batch); a 429 waits Retry-After and tries again
Clean-up	stops every crawl the run started, also on a failed check or Ctrl-C; retries a stop that gets 503 crawl_busy, and records a crawl it cannot reach
Key	never printed; the run scans every file under out/ for it before it ends

Tests

npm run typecheck and npm test cover the unit tests.

File	Covers
tests/crawl-api.test.ts	request shape, response parsing and error mapping for each status in the error table; each row of the retry table, including a stop retried once on 429, 502 or 504 after a 1 s wait, a plain 503 stop that honors Retry-After, a stop that gives up after 3 retries with the mapped error, a 429 without too_many_crawls on stop that maps to CRAWL_RATE_LIMITED, and a 500 stop with no retry; a create without depth that sends none; a Retry-After above 30 s that waits 30 s; server_code and crawl_id; paging on running, ended and stopped crawls; the end and timeout of a wait; the streamed download, its line count, and a restart after a mid-stream failure
tests/crawl-command.test.ts	subcommands and flags; a create without --depth that sends only url and prints a crawl with no depth; an unset --max-pages counted as 100 against max_pages_per_run; a create 429 too_many_crawls as CRAWL_TOO_MANY_CRAWLS and a stop 429 without it as CRAWL_RATE_LIMITED; rejection, before any request, of removed flags (--html, --include, --exclude, --wait, results --download, results --cursor) and removed subcommands (start, status); flag validation; JSON keys; the --out --json envelope; a wait that runs out; Ctrl-C during a wait; a failed download that keeps the earlier file; --out into a missing directory; a content id with a bad % escape
tests/e2e/crawl.e2e.test.ts	the built binary, with npm run test:e2e; skips when ZENROWS_E2E=1, ZENROWS_API_KEY or ZENROWS_E2E_CRAWL_URL is missing; docs/contributing.md has a "Running the Crawl e2e test" section

The e2e test passed once against a live API deployment on the first commit. The test now uses the new subcommands and flags. Nobody ran it again after these changes.

How to try it

export ZENROWS_API_KEY="$(tr -d '\n' < path/to/api.key)"
npm run zenrows -- crawl create <start-url> --depth 1 --max-pages 10 \
  --include-patterns <pattern> --follow --json
npm run zenrows -- crawl results <crawl-id> --out urls.jsonl

ZENROWS_E2E=1 ZENROWS_E2E_CRAWL_URL=<start-url> \
  ZENROWS_E2E_CRAWL_INCLUDE=<pattern> npm run test:e2e

# acceptance test: needs bash, git and Docker
bash acceptance/crawl/run.sh              # every check
bash acceptance/crawl/run.sh --no-key     # offline checks only; live checks SKIP
bash acceptance/crawl/run.sh --report     # rebuild the report from out/
REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub

With a key that has Crawl access, create --follow prints the ended crawl, and urls.jsonl holds the kept URLs. A key without Crawl access gets CRAWL_NOT_ENABLED. The acceptance report is acceptance/crawl/out/crawl-first-install-report.md.